Cloudflare took down our website after trying to force us to pay $120000 within 24h : technology

[-] dojan@lemmy.world 383 points 6 months ago* (last edited 6 months ago)

Cloudflare took down our website after trying to force us to pay 120k$ within 24h

Yikes. That sounds bad.

I'm a SysOps engineer at a fairly large online casino.

Okay all my sympathy is gone. Online casinos deserve to die.

That said, my feelings towards economic vampires aside, the way the events unfolded is concerning to say the least. Cloudflare has been racking up evil-corp points quite rapidly in recent months.

[-] Potatos_are_not_friends@lemmy.world 167 points 6 months ago

As a person who works in server hosting (not as devops or IT), I'm often privy to customer interactions. I feel like my company does a really good job at damage control - where if we fuck up, some rep gets on the phone and makes things right. We've eaten costs on behalf of our customers.

But sometimes, you just gotta tell a customer to go fuck themselves.

And those customers, those biggest complainers are often in online gambling, crypto, adult content, or racist shit.

We get DDos'd a lot from it. But I'm glad the company I work for doesn't bow down to garbage companies.

[-] dojan@lemmy.world 75 points 6 months ago

I'm honestly not surprised.

I used to hook up with a guy who was 100% convinced that he could game the system. It had something to do with break frequencies from various services and certain time windows for playing. He won sometimes, but he obviously didn't talk much about his losses. He wasn't a very happy person, and I think gambling offered an easy release.

That's my big issue with gambling. It's a business preying on addicts leaving many in financial ruin, and overall they do nothing for society at large. Here in Sweden it is regulated, but you honestly don't notice it. There are so many internet casinos vanishing and cropping up on an almost daily basis. If you turn on the radio the adverts are like 40% online casinos, 40% sex toy sites, and 20% various services, like tyre shifting, glass repairs, etc.

load more comments (7 replies)

[-] goferking0@lemmy.sdf.org 63 points 6 months ago

I just wonder how much was left out

[-] dojan@lemmy.world 31 points 6 months ago

That's fair, this is one part of the story, and it's not like screenshots can't be doctored. Any screenshot taken from the web is ridiculously easy to manipulate.

load more comments (1 replies)

load more comments (19 replies)

[-] Tramort@programming.dev 198 points 6 months ago

Jesus. Something shady is happening with cloudflare.

That does not inspire confidence.

[-] Vlyn@lemmy.zip 176 points 6 months ago* (last edited 6 months ago)

Is there? The casino is on a cheap $250 a month plan they don't belong on and they broke ToS with the domains. While also costing Cloudflare money each month (as the casino admits themselves, their traffic alone is worth up to $2000 a month).

It's absolutely in the right of Cloudflare to drop a customer that's bothersome. Casinos usually are (regulations, going around country restrictions), them costing them money on top is a massive issue.

120k a year is a big slap of course, but it's probably the amount Cloudflare would want to keep them on as a customer. If they leave, so be it.

I've seen it several times before at companies I worked at. They cheaped out and went with a tiny service plan to coast by. Or even broke ToS because it would be cheaper. That usually got stopped by plans getting dropped (GitLab Bronze for example), cheap plans getting limited, or the sales team sending a 'friendly' message that we're abusing their plan and how we're going to fix it. If you don't play along at that point you're going to get the hammer dropped on you.

It also wasn't 24h as the title says, the first communication happened in April. At that point they should have started to scramble, either upgrading to a bigger tier immediately or switching providers. And it's totally normal to go to the sales team when you break the ToS of your plan or you abuse a smaller plan. They're going to discuss terms, it's not a technical issue.

Edit: And I should also say, the whole "paying for a whole year is extortion" is bullshit too. Their CFO or CEO told Cloudflare they are looking at switching providers (as they looked at Fastly). So of fucking course Cloudflare is going to demand a full year upfront. Otherwise the casino could pay for a single month and during that month they switch away to another provider. So Cloudflare would still be thousands in the red with that ex-customer after they used so much traffic the last few years.

[-] tiramichu@lemm.ee 83 points 6 months ago* (last edited 6 months ago)

That Cloudflare were justifiably unhappy with the situation and wanted to take action is fine.

What's not fine is how they approached that problem.

In my opinion, the right thing for Cloudflare to do would have been to have an open and honest conversation and set clear expectations and dates.

Example:

"We have recently conducted a review of your account and found your usage pattern far exceeds the expected levels for your plan. This usage is not sustainable for us, and to continue to provide you with service we must move you to plan x at a cost of y.

If no agreement is reached by [date x] your service will be suspended on [date y]."

Clear deadlines and clear expectations. Doesn't that sound a lot better than giving someone the run-around, and then childishly pulling the plug when a competitor's name is mentioned?

[-] realbadat@programming.dev 57 points 6 months ago

Considering the perspective of the poster, the misleading title, etc - are you actually sure they didn't?

load more comments (2 replies)

[-] slaacaa@lemmy.world 126 points 6 months ago* (last edited 6 months ago)

The biggest red flag is the up-front payment for a year, gives the indication that they are in actual financial trouble, meaning short in cash right now.

Fucking idiots could have been just increasing the price yearly without any resistance, it’s unlikely a big casino would care about an extra 50-100 per month.

[-] Vlyn@lemmy.zip 38 points 6 months ago

As I said in another comment: The up-front payment is the only thing that makes sense for Cloudflare. You got a customer that's costing you money each month. They broke ToS. You offer them a deal still to keep the services running. And their CEO/CFO tells you they are looking at other providers like Fastly.

If Cloudflare gave them a monthly contract then the casino would simply pay for a month and switch over their services to a competitor in that time. So Cloudflare loses all the money from the past (where the casino used far too much traffic) and will barely recoup 10k (minus the running cost, so more likely 7k at the high end) for a single month. It's just not worth it.

So they offer: Stick with us for a full year at least or get fucked. Which is fair.

load more comments (4 replies)

load more comments (3 replies)

[-] HowManyNimons@lemmy.world 26 points 6 months ago

CloudFlare don't need to subsidise an online casino with millions of subscribers, at everyone else's expense. Sure CF are a bunch of gigglefucks but this time I think they made a good decision.

[-] xxd@discuss.tchncs.de 52 points 6 months ago* (last edited 6 months ago)

Unless the casino is doing something illegal, it's really not their decision to make. If they don't want to subsidize them, all they'd have to do is be transparent and fair in their pricing. They way CF handled it instead just seems unprofessional and deceitful.

load more comments (1 replies)

[-] catloaf@lemm.ee 41 points 6 months ago

Now they're getting $0 and bad press, so no I don't think they did.

[-] HowManyNimons@lemmy.world 29 points 6 months ago

$0 is better than having a customer whose costs exceed their revenue; it looks like the bad press is being managed; and also fuck online casinos very much.

[-] FederatedSaint@lemmy.world 34 points 6 months ago

Just because you don't like online casinos, doesn't mean cloudflare didn't completely fuck this up. They could have negotiated reasonable terms to increase their revenue on this account instead of going the route of stonewalling and extortion.

So not only did they lose this customer, but this bad press will ensure a lot of others never sign up with them, potentially costing them millions in foregone sales.

Yeah this was a massive boondoggle..

load more comments (2 replies)

load more comments (1 replies)

load more comments (5 replies)

load more comments (2 replies)

[-] Gestrid@lemmy.ca 166 points 6 months ago

Found the thread on HN. Here's what (I'm guessing) a mod had to say:

It set off the flamewar detector, got flagged by users, and got downweighted by a mod.

The 'customer support of last resort' genre is common and not usually a good fit for HN [1]. If people feel this story is unusually relevant and interesting, I'm not sure I agree—long experience has taught us that one-sided articles like this nearly always leave out critical information—but I also don't mind yielding in an occasional specific case, so I've rolled back the penalties on this thread.

The issue from our point of view is not about story X or company Y—it's a systemic one: the most popular genres of submission (especially the rage-inducing ones) get massively over-represented by default, so countervailing mechanisms are needed [2] if we're to have a space for the more intellectually curious stories that the site is meant for.

[1] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20%22last%20resort%22%20support&sort=byDate&type=comment

[2] https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20countervail&sort=byDate&type=comment

[-] starman@programming.dev 26 points 6 months ago

Okay, that's understandable

load more comments (2 replies)

[-] solrize@lemmy.world 141 points 6 months ago* (last edited 6 months ago)

HN thread is here and it's on the front page 7 hours old: https://news.ycombinator.com/item?id=40481808

Many mentions made that a significant part of the issue seemed to be Cloudflare IP addresses getting banned in some countries. They wanted the customer to switch to a bring-your-own-IP plan.

Also, the discussion took place over 1 month, not 24 hours.

I think the HN thread is reasonably informative and nuanced. CF didn't do great but it was somewhat a fog of war situation.

[-] goferking0@lemmy.sdf.org 61 points 6 months ago

Yeah this substac just reads as we abused cloudflare then were surprised they didn't take us saying no well.

[-] draughtcyclist@lemmy.world 120 points 6 months ago

Realistically, this is why you pay for Akamai. You don't get these shenanigans.

How the fuck were they still on a $250 dollar a month plan when they pumped through $2000 a month worth of traffic? That's shady on the companiy's part and Cloudflare shouldn't have allowed it to happen in the first place.

Each party played their part here and did shitty things. Sounds like the tech equivalent of a crackhead arguing about selling stuff to the pawn shop employee.

[-] ryven@lemmy.dbzer0.com 103 points 6 months ago* (last edited 6 months ago)

The $250/month plan supposedly includes unlimited traffic. If there's actually a limit where you're supposed to switch to a more expensive plan with no standardized price, maybe CF should say what the limit is?

[-] draughtcyclist@lemmy.world 67 points 6 months ago

They absolutely should have outlined a traffic limit for the $250 a month plan. That's on Cloudflare for allowing it.

That said, if you make wildly excessive use of that loophole it probably shouldn't surprise you if they do something like this. They called it "trust and safety" because it allows them to do anything they want under the guide of security.

Really, they didn't define their service clearly and wanted to fire them as a customer unless they paid up for what they felt they were owed.

[-] TheTetrapod@lemmy.world 79 points 6 months ago

If something is marketed as "unlimited", I don't think there is such a thing as "wildly excessive use". This isn't a competitive eater going to an all-you-can-eat buffet and being mad about getting kicked out. It's a business using a service in a way that's seemingly in-line with what they paid for.

load more comments (2 replies)

[-] Gullible@sh.itjust.works 35 points 6 months ago

A man walks into whorehouse at half past seven, inquires about prices, and learns that it’s 250 per night, per person for the room. “Everything they consent to is available to the customer” says the proprietor. Gladly he pays and climbs up the steps with his hand clasped tenderly, finally landing upon a plain pink cushion, whereupon he proceeds to fuck the absolute shit out of his companion for six full hours. The brothel quakes in rhythm with their dual shrieks of ecstasy for the full duration.

As he begins dressing himself across from the nearly comatose prostitute, the proprietor returns, requesting two hundred and ninety dollars for the extended stay and sixty for the damage to her employee. It was at that moment that the man realized that the madame was a 70 foot tall crustacean from the Paleozoic era. He yells “goddamn Loch Ness monster, I ain’t giving you no three fifty!”

load more comments (2 replies)

load more comments (1 replies)

[-] neuracnu 71 points 6 months ago

I worked for Akamai for 7 years.

This is why, if your CDN infra is core to the operation of your business, you make your systems accommodate multi-CDN integration. Cutting one CDN off shouldn't be significantly difficult, and it comes in handy during contract negotiations. All the major players work this way.

[-] NOT_RICK@lemmy.world 111 points 6 months ago

Well that all reads like extortion.

[-] Speculater@lemmy.world 37 points 6 months ago

"Pay us money or we will destroy your business." Pretty cut and dry extortion. The entire article was infuriating to read.

[-] Eyeuhnluuung@lemmy.world 108 points 6 months ago

The irony here, is this is the kind of vague and obtuse fuckery online casinos and sportsbooks pull with their customers all the time.

[-] VantaBrandon@lemmy.world 85 points 6 months ago

The tl;dr seems to be this was a money losing account for Cloudflare, and they couldn't squeeze them so they weaseled out with some TOS violation to prevent losing money on what was promised to be unlimited traffic, they have better lawyers so they're not worried.

Cloudflare 100% in the wrong here, they are closing accounts for TOS violations when they are just unprofitable, I would very strongly consider how tightly to couple with them knowing how cavalier they are about squashing small businesses.

If enough of these happen though, they'll get destroyed by a class action lawsuit, and they'd deserve every bit of it

[-] daq@lemmy.sdf.org 83 points 6 months ago

CF doesn't give a fuck about 80tb of traffic. These guys were in severe TOS violation that could affect all CF customers if CF IPs got blocked. Given 48 hours to bring their own IPs and switch to (expensive AF anywhere) enterprise account and finally shut down TWO WEEKS later after trying to weasel their way out of this instead of accepting they need to pay to play this stupid game.

We've been CF customers forever and enshitification is definitely affecting all of their services and mostly customer support, but in this instance I'm 100% on the side of CF.

[-] merc@sh.itjust.works 49 points 6 months ago

I’m 100% on the side of CF.

100%?

We scheduled a call with their “Business Development” department. Turns out the meeting was with their Sales team,

...

So we scheduled another call, now with their "Trust and Safety" team. But it turns out, we were actually talking to Sales again.

This is the part that's ridiculous to me. If CloudFlare thinks they're violating TOS that's fine. If they're willing to let them continue with their business as-is as long as they pay more? That's fine. But, scheduling calls with one group and it turns out it's actually CloudFlare's sales team on the phone, that's ridiculous.

[-] SquiffSquiff@lemmy.world 25 points 6 months ago

It seems that you've misunderstood what the issue is here from cloudflare's perspective. The customer was using cloudflare IP addresses, which is causing a knock-on effect for the rest of cloudflare's customers and putting cloudflare as a business themselves at risk. The alternative was for the customer to use their own IP addresses as cloudflare advised . I'm not sure what you think 'Business development' teams do but I certainly wouldn't be expecting engineering advice from them.

load more comments (1 replies)

load more comments (2 replies)

load more comments (8 replies)

[-] KairuByte@lemmy.dbzer0.com 48 points 6 months ago

Okay, yes this is an issue. But small business? This was a multinational casino site… that doesn’t scream small business to me.

load more comments (3 replies)

[+] ssj2marx@lemmy.ml 75 points 6 months ago* (last edited 4 months ago)

[deleted]

[-] UnderpantsWeevil@lemmy.world 36 points 6 months ago

Sounds like a shake down, and it couldn't have happened to a more deserving group.

Still, real lesson in how Cloudflare does business.

load more comments (3 replies)

[-] todd_bonzalez@lemm.ee 52 points 6 months ago

Regarding the HN shenanigans, their algorithm does some weird things.

If a new post gets too many upvotes and not enough comments, it gets demoted very quickly.

If any of the activity appears manufactured, it basically delists the post.

Very exploitable, but also prevents popular articles that don't stimulate conversation from sticking around on page 1 for too long, and makes botting upvotes do more harm than good.

load more comments (1 replies)

[-] Chriszz@lemmy.world 39 points 6 months ago

250$ a month for their service seems like cloudflare was straight up losing money on the deal. Although cloudflare seemed to have given them extra time than they said before terminating service, which they didn’t have to do. That being said, I think both sides suck here.

[-] bane_killgrind@lemmy.ml 66 points 6 months ago

Nah. CF initiated a contract renegotiation, and then suspended services right after being informed the customer was price leveling.

That's crappy.

They gave less than a single billing period notice for a price increase.

That's crappy.

They sent a price increase for 40x the current billings, with no corroborated cost or value.

See where I'm going here?

load more comments (3 replies)

[-] catalog3115@lemmy.world 35 points 6 months ago

I really love cloudflare especially for my hobby projects but in this case they asked for outright Ransome. From this I learnt to keep Nameservers & domain sellers different. I am going to transfer domain away from nameserver.

[-] drdabbles@lemmy.world 27 points 6 months ago

Not just "this case", there's been countless cases like this with CF.

load more comments (4 replies)

[+] fartington@lemm.ee 32 points 6 months ago* (last edited 4 months ago)

[deleted]

[-] gdog05@lemmy.world 49 points 6 months ago

I didn't see anyone saying that but the correct response is: in the next billing cycle we're going to start charging X amount per month for X amount of traffic or we have to bump you to to the next tier.

If things are so dire that the traffic is causing problems the correct response is to throttle certain domain traffic until it gets figured out in some way.

Pay X amount in 24 hours or we remove you entirely is extortion.

[-] HowManyNimons@lemmy.world 25 points 6 months ago

The guy's running an online casino. With millions of subscribers. CloudFlare can kick them off the internet and steal the copper off the eyelets of their boots for all I care.

load more comments (1 replies)

load more comments (5 replies)

[-] someguy3@lemmy.world 32 points 6 months ago

Also, interesting comment I found on HN:

What is HN?

[-] catloaf@lemm.ee 53 points 6 months ago

Hacker News: https://news.ycombinator.com/

load more comments (11 replies)

[-] the_crotch@sh.itjust.works 31 points 6 months ago

Repoint your DNS, send everything to legal, delete Facebook hit the gym

[-] anticurrent@sh.itjust.works 29 points 6 months ago* (last edited 6 months ago)

Don't believe anything advertised as unlimited , cause it isn't, they always cover their asses in the fine prints in their TOS.

Technology

Our Rules

Approved Bots