Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

221–230 of 268 posts

Re: Slashing data transfer costs in AWS

#221

Post by cstrahan »

greyface- wrote: post #207
sangnoir wrote: post #205

Earlier quoted context omitted.

> You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services This requires proving the users intent, which is not obvious except in the most blatant of cases (i.e. using Lightsail as a bent-pipe by writing the exact bytes you're reading). If it is a "CSV to Parquet translation layer", how would AWS possibly prove it's anything other than what it claims to be? You'd be payin…

> This requires proving the users intent Companies are permitted to deny service to anyone at any time for any (non-protected) reason. They typically don't have to justify service terminations to a court of law. Who would they be required to prove user intent to, and why?

I don't think you parsed their message correctly. It's not about litigation.

Re-posting a bit of the service terms for easy reference:

> 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services [...]

As you point out, they may terminate your service without any justification in a court of law. So how do they go about terminating the offenders? Well, one trivial way (from a technology and/or policy perspective): terminate everyone's service! If you blindly terminate everyone's service, that will certainly prevent anyone abusing LightSail.

But that's, uh, not good for business. So they probably want to terminate the service of only those people actually abusing it. But how do you do that?

You'd have to look at each account's usage and do something to determine if that traffic is or isn't a means of avoiding data fees from other services. In other words, you'd have to determine the intent of that traffic. Or, put yet another way: "this requires proving the users intent".

If doing so was as trivial as detecting any traffic between LightSail and the other services, they'd just prevent such connections in the first place. So how can AWS tell if some traffic between services is legitimate or not? The unspoken premise of the person you're replying to is that this probably isn't feasible for AWS to catch any and all people abusing LightSail in this way, with the conclusion being that you can (in practice) probably get away with it unnoticed.

Re: Slashing data transfer costs in AWS

#222

Post by cstrahan »

manquer wrote: post #210
sangnoir wrote: post #205

Earlier quoted context omitted.

> You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services This requires proving the users intent, which is not obvious except in the most blatant of cases (i.e. using Lightsail as a bent-pipe by writing the exact bytes you're reading). If it is a "CSV to Parquet translation layer", how would AWS possibly prove it's anything other than what it claims to be? You'd be payin…

This is not a court. Amazon does not have prove anything to anyone. There is going to be a program that will have rules to detect patterns in customer traffic and automatically block when those patterns are tripped. At best you could complain in the forums and maybe if you are lucky a sympathetic community manager may look into your use case.

> Amazon does not have prove anything to anyone.

True. And no one has said that they must prove anything to anyone.

Amazon wants to make money, so they probably don't want to terminate the service of people who are acting in good faith. But that's just another way of saying that they probably want to determine with some certainty that someone is not acting in good faith before terminating their service.

So it's not that Amazon needs to prove anything to anyone. But they do want to prove something to themselves.

Re: Slashing data transfer costs in AWS

#223

Post by quickthrower2 »

api wrote: post #32
quickthrower2 wrote: post #19

This is a loophole. Hitting some loss leader at AWS, but if everyone only buys the $1 hotdog and nothing else then the $1 hotdog gets removed.

It’s not a loss leader. Cloud bandwidth pricing is almost pure profit.

Ok “loss” is a relative word here… a loss compared to what they could have got from you.

Some how AWS has to rip you off so if there is a non rip off gateway to the ripoff then if you can use the non rip off to avoid another rip off, they will close the “loophole”.

Re: Slashing data transfer costs in AWS

#224

Post by greyface- »

cstrahan wrote: post #221
greyface- wrote: post #207

Earlier quoted context omitted.

> This requires proving the users intent Companies are permitted to deny service to anyone at any time for any (non-protected) reason. They typically don't have to justify service terminations to a court of law. Who would they be required to prove user intent to, and why?

I don't think you parsed their message correctly. It's not about litigation. Re-posting a bit of the service terms for easy reference: > 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services [...] As you point out, they may terminate your service without any justification in a court of law. So how do they go about terminating the offenders? Well, one trivial way…

We disagree on the definition of "prove". I would not object to the claim if it had used "determine" or "detect" instead of "prove".

That said, detection is easy. Look for users who spin up a Lightsail instance and use close to 100% of its bandwidth quota before spinning it down. Sort by number of such instances, and tell all users above some cutoff that in your sole discretion you believe they have violated your TOS, and are terminating their service. Doing so is completely legally defensible.

Re: Slashing data transfer costs in AWS

#225

Post by spieden »

issafram wrote: post #179

I've been looking for a place to store files for backup. Already keeping a local copy on NAS, but I want another one to be remote. Would you guys recommend S3? Wouldn't be using any other services.

I use S3 with the DEEP_ARCHIVE storage class for disaster recovery. Costs go up if you have many thousands of files so careful there. Hopefully will never need to access the objects and it's the cheapest I could find.

Re: Slashing data transfer costs in AWS

#226

Post by sangnoir »

greyface- wrote: post #224
cstrahan wrote: post #221

Earlier quoted context omitted.

I don't think you parsed their message correctly. It's not about litigation. Re-posting a bit of the service terms for easy reference: > 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services [...] As you point out, they may terminate your service without any justification in a court of law. So how do they go about terminating the offenders? Well, one trivial way…

We disagree on the definition of "prove". I would not object to the claim if it had used "determine" or "detect" instead of "prove". That said, detection is easy. Look for users who spin up a Lightsail instance and use close to 100% of its bandwidth quota before spinning it down. Sort by number of such instances, and tell all users above some cutoff that in your sole discretion you believe they have violated your TOS…

GP here - feel free to replace "prove" with determine because that's what I meant. My point was that it is really hard for Amazon to detect data exfiltration when its disguised as some other run-of-the mill service. Amazon can cancel anyone's service at anytime, but they can't afford to piss off legitimate customers with capricious, undeserved bans due to false positives. Regardless of where AWS draws the line to separate abuse from legit usage,it will always be possible to skirt underneath it. The crux of my argument is that AWS will tolerate false negatives over false positives.

Re: Slashing data transfer costs in AWS

#227

Post by mad182 »

martinald wrote: post #58
api wrote: post #32

Earlier quoted context omitted.

It’s not a loss leader. Cloud bandwidth pricing is almost pure profit.

It's absolutely amazing that so many devs don't realise this. They seem to think that bandwidth should cost a few cents a month, when in reality it is virtually free. Perhaps the 7c/GB charge was reasonable when AWS came out 15 years ago, but networking has got orders of magnitude cheaper and faster in the intervening time period. What's more odd now that 1gigabit+ home connections are available, it should be obvious…

Yep. I got over 250TB/month traffic (150TB+ outbound) on Hetzner, and I don't pay anything additional for that, just ~800$ month for 11 servers.

At 7c/GB that would be over 10500$ just for the traffic alone, and probably about the same for the processing power.

Re: Slashing data transfer costs in AWS

#228

Post by manquer »

cstrahan wrote: post #222
manquer wrote: post #210

Earlier quoted context omitted.

This is not a court. Amazon does not have prove anything to anyone. There is going to be a program that will have rules to detect patterns in customer traffic and automatically block when those patterns are tripped. At best you could complain in the forums and maybe if you are lucky a sympathetic community manager may look into your use case.

> Amazon does not have prove anything to anyone. True. And no one has said that they must prove anything to anyone. Amazon wants to make money, so they probably don't want to terminate the service of people who are acting in good faith. But that's just another way of saying that they probably want to determine with some certainty that someone is not acting in good faith before terminating their service. So it's not t…

In this case they are actually losing money not gaining by allowing this kind of abuse, both because the bandwidth usage costs money and also because of potential lost billing from other services which now is not billed.

The Lightsail style billing model works same way shared vs leased lines works, if everyone fully used their max allocation it won't be possible to offer service at that price point. They can offer 2TB or 4TB for price because the usage modelling of target users supported that.

No company wants a customer to bypass their usage and pricing ToS even if they are not actively enforcing it, it is lost revenue and/or bringing in customers who you don't really want.

Re: Slashing data transfer costs in AWS

#229

Post by rfoo »

overstay8930 wrote: post #96
rfoo wrote: post #40

Earlier quoted context omitted.

Here's another one: You can download 1TB of data for free from AWS each month, as Cloudfront has a free tier [1] with 1TB monthly egress included. Point it to S3 or whatever HTTP server you want and voila. [1] It used to be 50GB per month for the first 12 months. It was changed to 1TB free forever shortly after Cloudflare posted https://blog.cloudflare.com/aws-egregious-egress

That "shortly" was 2 years, that Cloudflare post had nothing to do with it, Amazon barely considers them a competitor to begin with.

Sigh, I should have quoted both side, here is it: https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer...

Cloudflare rant was posted in July 2021, new Cloudfront free tier was there Nov 2021. I consider changing pricing like this for AWS in 4 months pretty fast.

Where is your 2 years number comes from?

Re: Slashing data transfer costs in AWS

#230

Post by usr1106 »

greyface- wrote: post #224
cstrahan wrote: post #221

Earlier quoted context omitted.

I don't think you parsed their message correctly. It's not about litigation. Re-posting a bit of the service terms for easy reference: > 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services [...] As you point out, they may terminate your service without any justification in a court of law. So how do they go about terminating the offenders? Well, one trivial way…

We disagree on the definition of "prove". I would not object to the claim if it had used "determine" or "detect" instead of "prove". That said, detection is easy. Look for users who spin up a Lightsail instance and use close to 100% of its bandwidth quota before spinning it down. Sort by number of such instances, and tell all users above some cutoff that in your sole discretion you believe they have violated your TOS…

I always assumed that your free quota is proportional to the time you pay. Even the price is not the advertised fixed $3.5, you pay less in months with 30 days than in months with 31 days.

I have not checked my cost and usage reports every time I have some experimental instance for a shorter time, so I am not sure. Just from the general knowledge that AWS is permanently counting every fraction of a peanut. But as the submission shows, exceptions to the rule can exist.

Post reply on HN