Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

211–220 of 268 posts

Re: Slashing data transfer costs in AWS

#211

Post by vidarh »

Hamuko wrote: post #45
jonatron wrote: post #28

Earlier quoted context omitted.

https://aws.amazon.com/service-terms/ > 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services (e.g., proxying network traffic from Services to the public internet or other destinations or excessive data processing through load balancing or content delivery network (CDN) Services as described in the technical documentation), and if you do, we may throttle or suspe…

At least AWS is fully aware how premium their normal data transfer is and that one might want to optimise those costs.

It's extreme enough that I never willingly serve data directly from AWS without a caching proxy elsewhere in front unless the egress is tint.

It takes very low hitrates before it pays for itself several times over including management overheads.

Sometimes you can justify a complete replica outside AWS (one of the things I will gladly pay AWS for is durability)

Re: Slashing data transfer costs in AWS

#212

Post by adrianmonk »

vlovich123 wrote: post #116

> it’s almost as if S3 doesn’t charge you anything for transient storage? This is very unlike AWS, and I’m not sure how to explain this. I suspected that maybe the S3 free tier was hiding away costs, but - again, shockingly - my S3 storage free tier was totally unaffected by the experiment, none of it was consumed (as opposed to the requests free tier, which was 100% consumed). It’s also possible their billing system…

Maybe some sampling mechanism comes along and takes a snapshot once per hour.

Suppose you store the data there for 6 minutes. Then there's an 90% probability that the sampler misses it entirely and you pay $0. But there's a 10% probability that the sampler does catch it. Then you pay for a whole hour even though you used a fraction of that.

Over many events, it averages out close to actual usage[1]. In 9 out of 10 cases, you pay 0X actual usage. In 1 out of 10 cases, you pay 10X actual usage. (But you can't complain because you did agree to 1-hour increments.)

---

[1] Assuming no correlation between your timing and the sampler's timing. If you can evade the sampler by guessing when it runs and carefully timing your access, then you can save a few pennies at the risk of a ban.

Re: Slashing data transfer costs in AWS

#213

Post by emmanueloga_ »

For those suggesting VPSs instead of cloud based solutions, how do you deal with high availability? Even for a small business you may need it to stay up at all times. With a VPS this is harder to accomplish.

Do you setup the same infrastructure in two or more VPS instances and then load balance? (say, [1]). Feels a bit of an ... artisanal solution, compared to using something like AWS ECS.

1: https://www.hetzner.com/cloud/load-balancer

Re: Slashing data transfer costs in AWS

#214

Post by ipaddr »

Turing_Machine wrote: post #204
ipaddr wrote: post #202

Earlier quoted context omitted.

Cloud doesn't make sense for small business. A vps would. If you are spending less than 100,000 you probably don't need it for your 10,000 million or less daily visitors

If you're spending less than 100,000, you almost certainly aren't spending enough to pay salary + benefits for a sysadmin.

Why would you need additional people to manage a vps? The person managing amazon can very easily use cpanel because it is much simpler.

Re: Slashing data transfer costs in AWS

#215

Post by cedws »

greyface- wrote: post #206
cedws wrote: post #84

Earlier quoted context omitted.

I know of a way to get data out of GCP for free, although I haven't tried it in practice. Wonder if I could find a buyer for this info ;)

A guess: tunnel through 169.254.169.254 DNS server?

It's a good guess, but what I have in mind would have high bandwidth.

Re: Slashing data transfer costs in AWS

#216

Post by owenmarshall »

dzikimarian wrote: post #161
overstay8930 wrote: post #152

Earlier quoted context omitted.

Cloud engineers can do the job of 4-5 on-prem people. Our AWS devs don't need to be BGP or ZFS experts, they just need to be AWS experts.

Well our on-prem team doesn't need AWS pricing calculation and optymization expert, so there's that :-)

AWS pricing and optimization is just capacity planning, which doesn’t go away if you run on prem - it just looks different, with longer time horizons & financial implications.

“Will my data center run out of floor space & I need to expand?” (years+)

“Will I have enough cooling & power to support the new racks we need?” (6 months+)

“When do I need to get the server order out to ensure we meet our capacity needs?” (6+ weeks)

Every one of those are capital expenditures, so line them up with the annual budget cycle - be sure to keep enough spare capacity to be responsive for last minute asks.

Don’t think my intent is to romanticize the cloud, either. It’s not better, nor worse, just a different way to manage things.

Of course if your company is sufficiently small, do whatever you know and can do quickly - customer acquisition will be more important than debating the cost of either infra in aws or a colo’d server or two in some racks somewhere. But the complexity doesn’t go away if you go to the cloud, OR if you are all on prem. TINSTAAFL.

Re: Slashing data transfer costs in AWS

#217

Post by echelon »

overstay8930 wrote: post #143
echelon wrote: post #105

Earlier quoted context omitted.

You act like these problems are especially hard. Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder. This is what we're paid for.

"this is what we're paid for" Nope, it's what YOU'RE paid for. I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed). I have rarely m…

That feels really grumpy.

Compare something like rocketry or chemical engineering with running an on-prem DC. I don't see what the complaining is about. It's still a luxury compared to what other professions have to deal with.

Re: Slashing data transfer costs in AWS

#218

Post by no_wizard »

rented_mule wrote: post #172
no_wizard wrote: post #147

Earlier quoted context omitted.

This is a quirk of the business that is Amazon and AWS, as they started by selling excess compute and expertise, due to how Amazon was built as API first internally it was almost natural. It’s in no way the norm for a smaller business.

This matches the public (i.e., non-Amazon) speculation I was hearing around the launch of S3 and later EC2. But not what I was hearing internally when I worked at Amazon. I was there when S3, the first AWS service, and EC2 were launched. I was working on what I believe to be the first Amazon (non-AWS) application that used S3 for storage. Getting that approved was not easy - all the same skepticism existed internally…

In broad strokes, the main point I was makings till stands though: AWS was deliberately made to back the demanding scale of Amazon, it was a bet on the future and the Amazon model as much as it was product service, and that did mean they built expertise and hardware up and sold that as a product none the less.

This still isn't the norm for most businesses, even big ones.

Re: Slashing data transfer costs in AWS

#219

Post by sgarland »

xboxnolifes wrote: post #191
sgarland wrote: post #53

Earlier quoted context omitted.

Next you’ll tell me that full-stack is a lie, and devs don’t actually know how to run a DB.

Full stack is a lie because devs don't build their own hardware.

I don’t think it’s that ridiculous (the next obvious goalpost being producing your own silicon), but since microservices means your DB is purely for your service, it stands to reason that you should also then know how to configure, backup, maintain, and tune the DB.

This is of course a wildly unrealistic ask, which is why I think the idea of merging jobs to save money is stupid. Let people who like frontend do frontend. Let people who like DBs do DBs. Let people who tolerate YAML do DevOps.

Re: Slashing data transfer costs in AWS

#220

Post by icedchai »

mlhpdx wrote: post #208
hipadev23 wrote: post #198

Earlier quoted context omitted.

VPCs don’t cost anything.

Sorry, that was indeed nonspecific, you’re right. The add-on features for VPCs are commingled with the concept for me since they almost always go hand in hand. Internet gateways, transit gateways, EIPs, service endpoints, etc., and their fixed costs. Yuck.

All that stuff definitely adds up. I'm familiar with some low traffic projects with high "security" requirements that have so much overhead due to those sorts of adds-on. All the overhead winds up costing more than the actual compute + bandwidth running the site.
Post reply on HN