Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slashing data transfer costs in AWS

bitsand.cloud

Post reply on HN
Print view View original

191–200 of 268 posts

Re: Slashing data transfer costs in AWS

#191

Post by xboxnolifes »

sgarland wrote: post #53
antonvs wrote: post #35

Earlier quoted context omitted.

> Host it yourself. If you're actually using the features of cloud - i.e. managed services - then this involves building out an IT department with skills that many companies just don't have. And the cost of that will easily negate or exceed any savings, in many cases. That's a big reason that the choice of cloud became such a no brainer for companies.

Next you’ll tell me that full-stack is a lie, and devs don’t actually know how to run a DB.

Full stack is a lie because devs don't build their own hardware.

Re: Slashing data transfer costs in AWS

#192

Post by shermantanktop »

photonthug wrote: post #154
shermantanktop wrote: post #127

Earlier quoted context omitted.

It is absolutely true that some devs have the AWS product set as the tech toolkit they know best. Whatever their fundamental skills are, the most important way they add value is by optimizing things like lambda startup time or EC2 CPU utilization. Does this allow them to mask deep problems with fundamentals? I guess it could, but that sounds a bit gatekeep-y to me.

Sort of, but IDK, If you have specific needs this might be a somewhat reasonable heuristic for hiring. Devs who came up building software more or less from scratch really do have a different skillset than ones who stick to working in service-rich environments because there's a significant difference between glueing services together vs building out those same services. For example something like using a paginated API…

Paginated api is a great example. For me, I learned C from K&R and producing a.out files that would default and leave a core file in $HOME. If I wanted a list structure, I had to build it out of resizable arrays of pointers, etc.

I ended up years later at AWS, and while I was there I built internet-facing paginated APIs over resources which had a variety of backing stores, each of which was had some behavior I had reason about.

So I don’t doubt the difference between API builder and API user, I’ve been both. I think it’s less about what you are doing and more about how you do it (with curiosity about how things work, vs. as an incurious gluer).

That said, looking at the code inside MySQL is highly instructive for the curious; AWS doesn’t provide that warts-and-all visibility into their implementations, which cuts off the learning journey through the stack.

Re: Slashing data transfer costs in AWS

#193

Post by mlhpdx »

hipadev23 wrote: post #117
mlhpdx wrote: post #82

I’ve been deploying 3xAZs in 3xRegions for a while now (years). The backing store being regional s3 buckets (keeping data in the local compliance region) and DDB with replication (opaque indexing and global data) and Lambda or Sfn for compute. So far data transfer hasn’t risen to the level of even tertiary concern (at scale). Perhaps because I don’t have video, docker or “AI” in play?

I’m guessing either you don’t have much data or your infra is already so absurd that, yeah, the transfer costs are irrelevant by comparison.

Not using VPCs (no need without instances/containers/RDS) mean most of the “absurd” costs go away. It’s cheap by any standard.

Re: Slashing data transfer costs in AWS

#194

Post by danielklnstein »

xbar wrote: post #188

After my account started getting bills this month for pennies for which there was no obvious accounting, I slashed my AWS costs by 100%. I'm back to managing my own systems. So much cheaper and less chance of nonlinear bills.

In case you ever decide to return to AWS, its Cost Explorer is far from perfect but it can show you where your expenses are coming from, especially if your costs are pennies. In the last re:invent they even released daily granularity when grouping by resources (https://aws.amazon.com/blogs/aws-cloud-financial-management/...).

Re: Slashing data transfer costs in AWS

#195

Post by FireBeyond »

rrrix1 wrote: post #180
overstay8930 wrote: post #152

Earlier quoted context omitted.

Cloud engineers can do the job of 4-5 on-prem people. Our AWS devs don't need to be BGP or ZFS experts, they just need to be AWS experts.

Hilariously ironic, with a sufficiently large cloud footprint, things like BGP (and more/other internetworking protocols) and OpenZFS become required skillsets. I have firsthand experience of this. :)

Yeah, there was an amusement there. I've definitely had to understand BGP to configure cloud VPC setups.

"They just have to be an AWS expert".

Right. They just have to be experts in: EC2, S3, Aurora, DynamoDB, RDS, Lambda, VPC, LightSail, Athena, EMR, RedShift, MQ, SQS, SNS, ECR, ECS, EKS, ElastiCache, CloudWatch, CloudTrail, IAM, Cognito, and a few more. No big deal.

Re: Slashing data transfer costs in AWS

#196

Post by tiffanyh »

overstay8930 wrote: post #91
andrewstuart wrote: post #3

An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…

If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…

> on premise have no idea how much the salary costs of someone

I haven’t yet meet anyone at a company that heavily uses cloud and still doesn’t have the same number of salaried infra people as on-prem.

Re: Slashing data transfer costs in AWS

#198

Post by hipadev23 »

mlhpdx wrote: post #193
hipadev23 wrote: post #117

Earlier quoted context omitted.

I’m guessing either you don’t have much data or your infra is already so absurd that, yeah, the transfer costs are irrelevant by comparison.

Not using VPCs (no need without instances/containers/RDS) mean most of the “absurd” costs go away. It’s cheap by any standard.

VPCs don’t cost anything.

Re: Slashing data transfer costs in AWS

#199

Post by intelVISA »

develatio wrote: post #26

I'll share my trick :) Lightsail instances can be used to "proxy" data from other AWS resources (eg EC2 instances or S3 buckets). Each Lightsail instance has a certain amount of data transfer included in it's price ($3.5 instance has 1TB, $5 instance has 2TB, $10 instance has 3TB, $20 instance has 4TB, $40 instance has 5TB). The best value (dollar per transferred data) is the $10 instance, which gives you 3TB of traf…

Nice trick but you are playing with fire due to the AWS' terms.

Re: Slashing data transfer costs in AWS

#200

Post by rstuart4133 »

overstay8930 wrote: post #150
SOLAR_FIELDS wrote: post #148

Earlier quoted context omitted.

Is that really a fair comparison though? AWS is a very weird argument to make because you could say that AWS is kind of “on premise” for Amazons purposes. Internally Amazon.com does not pay retail pricing or have the same level of support as third party end users. A better example would be looking at Jet.com/Walmart and asking if it runs on AWS.

Nobody who is big is paying retail prices, that's why saying "on premise is cheaper" is total copium. As soon as you start factoring in discounts (i.e. bandwidth is nearly free at some point), the math of being on premise completely falls apart to the point you are paying more for licensing and support for the hardware than you are the entire lifecycle of your infrastructure in the cloud. It's just that bad to do it…

> licensing and support for the hardware

Sorry, but who pays for licensing and support of the hardware? I've never done that. You buy a Dell server or whatever, you pay for the 5 or 7 years warranty up front, you put in in a rack and you never literally touch it again until it's EOL'ed. If something breaks Dell touches it, not you. That typically costs around $500/year, albeit paid upfront.

But you usually don't do that. Instead you rent a dedicated metal from someone. The cost if of the order of $1000/year including some storage, no more to pay unless you exceed 100TB/year. A similarly configured AWS EC2 instance is $13,000/year, plus bandwidth, plus whatever other services you get sucked into. And you will get sucked into it because if you ask AWS about any problem (like say, monitoring why your bills are so high), the answer is invariably "use this paid service of ours".

You're kidding yourself if you think using AWS is cheaper than the alternatives. Those discounts you speak of are from an absurdly high starting point. I'm sure there are lots of reasons to use AWS or a similar cloud service, but unless you only need a lot of grunt for at most a few months price isn't one of them.

Post reply on HN