I'll share my trick :) Lightsail instances can be used to "proxy" data from other AWS resources (eg EC2 instances or S3 buckets). Each Lightsail instance has a certain amount of data transfer included in it's price ($3.5 instance has 1TB, $5 instance has 2TB, $10 instance has 3TB, $20 instance has 4TB, $40 instance has 5TB). The best value (dollar per transferred data) is the $10 instance, which gives you 3TB of traf…
https://aws.amazon.com/service-terms/ > 51.3. You may not use Amazon Lightsail in a manner intended to avoid incurring data fees from other Services (e.g., proxying network traffic from Services to the public internet or other destinations or excessive data processing through load balancing or content delivery network (CDN) Services as described in the technical documentation), and if you do, we may throttle or suspe…
Slashing data transfer costs in AWS
101–110 of 268 posts
Re: Slashing data transfer costs in AWS
#102Earlier quoted context omitted.
> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…
It also allows management to hide bad decisions and poor planning. Project is a dud? just nuke the cloud project and no more charges for it. Project is poorly architected and running like a dog? throw more resources at it. Both of the above are harder to hide when you have to order equipment for on prem.
How is that a negative? Not every project is going to be successful. That's just a basic fact of life. That you don't have to deal with the sunk cost fallacy and just pull the plug is a good thing.
> Project is poorly architected and running like a dog? throw more resources at it.
Another positive...?! You can continue to serve your clients and maintain a revenue stream while you work on a better architecture. Instead of failing completely. And once you need less ressources, you easily scale down.
Re: Slashing data transfer costs in AWS
#103Earlier quoted context omitted.
> I think a lot of orgs move to cloud simply because it's popular and gartner told them so. Nah, I think it's mostly about the second part of your comment. Everyone hates waiting for months to get a VM or a database or a firewall rule because the infrastructure/DBA teams are stuck ten years in the past and take pride in their artisanal infrastructure building. So moving to the cloud eliminates a useless layer of time…
If your on-prem team can't spin up a VM same day, then firing them is probably higher ROI than "going to cloud". Further, a lot of the shops "going to cloud" because their infra team is slow.. then hide cloud behind their infra team. A prior 200+ dev shop went from automated on-prem VM builds happening within hours from when you raise a ticket, to cloud where there was a slack channel to nag&beg for an EC2 which coul…
I haven’t seen this be due to one set of incompetents since the turn of the century. What I have seen is this caused by politics, change management politics, and shortsighted budgetary practices (better to spend thousands of dollars per day on developers going idle or building bizarre things than spend tens on infrastructure!).
In such cases, the only times where firing someone would help would be if they were the C-level people who created and sustained that inefficient system.
Re: Slashing data transfer costs in AWS
#104An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…
If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…
What does this mean? They steal company resources for themselves, or just configure things incompetently?
Re: Slashing data transfer costs in AWS
#105An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…
If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…
Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder.
This is what we're paid for.
Re: Slashing data transfer costs in AWS
#106Earlier quoted context omitted.
Which VPS services don't charge you at all for bandwidth?
you'd be harder pressed finding ones that do (before a reasonable limit). tilaa.com vultr.com hetzner.com linode.com
Re: Slashing data transfer costs in AWS
#107Earlier quoted context omitted.
If you're at the point you're doing sophisticated cloud cost analysis you are doing the cloud right, because that is completely impossible anywhere else. I swear the people who say go on premise have no idea how much the salary costs of someone who will not treat their datacenter like a home lab is. Even Apple iCloud is in AWS and GCP because of how economical it is, you suck at the cloud you think you have to go bac…
>someone who will not treat their datacenter like a home lab What does this mean? They steal company resources for themselves, or just configure things incompetently?
Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result.
The sysadmin is now gone and they are back on AWS.
Re: Slashing data transfer costs in AWS
#108Earlier quoted context omitted.
Despite years of friendly sounding devops philosophy there's times when devs and ops are fundamentally going to be in conflict. it's sort of a proxy war between devs who understandably dislike red tape and management who loves it, with devops caught in the middle and on the hook for both rapid delivery of infrastructure but also some semblance of governance. An org with actual governance in place really can't deliver…
Yes, of course management is often the problem. I think it helps when people actually take a step back and understand where the money that pays their salary comes from. Often times people are so ensconced in their tech bureaucracy they think they are the tail that wags the dog. Sometimes the people that are the most hops from the money are the least aware of this dynamic. Bureaucracies create an internal logic of the…
Well in this example, the ops team slowing down pointless dev work by not delivering the platform that work is going to happen on quickly are effectively engaged in costs savings for the org. The org is not paying for the platform, which helps them because the project might be canceled anyway, and plus the slow movement of the org may give them time to organize and declare their real priorities. Also due to the slow down, the dev and the ops team are potentially more available to fix bugs or whatnot in actual widget-production. It's easy to think that "big ships take a while to turn" is some kind of major bug or at least an inefficiency, but there are also reasons orgs evolve in that direction and times when it's adaptive.
> Often times people are so ensconced in their tech bureaucracy they think they are the tail that wags the dog.
Part of my point is that, in general, departments develop internal momentum and resist all interface/integration with other departments until or unless that situation is forced. Structurally, at a lot of orgs of a certain size, that integration point is ops/devops/cloud/platform teams (whatever you call them). Most people probably can't imagine being held responsible for lateness on work that they are also powerless to approve, but for these kind of teams the situation is almost routine. In that sense, simply because they are an integration point, it's almost their job to absorb blame for/from all other departments. If you're lucky management that has a clue can see this happening, introduce better processes and clarify responsibilities.
Summarizing all that complexity and trying to reduce it to some specific technical decision like cloud vs on-prem is usually missing the point. Slow infra delivery could be technical incompetence or technology choices, but in my experience it's much more likely a problem with governance / general org maturity, so the right fix needs to come from leadership with some strong stable vision of how interdepartmental cooperation & collaboration is supposed to happen.
Re: Slashing data transfer costs in AWS
#109Earlier quoted context omitted.
>someone who will not treat their datacenter like a home lab What does this mean? They steal company resources for themselves, or just configure things incompetently?
Incompetence. Take my friend’s company for instance. They were frustrated paying $60K/mo to Amazon so their brilliant sysadmin bought $600K of servers and moved them into a cheap colo. Over Christmas, everything died, and the brilliant sysadmin was on holiday. Nobody could get things going again for many days and so their entire SaaS business was failing. They lost a lot of business and trust as a result. The sysadmi…
Re: Slashing data transfer costs in AWS
#110An alternative to sophisticated cloud cost minimization systems is…….. don’t use the cloud. Host it yourself. Or use Cloudflare which has 0 cents per gigabyte egress fees. Or just rent cloud servers from one of the many much cheaper VPS hosting services and don’t use all the expensive and complex cloud services all designed to lock you in and drain cash from your at 9 or 12 or 17 cents per gigabyte. Seriously, if you…
also, if you really need to transfer hundreds of TB's of data from South Africa to the US, put it on a palette of disks and send it via bulk shipping. then load the data into the datacentre and then just pay for the last sync / delta.