Earlier quoted context omitted.
> Last time I checked baking my own cookies is more expensive than just buying a bag at Walmart. Yes, but most likely they aren't of comparable quality
Indeed, off-topic, but I find it fascinating that I can buy a year of Kosher, Vegan, Gluten Free Survival Food on Amazon from ~$1469. https://www.amazon.com/NorthWest-Fork-Gluten-Free-Emergency-... That drops to $1149 if I don't require Vegan, non-GMO, Kosher: https://www.amazon.com/Healthy-Survival-Food-Emergency-Prepa... SNAP benefits at anywhere from $134 to $192 a month are both more than sufficient for this. So…
Building your own deep learning computer is 10x cheaper than AWS
161–170 of 269 posts
Re: Building your own deep learning computer is 10x cheaper than AWS
#162I have seen sys admins stand-up a bunch of EC2's in AWS and install PostGres and Docker on them (because the dev's said they need a DB and a docker server). They don't get the services model (use RDS and ECS). Sys admins have to change. Orgs can't afford this cost nor be slowed down by this 1990s mindset. Standing up a bunch of EC2's in AWS is just a horrible idea and an expensive one as well. It also moves all of th…
Yeah that's nice and all, but then you have total and complete vendor lock-in. I decided to do the EC2 thing when I built one of my products, knowing that I couldn't have vendor lock in - and that decision was critical to the survival of my company when: 1) A customer wanted to run on azure. We would have lost a 2.5 million pound contract if we couldn't do that 2) Another customer wanted on-prem solution, we would ha…
The dockerfile used to build the containers in ECS can be pushed to any registry and the resulting containers can run on any docker service.
What vendor lock in? I just don't buy that argument.
Use the services Luke ;) it will cost much less and you can focus on other things!
Re: Building your own deep learning computer is 10x cheaper than AWS
#163Earlier quoted context omitted.
Yeah that's nice and all, but then you have total and complete vendor lock-in. I decided to do the EC2 thing when I built one of my products, knowing that I couldn't have vendor lock in - and that decision was critical to the survival of my company when: 1) A customer wanted to run on azure. We would have lost a 2.5 million pound contract if we couldn't do that 2) Another customer wanted on-prem solution, we would ha…
The DB schema in RDS is identical to the DB schema on a Postgres server in your data center and the RDS data dumps can be loaded into your very own personal DB wherever you like. The dockerfile used to build the containers in ECS can be pushed to any registry and the resulting containers can run on any docker service. What vendor lock in? I just don't buy that argument. Use the services Luke ;) it will cost much less…
Sure RDS can be swapped out easily, but what if I'm using 10 or 20 services?
Re: Building your own deep learning computer is 10x cheaper than AWS
#164Earlier quoted context omitted.
Respectfully, those companies' cloud architects suck. If someone goes to the trouble to migrate onto cloud, and then replicates pre-devops workflows... wow.
This has less to do with cloud vs own hardware and more about how the company is structured. I've worked in companies before with an ops department: a few people responsible for managing all the cloud servers. All the devs (like me at the time) work locally and have access to an isolated dev/uat environment provided by that team. That team had most of the show automated, they weren't really provisioning any machines…
I kinda shake my head if people call for devs to handle all the infrastructure and everything in prod. Why should a developer concern himself with details of scaling and tuning postgres, elasticsearch or loadbalancers? If you don't outsource that, that's the job for the ops team. However, if that's the responsibility of the ops team, there's no reason for dev to have access to these systems beyond the application layer. That just seems like a good way to split the work of running and scaling an application.
Now if we're talking about applications, that's something different. In fact, it's the contrary. I am a big proponent of having developers manage and configure their own applications, including prod. It's fast and makes them develop better applications. However, doing this for critical systems in a responsible way requires automation to enforce best practices. At our place, devs don't have root access to productive application servers, but devs have permissions to configure and use the automation controlling the productive application servers. And it's safe and rather risk-free because the automation does the right thing.
And it's also a different thing if we're talking about test. By all means, deploy a 3 container database cluster to poke around with it. I like experimentation and volatility in test setups and PoCs. Sometimes its just faster to solve the problem with 3 candidate technologies and go from there. Just don't expect things to go into production just like that. We'll have to think about availability, scaling and automating that properly first.
Re: Building your own deep learning computer is 10x cheaper than AWS
#165You're forgetting the cost of fighting IT in a bureaucratic corporation to get them to let you buy/run non-standard hardware Much easier to spend huge amounts of money of Azure/AWS and politely tell them it's their own fucking fault when they complain about the costs. (what me? no I'm not bitter, why do you ask?)
Last week I was in an event with the CTOs of many of the hottest startups in America. It was shocking how much money is wasted on the cloud because inefficiencies and they simply don't care how much it costs.
I guess since they are not wasting their own money, they can always come up with the same excuse: developers are more expensive than infrastructure. Well... that argument starts to fall apart very quickly when a company spends six figures every month on AWS.
I'm on the other extreme. I run my company stuff on ten $300 servers I bought on eBay in 2012 and put inside a soundproof rack in my office in NJ, with a 300 Mb FIOS connection using Cloudflare as a proxy / CDN. The servers run Proxmox for private cloud and CEPH for storage. They all have SSDs and some have Optane storage. In 6 years, there were only 3 outages that weren't my fault. All at the cost of office rent ($1000) + FIOS ($359) + Cloudflare costs and S3 for images and backups.
With my infrastructure, I can run 6k requests per minute on the main Rails app (+ Scala backend) with a 40ms response time with plenty of resources to spare.
Re: Building your own deep learning computer is 10x cheaper than AWS
#166Earlier quoted context omitted.
Innovation being sidelined in favor of status quo and 'cover-my-ass' risk management highlights exactly the reason I won't work for a bureaucratic corporation...
Money^, benefits^^, ample PTO^^^, and a 8-5 workday allowing plenty of time for the important things in life like my family are exactly the reasons I do. ^: You know, like real money, not 90k plus 2.5% of something that will probably not exist next year ^^: You know, like real benefits such as low-cost, low-deduct. health insurance, matching 401k, etc. ^^^: As in, real, actual PTO, not this "unlimited" bullshit some…
Re: Building your own deep learning computer is 10x cheaper than AWS
#167Re: Building your own deep learning computer is 10x cheaper than AWS
#168Earlier quoted context omitted.
According to the article you can break even after only two weeks which is far from "10k cookies every day for multiple years".
It's fine for exactly what the article describes: a student doing projects at home. For anything bigger than that, you'll quickly run into issues. Try talking to your ops team and telling them you want to set up a mid-range desktop PC with extra RAM and a high end graphics card as a DL workstation. I think you'll very quickly find some friction, especially once you want to go into production with it.
IIRC their original hardware plan was based around commodity hardware and getting the most performance per dollar since they built the crawler to be distributed pretty early on.
Point being, consumer grade hardware has come a long way since then, and if you're doing something cutting edge like DL it's not outlandish to expect that rolling your own might be worthwhile and totally justified.
Re: Building your own deep learning computer is 10x cheaper than AWS
#169Earlier quoted context omitted.
I think your pricing information might be slightly out of date. AWS Lightsail's $5 plan is equivalent or better than Linode's $5 plan. Plus AWS Lightsail has an even more barebones plan at $3.5 per month. Two years ago you would be perfectly correct. The cheapest AWS option back then would be a micro EC2 instance at around $20 per month IIRC.
Lightsail is basically a toy to compete with Vultr/DO/OVH/etc. You can't even rename a server or change its metadata. You can't resize a server (except for taking a snapshot and building a new server). Essentially, you can't make any changes to a server and there are zero tools for organizing, tagging, or even adding a description field. Of course, you cannot manage Lightsail instances using the AWS console (or even…
I wonder why that's almost never discussed, like that obvious downside is non-existant.