71–80 of 134 posts
I was in same situation. We bough 2 Dell servers via their financing program. Each server is about 19-25K. We paid AWS $60K per month before that. We pay $600 for co-location. So my advice is try to get hardware via financing of provider Dell had a good program I think.
We managed to cut our costs by about 2/3rd by doing two things:
1) moving to batch (this spins up machines to run docker containers without much hassle. You can also share instances) 2) use spot instances.
Spot instances integrate nicely into batch, and depending on how you set it up, you can optimise for speed or cost. for example a p2.xlarge is still $.9 but on spot its about $0.25-0.35
AWS is super expensive. Switch to another cloud provider. For example : Scaleway, OVH, or Hetzner.
I was in same situation. We bough 2 Dell servers via their financing program. Each server is about 19-25K. We paid AWS $60K per month before that. We pay $600 for co-location. So my advice is try to get hardware via financing of provider Dell had a good program I think.
Are you sure you are using the right type for what you need to generate? Can you have your model generator self kill (stop) the instance when it finishes the model?
100% If it doesnt need JIT go spot and build models off queue
Put in for the activate program. They can give you up to 100k of credits.
Earlier quoted context omitted.
I was in the same boat and this is good advice! I stopped using gpu's, "Vectorized inference isn’t bad at all!". This soo much, I was blinded with gpu speed, using tensorflow builds with avx optimization is actually pretty fast. My discovery: + Stop expensive GPU's for inference and switch to avx optimized tensorflow builds. + Cleaned up the inference pipeline and reduced complexity. + Buying compute instance for a y…
Also, consider physical servers from providers like Hetzner. These can be several times cheaper than EC2.
However, this morning I was playing around with Scaleway bare metal [1] and General Purpose instances [2] -- I am thinking of making a switch for high CPU tasks.
[1] https://www.scaleway.com/en/bare-metal-servers/
[2] https://www.scaleway.com/en/virtual-instances/general-purpos...
Here's a direct timestamp link to that point in the podcast where it came from: https://runninginproduction.com/podcast/33-zego-lets-you-eas...
I was in same situation. We bough 2 Dell servers via their financing program. Each server is about 19-25K. We paid AWS $60K per month before that. We pay $600 for co-location. So my advice is try to get hardware via financing of provider Dell had a good program I think.
What does colocation mean in this context? Did you buy the servers and AWS hosted on their premises?