Could you train a ChatGPT-beating model for $85k and run it in a browser?
1–10 of 175 posts
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#2The WebGPU demo mentioned in this post is insane. Blows any WASM approach out of the water. Unfortunately that performance is not supported anywhere but chrome canary (behind a flag)
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#3> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour.
AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking.
You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#4> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour. AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking. You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.
Maybe they are using spot instances? $1/hr is about right for those.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#5Keep in mind that image transformer models like stable diffusion are generally smaller than language models, so they are easier to fit in wasm space.
Also. you can finetune llama-7b on a 3090 for about $3 using LoRA.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#6> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour. AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking. You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.
AWS certainly isn't the cheapest for this, did they mention using AWS? Lamdba Labs is 12$/hr for 8xA100's, and there's others relatively close to this price on demand, I assume you can get a better deal if you contact them for a large project.
Replicate themselves rent out GPU time so I assume they would definitely know as that's almost certainly the core of their business.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#7> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour. AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking. You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.
Lambda labs charges about 11-12/hr for 8xA100.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#8If you could, you should have done it 6 months ago.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#9Keep in mind that image transformer models like stable diffusion are generally smaller than language models, so they are easier to fit in wasm space. Also. you can finetune llama-7b on a 3090 for about $3 using LoRA.
Only for images. People want to generate videos next and those models will be likely GPT-sized.
Re: Could you train a ChatGPT-beating model for $85k and run it in a browser?
#10> My friends at Replicate told me that a simple rule of thumb for A100 cloud costs is $1/hour. AWS charges $32/hr for an 8xA100s (p4d.24xlarge) which comes out to $4/hour/gpu. Yes you can get lower pricing with a 3 year reservation but thats not what this question is asking. You also need 256 nodes to be colocated on the same fabric -- which AWS will do for you but only if you reserve for years.
lambdalabs will let you do on-demand 8xa100 @ 80GB VRAM/GPU for $12/hr, or reserved @ $10.86/hr
8xA100 @ 40gb for $8/hr
Replicate friend isn't far off.