41–50 of 113 posts
Earlier quoted context omitted.
Databricks wasn't the one purchased, MosaicML was. Mosaic's last round had a valuation of $136m, so this is a pretty big jump for them.
The Databricks valuation is relevant if any part of the acquisition was paid for in Databricks equity.
Coincidentally this comes after MosaicML released the best open source commercially usable LLMs on huggingface: mpt-30b, the first open source LLM with 8k context length that can be extended even further with ALiBi and has been trained on a whopping 1 trillion tokens vs. 300 billion for Pythia and OpenLLaMA, and 800 billion for StableLM.
OpenLLaMA models up to 13B parameters have now been trained on 1T tokens: https://github.com/openlm-research/open_llama
Congrats to the MosaicML and wish all the luck! If anyone is interested in MosaicML alternatives, check out dstack, we are building its OSS and cloud-agnostic alternative: https://dstack.ai . Disclaimer: I'm the founder and CEO at dstack.
It would be really cool to support more vendors (eg Lambda Labs, Runpod, etc) - getting GPU availability feels like 'any port in a storm' sometimes :)
> bringing down the cost of using generative AI—from tens of millions of dollars to hundreds of thousands of dollars per model, That's a lot of people you have to replace with one model--even if you get the cost basis down. Do those costs include the period refitting?
What do these companies do? I consider myself mildly well versed in the ML space at least, but I’ve never heard of MosaicML before (have they trained anything popular?). I’ve heard of Databricks but know very little about what service they provided before/after the recent deep learning craze.
[1] https://huggingface.co/replit/replit-code-v1-3b [2] https://www.youtube.com/watch?v=roEKOzxilq4
This class of startup, "build domain specific LLMS using your own data", is extremely crowded right now but I am not optimistic about their future. For large companies, the actual modeling work for this is already easy for any ML team, thanks to existing FOSS work on stuff like PEFT and LoRA. The hard part is figuring out what data goes into the fine tuning process and how to get this data in a usable form, but this…
> thanks to existing FOSS work on stuff like PEFT and LoRA YMMV. Sometimes a LORA is fine, but sometimes a full finetune is necessary for higher quality output. That being said, backwards pass free training keeps making more and more progress. Seems like a short matter of time before it becomes practical.
I just fine tuned a ~30b parameter model on my 2x 3090s to check it out. It worked fantastically. I should be able to fine tune up-to 65b parameter models locally but wanted to get my dataset right on a smaller model before trying.
Falcon 7B/40B is good, but hasn't really caught on like LLaMA has. One big reason is that its not well supported by 4-bit inference code (namely llama.cpp and GPTQ). Mosaic's 7B (and 30B?) models have the same issue, and 7B kinda paled in comparison to LLaMA 7B... But maybe it would be better if finetuned? To me, its kinda baffling that Mosaic didn't work on adding highly quantized inference to the popular frameworks…
Thrilled to hear about MosaicML's successful exit! Nonetheless, this could serve as an indication to explore other options. Given Databricks' past acquisition of Redash which led to its downfall, there's no absolute assurance that Mosaic's fate won't be similar.