Any info on Pythia base model performance versus GPT-3 or 3.5? Couldn't find any benchmarks in the paper. I imagine LLaMA is ahead there.
I've not done any real benchmarking but the OpenAssistant fine tuning from LAION has been done on it. It worked reasonably well for something local but definitely felt like it wasn't nearly as complete/advanced as any of the ChatGPT stuff. I imagine this Databricks setup is more complete there but I personally wouldn't expect too much more than GPT-3 level performance. That said if this dataset is open (I haven't really looked too much at the article yet) then you could quite easily use it to tune LLaMA just like the stanford alpaca models, which might be a better combo. Though that wouldn't be licensed for commercial use then given the underlying license.