I use scientific language models professionally. I skimmed the paper and was immediately disappointed. - They benchmarked against general models like GPT-3 but not well-established specific models that have been trained for specific tasks like SPECTER[0] or SciBert[1]. Specter outperformed GPT-3 on tasks like citation prediction two years ago. Nobody seriously uses general LLMs on science tasks, so nobody who actuall…
There is no doubt in my mind that Galactica fine-tuned on these specific datasets will outperform all these previous models. But yeah, someone should definitely do that and perform the benchmarks.
I’ve been vaguely following all the AI news on text to image and text that comes out from promos. But I have no idea how a benchmark for text would work. Is benchmarking subjective? Is it based on accuracy of information? How do you actually measure a benchmark for something like this?