They have started putting some models in huggingface:
https://huggingface.co/collections/microsoft/phi-3-6626e15e9...
Phi-3 Technical Report
101–110 of 132 posts
Re: Phi-3 Technical Report
#102Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…
Not trying to disparage them, but their models always give a feeling that it is overfitted on benchmarks hence they perform so well. On everyday tasks, it's much worse - chat or simple completion tasks. Distilling can work and there are papers which suggest it does, but we still do not have a reliable mechanism which can distill knowledge from larger teacher models to smaller student models.
This was the case for Phi-2, it was notoriously rubbish in practical use.
Re: Phi-3 Technical Report
#103Earlier quoted context omitted.
> (weights haven't been released, though) Phi-1, Phi-1.5, and Phi-2 have all had their weights released, and those weights are available under the MIT License. Hopefully Microsoft will continue that trend with Phi-3. > outputs 1212 tokens/sec on iOS I think you meant "12 tokens/sec", which is still nice, just a little less exciting than a kilotoken/sec.
Weights are coming tomorrow.
tomorrow is now: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
Re: Phi-3 Technical Report
#104the weights have been released, 4k https://huggingface.co/microsoft/Phi-3-mini-4k-instruct and 128k context https://huggingface.co/microsoft/Phi-3-mini-128k-instruct
Re: Phi-3 Technical Report
#105Incredible, rivals Llama 3 8B with 3.8B parameters after less than a week of release. And on LMSYS English, Llama 3 8B is on par with GPT-4 (not GPT-4-Turbo), as well as Mistral-Large. Source: https://chat.lmsys.org/?leaderboard (select English in the dropdown) So we now have an open-source LLM approximately equivalent in quality to GPT-4 that can run on phones? Kinda? Wild. (I'm sure there's a lot of nuance to it, f…
It’s not open source, but is open weight - like distributing a precompiled executable. In particular what makes it open weights rather than just weights available is that it is licensed using an OSI approved license (MIT) rather than a restricted proprietary license.
I really wish these companies would release the training source, evaluation suites, and code used to curate/filter training data (since safety efforts can lead to biases). Ideally they would also share the training data but that may not be fully possible due to licensing.
Re: Phi-3 Technical Report
#106Earlier quoted context omitted.
This won't dethrone Llama 3, but it's equally impressive. They mention this model's relative weakness in the TruthfulQA eval, since it's more lossy trying to pack 'knowledge' into a small model relative to problem-solving skills (which shine on MMLU) Regardless - still a very useful thing to have offline and on the fly. Those scores are nothing to scoff at. Given that these pipelines are likely harder harder to imita…
Even llama3 has its issues. Ive been quite impressed so far but if the context gets a little long it freaks out, gets stuck repeating the same token or just fails to finish an answer. This is for the full f16 8B model, so it cant be put down to quantization. It also doesnt quite handle complex instructions as well as the benchmarks would imply should.
I had this same issue with incomplete answers on longer summarization tasks. If you ask it to "go on" it will produce a better completion, but I haven't seen this behaviour in any other model.
Re: Phi-3 Technical Report
#107Earlier quoted context omitted.
Couldn't it be that training it on 4chan makes it more truthful for some reason?
Could it be that people who can talk anonymously with no reputation to gain or lose and no repercussions to fear actually score high on truthfulness? Could it be that truthfulness is actually completely unrelated to the offensiveness of the language used to signal in-group status?
This unironically feels like good research & paper potential.
Re: Phi-3 Technical Report
#108They have started putting some models in huggingface: https://huggingface.co/collections/microsoft/phi-3-6626e15e9...
And with a MIT license!
Re: Phi-3 Technical Report
#109Earlier quoted context omitted.
When I tried Phi2 it was just bad. I don't know where you got this fantasy from that people accept obviously wrong answers, because of "pandering".
Obviously correct answer matters more but ~100-200 elo points could be gained just for better writing. Answer would be range of 500 elo in comparison.
> just for better writing
in my use cases, better writing makes a better answer
Re: Phi-3 Technical Report
#110Everyone needs to take these benchmark numbers with a big grain of salt. According to what I've read, Phi-2 was much worse than its benchmark numbers suggested. This model follows the same training strategy. Nobody should be assuming these numbers will translate directly into a high ranking on the LMSYS leaderboard, or usefulness in everyday tasks. Let's not dethrone Llama 3 until some real world testing can be done.…
The Chinchilla paper is about how to design/train a model to optimize use of computing power, which we can equate to cost (FLOPs cost dollars).
The question that Chinchilla tries to answer is: for a given training budget (which you can think of as dollars or FLOPs), what is the optimal trade off of model size and quantity of training data to get the most performant model? Build a large model and train with less data, or build a smaller one and train with more data?
However, another consideration is minimizing total lifetime cost of the model: training cost + inference cost. You could train a model for longer (costing more) in order to get a given level of performance from a smaller model that will be cheaper for inference, or vice versa. For any given projected model lifetime inference volume, there is going to be a different answer.
It's not that Chinchilla-optimal models stopped making sense, but rather that this sort of consideration has people willing to pump more money (tokens) into smaller models to reduce inference cost for that level of capability.