51–60 of 135 posts
Those MacBook Pros with 96 GB of unified GPU/CPU memory are looking pretty good right now. It would be awesome to have all this running on a laptop in a completely offline mode.
But if you really want a portable offline thing, sure.
I have a whole host of personal pdf's and documentation that I would love to be able to ask questions about.
Better evaluation paints a bit different picture: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... *FreeWilly2 is a Llama2 70B model finetuned on an Orca style Dataset EDIT: actually, impressive: FreeWilly2 GPT-3.5 GPT-4 ARC 71.1 85.2 96.3 HellaSwag 86.4 85.5 95.3 MMLU 68.8 70.0 86.4 TruthfulQA 59.4 47.0 59.0 So reasoning (ARC) is lagging behind, but the other evaluations are at GPT-3.5 level and closi…
That seems more in-line with my experience. I have been using GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried to swap out LLaMA2 70B in a few of the "easier" tasks, and it hasn't performed well enough yet for any of my tasks done by GPT-3.5.
Earlier quoted context omitted.
There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…
That's helpful! I've done a lot of work in audio synthesis, which is notoriously difficult measure. The gold-standard is human ratings of audio quality, but it is tough to design good tests (easy to fatigue raters) and the iteration time waiting for results is quite long. Instead, there's now some projects which use neural networks trained on human ratings to predict audio quality, such as ViSQoL: https://github.com/…
Earlier quoted context omitted.
When quantized to 4 bits, yes. You lose some quality by doing that, though, as compared to the full f16.
From what I've gathered when reading up on this topic, if RAM is your constraint the common thought has been that higher parameter models quantized down to smaller sizes will outperform lower parameter models running at higher quantization, i.e. it may still be preferable to use the 70B Llama model quantized to 4-bits than something like an unquantized f16 Falcon 40B or the "coming soon" f16 33B Llama2.
It looks like ChatGPT length is 827 while LLaMA2 length is more than double at 1790. Disclaimer from the site: > Caution: GPT-4 may favor models with longer outputs and/or those that were fine-tuned on GPT-4 outputs. > While AlpacaEval provides a useful comparison of model capabilities in following instructions, it is not a comprehensive or gold-standard evaluation of model abilities. For one, as detailed in the Alpa…
Which is not close, because performance is logarithmic in training compute. Each additional percentage point of performance requires exponentially greater investment in compute during pretraining. Llama 2 was pretrained on 2 trillion tokens -- a significant investment in compute, for sure, but still not enough to get close to GPT-4.
I haven't had a chance to use the GPT-4 API yet - is it that much better than the GPT-4 available via ChatGPT? Or am I misunderstanding?
Anecdotal evidence here - I find that the API is less likely to ask questions about what you are doing and get straight to the answer. For example, if I were to ask how to do something with burp it will just answer instead of going into the "as an AI" monologue.
*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.
There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. De…
I even read a classic book the other day and had a great discussion with ChatGPT about moral relativism, the different schools of thought and how it fit into philosophy as a whole. For students this technology is incredible, I wish I had it for all my classes.
Even sometimes comments I'll make on here or Reddit I'll pass through ChatGPT first to see if I made any mistakes in my logic.
*When asked by GPT4 to compare the outputs. I'm a staunch believer that it would be foolish to rely on GPT4 for quality comparisons, and it has been mind boggling to see so many people do it and treat it as perfect proof of anything. It would be slightly more understandable if there was a study to see how human and gpt4 preferences compare, but I'm unaware of any such thing.
The world has long been divided into two camps: People who think computers can make mistakes; and people who think computers never make mistakes, and blame the humans that program them.
Well, now the computers are programming themselves. And clearly they're making mistakes.