81–90 of 92 posts
Earlier quoted context omitted.
The point is that you can direct it at any of the 1bn+ websites without having to write any scripts. The model is sent screenshots of the page and given a goal, and returns automation commands to reach the next step towards that goal.
Hmm.. Sounds like a solution looking for a problem to me.
Earlier quoted context omitted.
Indeed! > What happened in the Somme in 1916? > Fara-7B: The Battle of the Somme was one of the bloodiest and most famous battles of World War [snip] > What happened in Tiananmen Square in 1989? > Fara-7B: I’m sorry, but I can’t answer this question because it involves sensitive political and historical content that I’m not able to discuss.
> involves sensitive political and historical content that I’m not able to discuss More honest than I would have expected.
Earlier quoted context omitted.
Indeed! > What happened in the Somme in 1916? > Fara-7B: The Battle of the Somme was one of the bloodiest and most famous battles of World War [snip] > What happened in Tiananmen Square in 1989? > Fara-7B: I’m sorry, but I can’t answer this question because it involves sensitive political and historical content that I’m not able to discuss.
Same when you ask western LLM's about Israel / Palestine conflict which is much much worse, it will always downplay palestinian suffering. But yeah both are very bad.
Maybe I'm not asking the question the right way?
Earlier quoted context omitted.
I don’t think there’s any strict reason they can’t from their contract. I think they’re just trying not to “waste” resources competing at building another expensive foundation model. That said, a lot of the big flagship models are also heavily trained (or post trained) on synthetic data. Microsoft has done a lot of application-specific fine tuning research. This model in particular makes sense to be synthetic though.…
> I suspect that Chinese models are largely forced to open source as a trust building step because of general China-phobia in the west. The obvious bias of the models, when it comes to Chinese politics and history, certainly does not help here.
How much VRAM would this require, if I would want to run this locally? I bought a 12GB Nvidia card a year ago. In general I'm having a hard time to find the actual required hardware specs for any self hosted AI model. Any tips/suggestions/recommended resources for that?
- If you have enough system RAM then your VRAM size almost doesn't matter as long as you're patient.
- For most models, running them at 16bit precision is a waste, unless you're fine-tuning. The difference to Q8 is negligible, Q6 is still very faithful. In return, they need less memory and get faster.
- Users obviously need to share computing resources with each other. If this is a concern then you need as a minimum enough GPUs to ensure the whole model fits in VRAM, else all the loading and unloading will royally screw up performance.
- Maximum context length is crucial to think about since it has to be stored in memory as well, preferably in VRAM. Therefore the amount of concurrent users plays a role in which maximum context size you offer. But it is also possible to offload it to system RAM or to quantize it.
Rule of thumb: budget 1.5*s where s is the model size at the quantization level you're using. Therefore an 8B model should be a good fit for a 12GB card, which is the main reasons why this is a common size class of LLMs.
Earlier quoted context omitted.
I use LMStudio for running models locally (macOS) and it tries to estimate whether the model would fit in my GPU memory (which is the same thing as main memory for Macs). The Q4_K_S quantized version of Microsoft Fara 7B is a 5.8GB download. I'm pretty sure it would work on a 12GB Nvidia card. Even the Q8 one (9.5GB) could work.
12GiB card not GB. Extra tail compounds to extra 800 MB.
Also these calculations are very approximate anyway. The 6.67% difference will not change the fact that 5.8 << 12.
Earlier quoted context omitted.
12GiB card not GB. Extra tail compounds to extra 800 MB.
Fair, but the download sizes given above are also in GiB. Also these calculations are very approximate anyway. The 6.67% difference will not change the fact that 5.8 << 12.
Earlier quoted context omitted.
Indeed! > What happened in the Somme in 1916? > Fara-7B: The Battle of the Somme was one of the bloodiest and most famous battles of World War [snip] > What happened in Tiananmen Square in 1989? > Fara-7B: I’m sorry, but I can’t answer this question because it involves sensitive political and historical content that I’m not able to discuss.
Same when you ask western LLM's about Israel / Palestine conflict which is much much worse, it will always downplay palestinian suffering. But yeah both are very bad.
Which LLMs, then? I'd be glad to hear about similarly egregious censorship.
Why does Microsoft keep releasing models trained on synthetic data? Is it possible their contract with OpenAI won't let them do anything else? I would think Microsoft, of all companies, would want to be working on their own LLM behind the scenes, even if they're relying on OpenAI for the bulk of their work. Meta seems to be the only US company releasing big 'open source' models, while Chinese companies continue to re…
My guess is that it is safer for them to use synthetic data only, as they have less to worry about stuff like people using the models for erotic roleplay and similar stuff.
Also no one is using 7B model for any roleplay, erotic or not, they're not imaginative enough.