Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

Post by yessen » Fri, Sep 08, 2023, 8:45 PM UTC

cjonas wrote: post #6

skilled wrote: post #4

This reads like something that a site using AdSense would use. 2.5 is interesting > Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdi…

I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable. If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claud…

Post by jsemrau » Sat, Sep 09, 2023, 11:04 AM UTC

jmorgan wrote: post #19

lardo wrote: post #15

Earlier quoted context omitted.

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"

Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well