Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

together.ai

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#11

Post by benxh »

cjonas wrote: post #6
skilled wrote: post #4

This reads like something that a site using AdSense would use. 2.5 is interesting > Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdi…

I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable. If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claud…

I've had some success using vast.ai[0] with the Oobabooga LLM WebUI (LLaMA2) instances. One click to start up, minimal editing in the interface settings to enable OpenAI compatible interface.

[0] https://cloud.vast.ai/

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#12

Post by jofi1 »

coder543 wrote: post #7

I can't recall hearing about this website before. In fact, only one link to this website has ever gained any traction here[0], and it wasn’t much. Why is it surprising that a random website offering AI services might not have a great privacy policy? I feel like there must be dozens of websites like this that offer hosting of Llama2 by now. My default assumption would be that they’re not as trustworthy as OpenAI, who…

I don't have any affiliation with them, but Together is hardly random and very important to the open source community, putting out some of the most useful work around things like new architectures, distributed training, datasets.

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#13

Post by jankovicsandras »

Better privacy might be running Llama2 locally, offline.

If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards.

  ! git clone https://github.com/ggerganov/llama.cpp.git

  ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q8_0.gguf" -P llama.cpp/models

  ! cd llama.cpp && make

  ! ./llama.cpp/main -m ./llama.cpp/models/codellama-7b.Q8_0.gguf --color --ctx_size 2048 -n -1 -ins -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1.1 -t 8

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#14

Post by _boffin_ »

agnokapathetic wrote: post #8

It's quite easy to opt-out of OpenAI using your data for training. API and ChatGPT Enterprise are private by default. # OpenAI Products and Privacy - ChatGPT Free - trains on your data unless you Opt Out - ChatGPT Plus - trains on your data unless you Opt Out - ChatGPT Enterprise - does _not_ train on your data - OpenAI API - does _not_ train on your data Opt-out link: https://help.openai.com/en/articles/5722486-how-…

It’s not just training, but data retention and the ability for employees to access the prompts and completions.

You only touch half the problem

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#15

Post by lardo »

jankovicsandras wrote: post #13

Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve…

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#17

Post by Palmik »

Privacy policy summaries + q&a would be a nice public service. Here's summary of this one:

> Personal data collected includes user names, text or image prompts, IP addresses, contact information, and more. The data is used for various purposes, such as providing and improving the service, managing user accounts, contacting users, and analyzing data. Personal data may be shared with service providers, third-party vendors, and other business partners. The company retains personal data for as long as necessary and transfers it to locations where data protection laws may differ. The security of personal data is important, although no method of transmission over the internet is 100% secure. The service is not intended for children under the age of 13, and the company does not knowingly collect their personal information. The privacy policy also includes information about links to other websites, changes to the policy, and supplemental terms and conditions for certain regions. For any questions or to exercise privacy rights, users can contact the company via email.

Generated here: https://dstill.ai/hackernews/item/37436498

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#18

Post by Patrick_Devine »

jankovicsandras wrote: post #13

Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve…

Or, for the easy way of doing it, just install Ollama and run: `ollama run codellama:7b`.

If you want the q8 version (which you probably don't), it's `ollama run codellama:7b-code-q8_0`.

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#19

Post by jmorgan »

lardo wrote: post #15
jankovicsandras wrote: post #13

Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve…

https://huggingface.co/TheBloke/CodeLlama-7B-GGUF describes codellama-7b.Q8_0.gguf as "very large, extremely low quality loss - not recommended"

Q4_0 is often mentioned as being the "tried and true" quantization level to try first. I've heard folks have had good results with 3-bit quantization (Q3_K_M) as well

Re: Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

#20

Post by rvz »

jankovicsandras wrote: post #13

Better privacy might be running Llama2 locally, offline. If somebody hasn't tried running LLMs yet, here are some lines that do the job in Google Colab or locally. The !s are for Colab, remove them for local terminal. The script downloads the ca. 8GB model, but Llama.cpp can run offline afterwards. ! git clone https://github.com/ggerganov/llama.cpp.git ! wget "https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve…

Or if you're on macOS just hit 'Download' and use Ollama locally. [0]

[0] https://ollama.ai/download

Post reply on HN