161–170 of 194 posts
Earlier quoted context omitted.
gpt3.5 turbo is (mostly likely) Curie which is (most likely) 6.7b params. So, yeah, makes perfect sense that it can't compete with a 70b model on cost.
I thought it was fairly well established that GPT 3.5 has something like 130B parameters and that GPT 4 is on the order of 600-1,000
- gpt-3.5 175b params
- gpt-4 1800b params
Earlier quoted context omitted.
Let's say a French company would offer the same service in the US, swearing no data would be ever siphoned out of the US and no French intelligence service would be allowed to review the data. Would you be comfortable with your patient records being stored there or the business secrets of US companies? Do you believe Microsoft can actually make the same promises and keep them? You don't have to answer the last questi…
I do think large tech companies do pretty well with customer data. As a former Googler I would be comfortable with my Gmail data residing in a foreign datacenter.
For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…
You can run 70B LLAMA on dual 4090s/3090s with quantization. Going with dual 3090s you can get a system that can run LLAMA 2 70B with 12K context for I built two such a systems after burning that much in a week on ChatGPT.
Can you share your system specs? I was looking into something similar but my costs were closer to 6 to 8k for the whole system.
Looks really well executed, nice! I'd shared this idea with a few people. GPT and other LLMs don't allow you to use their output to train competing models, but the implication is that it's fine to use their output to train your own internal alternative models. So you can't sell access to the output as an API, but you can use it to replace your GPT API calls. My other thoughts to extend this are that you could make it…
Earlier quoted context omitted.
Has anyone done such fine tuning on llama though? Afaik most projects like llama.cpp use grammars instead.
Yep! The linked notebook includes an example of exactly that (fine-tuning a 7b model to match the syntax of GPT-4 function call responses): https://github.com/OpenPipe/OpenPipe/blob/main/examples/clas...
For translation jobs, I've experimented with Llama 2 70B (running on Replicate) v/s GPT-3.5; For about 1000 input tokens (and resulting 1000 output tokens), to my surprise, GPT-3.5 turbo was 100x cheaper than Llama 2. Llama 7B wasn't up to the task fyi, producing very poor translations. I believe that OpenAI priced GPT-3.5 aggressively cheap in order to make it a non-brainer to rely on them rather than relying on oth…
Google Maps was also cheap. Initially. So it is aggressively cheap now, but would aggressively change later.