211–220 of 297 posts
Earlier quoted context omitted.
GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params
ChatGPT 3.5 is likely much smaller than GPT-3’s 175b parameters. Based on the API pricing, I believe 8k context GPT-4 is larger than 175b parameters, but less than 1t. https://openai.com/pricing
Anyone know what parameters are best for code generation? I tried something simple for Node.js and it wasn't horrible but not working. Maybe I used the wron parameters. I tried using 0 for the temperature and turning everything else down like I do with the OpenAI API.
Earlier quoted context omitted.
Didn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B
With 32k context the upper bound is more like 175B.
I agree it’s important to get right, but it seems like one of hundreds of safety/alignment issues and that many others are de-emphasised or ignored.
Earlier quoted context omitted.
Didn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B
Not really sure what exactly was said. But in a 2 GPU set, you can technically live load weights on 1 GPU while running inference on the other. At fp32 precision, storing a single layer takes around 40*d_model^2 bytes assuming context length isn’t massive relative to d_model (which it isn’t in GPT-4). At 80GB GPU size this means 40k model width could be stored as a single layer on 1 GPU while still leaving space for…
Earlier quoted context omitted.
If you mean asking it what it's running on, it just hallucinates. As others have noted in the comments here, you can get it to say that it runs on PaLM 3 quite easily.
In chat history you can see which model generated each request - for me it’s always LaMDA
So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?
Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/... The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod... Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the…
I don't understand how this can be considered a technical report. No information on model architecture, distributed training methodology, or optimizations. The "Training dataset" section is a pathetic 0.5 pages long. Come on, Google.
Here is their Chat Playground for PaLM 2 https://console.cloud.google.com/vertex-ai/generative/langua... (you have to be logged in to Google Cloud Console I think) Anyone know what parameters are best for code generation? I tried something simple for Node.js and it wasn't horrible but not working. Maybe I used the wron parameters. I tried using 0 for the temperature and turning everything else down like I do with the…
Edit: It seems I can't use my free credits on Vertex APIs... Not nice.
Here is their Chat Playground for PaLM 2 https://console.cloud.google.com/vertex-ai/generative/langua... (you have to be logged in to Google Cloud Console I think) Anyone know what parameters are best for code generation? I tried something simple for Node.js and it wasn't horrible but not working. Maybe I used the wron parameters. I tried using 0 for the temperature and turning everything else down like I do with the…
Isn't Chat-Bison-001 Palm 1? Edit: It seems I can't use my free credits on Vertex APIs... Not nice.