Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 API General Availability

openai.com

Post reply on HN
Print view View original

431–440 of 562 posts

Re: GPT-4 API General Availability

#431

Post by yard2010 »

ravenstine wrote: post #13
minimaxir wrote: post #2

With how good gpt-3.5-turbo-0613 is (particularly with system prompt engineering), there's no longer as much of a need to use the GPT-4 API especially given its massive 20x-30x price increase. The mass adoption of the ChatGPT APIs compared to the old Completion APIs proves my initial blog post on the ChatGPT API correct: developers will immediately switch for a massive price reduction if quality is the same (or bette…

My experience is that GPT-3.5 is not better or even nearly as good as GPT-4. Will it work for most use cases? Probably, yes. But GPT-3.5 effectively ignores instructions much more often than GPT-4 and I've found it far far easier to trip up with things as simple as trailing spaces; it will sometimes exhibit really odd behavior like spelling out individual letters when you give it large amounts of text with missing gr…

You're asking too much of it, it has its own existential crisis followed by a mental breakdown

Re: GPT-4 API General Availability

#432

Post by pbmonster »

wokwokwok wrote: post #419
PostOnce wrote: post #372

Earlier quoted context omitted.

Different quantizations can give you a big speedup if you've had "depressingly slow" issues. Even the slowest ones (that fit in RAM) will run at basically interactive speed, not instant, but also not "email speed". I have a laptop with a 2018 CPU and I'm working with them just fine. Text generation style instead of chat style is another avenue that makes the feedback time not so annoying for a developer. at 100ms/tok…

Whaaaaat, how are you getting 100ms per token on an 5 year old potato without a graphics card? Like, not vaguely hand wavey stuff, specifically, what model and what inference code? I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.

I'm running TheBlokes wizard-vicuna-13b-superhot-8k.ggmlv3 with 4-bit quantization on a Ryzen 5 that's probably older than OPs laptop.

I get around 5 tokens a second using the webui that comes with oogabooga using default settings. If I understand correctly, this does not get me 8k context length yet, because oogabooga doesn't have NTK-aware scaled RoPE implemented yet.

Using the same model with the newest kobold.cpp release should provide 8k context, but runs significantly slower.

Note that this model is great at creative writing, and sounding smart when talking about tech stuff, but it sucks horribly at stuff like logic puzzles or (re-)producing factually correct in-depth answers about any topic I'm an expert in. Still at least an order of magnitude below GPT4.

The model is also uncensored, which is amusing after using GPT4. It will happily elaborate on how to mix explosives and it has a dirty mouth.

Interestingly, the model speaks at least half a dozen languages much better than I do, and is proficient at translating between them (far worse than deepL, of course). Which is mindblowing for a 8GByte binary. It's actual black magic.

Re: GPT-4 API General Availability

#433

Post by travisjungroth »

concordDance wrote: post #429
PostOnce wrote: post #197

Promote and proliferate local LLMs. If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too. OpenAI has no moat, unless you give them money to write legislation. I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.

> OpenAI has no moat, unless you give them money to write legislation. Their moat is that they had access to data sources which have since been clamped down on, eg reddit and twitter apis.

You can still download Reddit archives with the same data they used.

Re: GPT-4 API General Availability

#434

Post by Obscurity4340 »

PostOnce wrote: post #197

Promote and proliferate local LLMs. If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too. OpenAI has no moat, unless you give them money to write legislation. I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.

Where can we aquire or access these local LLMs? How much space and specs does it actually require?

Re: GPT-4 API General Availability

#435

Post by mike_hearn »

redox99 wrote: post #219
mike_hearn wrote: post #75

Earlier quoted context omitted.

ChatGPT is definitely more restricted than the API. Example: https://news.ycombinator.com/item?id=36179783

They are comparing text-davinci-003 with ChatGPT which presumably uses gpt-3.5-turbo, so quite different models. They are killing text-davinci-003 btw.

We also compare ChatGPT4 vs GPT4 API in that thread and observe the same difference.

Re: GPT-4 API General Availability

#436

Post by yard2010 »

hungrigekatze wrote: post #99
it_citizen wrote: post #59

Earlier quoted context omitted.

I keep reading “GPT4 got nerfed” but I have been using from day 1, and while it definitely gives bad answers, I cannot say that it was nerfed for sure. Is there any actual evidences other than some user subjective experiences?

See my comment elsewhere on this post. Greg Brockman, head of strategic initiatives at OpenAI, was talking at a round table discussion in Korea a few weeks ago about how they had to start using the quantized (smaller, cheaper) model earlier in 2023. I noticed a switch in March 2023, with GPT-4 performance being severely degraded after that for both English-language tasks as well as code-related tasks (reading and wri…

Oh my god, this is how a lemon market[0] starts..

[0] https://en.m.wikipedia.org/wiki/The_Market_for_Lemons

Re: GPT-4 API General Availability

#437

Post by davedx »

hnfong wrote: post #357
esperent wrote: post #311

Earlier quoted context omitted.

This is a terrible excuse that I see trotted out far to often to justify going to developing countries and barely even paying workers that country's minimum wage. You absolutely can pay considerably more than minimum wage without disrupting the local economy. They're paying people as low as $1.32 per hour for an absolutely horrible job. I'm not expecting them to pay western wages. But even bumping that up to $2.50 or…

I'm curious. When you buy a loaf of bread from the local market, are they cheaper than first world prices? If so, do you pay double the listed price and demand the shop pay double the price to hire workers so as to not exploit them? Are your expenses in said developing country lower than what you would have paid if you were in a richer country? Are you donating the difference to the local community? Just curious.

Hi, I've been to Kenya and Tanzania, and while basic staples are cheaper than developed countries they're not that much cheaper these days. If you watch travelog videos where they ask locals how they're doing, many developing countries are struggling with massive inflation that's been partly caused by volatile energy prices (many people can no longer afford gas) and partly by food shortages from the Ukraine War.

Re: GPT-4 API General Availability

#438

Post by redox99 »

PostOnce wrote: post #368
andai wrote: post #275

Earlier quoted context omitted.

Which models are you using and for which tasks? I have found local models largely a waste of time (except for very simple tasks with very heavy prompting). But perhaps there are some recent breakthroughs I haven't seen yet.

I'm using a variety of 7 and 13B models (and a 3B one for fast feedback loop debugging) at between 8bit and 4_K_M quantizations. Depending on your pre-prompt, your fine-tune (i.e. which model you downloaded), and your specific task, the results can be startlingly good, it's crazy that you can do this on a $250 laptop. I stay up nights working on it lately, it's so interesting. More importantly, things change by the d…

I'm surprised you're having such a good time with 7B and 13B models. I find anything below 33B to be almost useless. And only 65B is close to GPT 3.5.

Re: GPT-4 API General Availability

#439

Post by tap-snap-or-nap »

santiagobasulto wrote: post #56
hospitalJail wrote: post #33

I imagine the API quality isnt nerfed on a given day like ChatGPT can be. There was no question something happened in January with ChatGPT, weirdly would refuse to answer questions that were harmless but difficult(Give me a daily schedule of a stoic hedonist) Every once in a while, I see redditors complain of it being nerfed. Sometimes I go back to gpt3.5 and am mind boggled how much worse it is. Makes me wonder if t…

I felt the same thing. The first version of GPT-4 I tried was crazy smart. Scary smart. Something happened afterwards…

I agree. It is difficult to say what happened exactly but I am certain that I got all the answers and very few canned responses. Whatever they did for safety has degraded the product.

Re: GPT-4 API General Availability

#440

Post by lynx23 »

PostOnce wrote: post #197

Promote and proliferate local LLMs. If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too. OpenAI has no moat, unless you give them money to write legislation. I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.

I tried, and decided it is not worth it. llama.cpp with a 13B model fit into RAM of my laptop, but pushes CPU temperature to 95 degrees within a few seconds, and mightily sucks the battery dry. Besides, the results were slow and rather useless. GPT is the first cloud application I deliberately use to push off computing and energy consumption to an external host which is clearly more capable of handling the request then my local hardware.

I sympathize with the idea of wanting to run a local LLM, but IMO, this would require building a desktop with a GPU and plenty of horsepower + silent cooling and put it somewhere in a closet in my apartment. Running LLMs on my laptop is (to me) clearly a waste of my time and its battery/cooling.

Post reply on HN