Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 API General Availability

openai.com

Post reply on HN
Print view View original

441–450 of 562 posts

Re: GPT-4 API General Availability

#441

Post by marci »

wokwokwok wrote: post #419
PostOnce wrote: post #372

Earlier quoted context omitted.

Different quantizations can give you a big speedup if you've had "depressingly slow" issues. Even the slowest ones (that fit in RAM) will run at basically interactive speed, not instant, but also not "email speed". I have a laptop with a 2018 CPU and I'm working with them just fine. Text generation style instead of chat style is another avenue that makes the feedback time not so annoying for a developer. at 100ms/tok…

Whaaaaat, how are you getting 100ms per token on an 5 year old potato without a graphics card? Like, not vaguely hand wavey stuff, specifically, what model and what inference code? I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.

I'm on a thinkpad with a 2016 CPU (i5-7300U) running ubuntu.

I don't know anything so I left default settings.

I get about 450ms/t with airoboros-7b and 350ms/t with orca-mini-3b.

edit: with oobabooga webui

Re: GPT-4 API General Availability

#442

Post by ranguna »

Art9681 wrote: post #373
kristianp wrote: post #297

Earlier quoted context omitted.

Gpt-4 runs on 8 x 220B params[1] and gpt is about 220B params(?). Local LLMs can be good for some tasks, but they are much slower and less capable than the size of model and hardware that openai brings to their apis. Even running a 7B model on the CPU in ggml is much slower than the gpt-3-turbo api, in my experience with a 12th gen i7 intel laptop. [1] GPT4 is 8 x 220B params = 1.7T params: https://news.ycombinator.c…

It's been well documented by now that the number of parameters does not necessarily translate to a better model. My guess is that OpenAI has learned a thing or two from the endless papers published daily that your "instance" of the model is not what it seems. They likely have a workflow that picks the best model suitable for your prompt. Some people may get a 13B permutation because it is "good enough" to produce a c…

> It's been well documented by now that the number of parameters does not necessarily translate to a better model.

That's certainly true, but it's hard to deny the quality of gpt 4. If the issue is the training data, let's just use their training data, it's not like they had to close up shop because of using restricted data.

I think the issue is more on the financial side, it must have been extremely expensive to train gpt 4. Open source models don't have that kind of money right now.

I'll finance open source models once they are actually good, or show realistic promises of reaching that level of quality on consumer hardware. Until then, open source will open source.

I've never bought any kind of subscription or paid api costs to openai, but if gpt 4 finally reached the point where I feel like it's a lot better than just good enough, I'll happily pay for it (while still being on the lookout for open source models that fit my hardware).

Re: GPT-4 API General Availability

#443

Post by kaba0 »

concordDance wrote: post #430
joeythedolphin wrote: post #396

Earlier quoted context omitted.

Sam tells Congress that AI is so dangerous it will extinct humanity. Why? So Congress can license him and only his buddies. Then get goes to euro and speaks with world leaders to remove consumer protection. Why? So he can mine data without any consequences. He is a narcissistic CEO who lies to win. If you are tired of the past decade of electronic corporate tyranny, abuse, manipulation and lies, then boycott OpenAi (…

> Sam tells Congress that AI is so dangerous it will extinct humanity. Why? So Congress can license him and only his buddies. No, he says it because its true and concerning. However, just because AGI has a good chance of making humanity extinct does not mean we're anywhere close to making AIs that capable. LLMs seem like a dead end.

> However, just because AGI has a good chance of making humanity extinct

How? I mean surely it will lead humanity down some chaotic path, but I would fear climate catastrophe much much more than anything AI-related.

Re: GPT-4 API General Availability

#444

Post by kaba0 »

NortySpock wrote: post #392
ed_mercer wrote: post #380

Earlier quoted context omitted.

Forgive me as I’m out of the loop. What propaganda are you referring to?

These ChatGPT tools allow anyone to write short marketing and propaganda prompts. They can then take the resulting paragraphs of puffery and post them using bots or sock puppets to whatever target community to create the illusion of action, consensus, conflict, discussion or dissention. It used to be this took a few people to come up with writing actual responses to forum posts all day, or marketing operations plans,…

I believe a new wave of authentication might come out of this, where it is tied to citizenship for example (or something related to physical reality). Otherwise we will find ourselves in a truly chaotic situation.

Re: GPT-4 API General Availability

#445

Post by xiphias2 »

superkuh wrote: post #84

Yikes. They're actually killing off text-davinci-003. RIP to the most capable remaining model and RIP to all text completion style freedom. Now it's censored/aligned chat or instruct models with arbitrary input metaphor limits for everything. gpt3.5-turbo is terrible in comparison. This will end my usage of openai for most things. I doubt my $5-$10 API payments per month will matter. This just lights more of a fire u…

It won't matter at all at the end of the year, open source llm's will surpass it by that time.

Re: GPT-4 API General Availability

#446

Post by lhl »

andai wrote: post #273
PostOnce wrote: post #197

Promote and proliferate local LLMs. If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too. OpenAI has no moat, unless you give them money to write legislation. I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.

Can you elaborate on scary smart and fast? It's been a month or two since I've tried but the results were depressingly slow and useless for more or less every task I tried. Every time a model is claimed to be "90% of GPT-3" I get excited and every time it's very disappointing. (On that note, after using GPT-4, GPT-3 now seems disappointing almost every time I interact with it.)

In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs.

You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings.

That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference, GPT-3.5 gets 60.4 on pass@1 and 68.9 on pass@10 [4]) - I've only just started playing around with it since replit model tooling is not as good as llamas (doc here: https://llm-tracker.info/books/howto-guides/page/replit-mode...).

I'm interested in potentially applying reflexion or some of the other techniques that have been tried to even further increase coding abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/)

[1] https://github.com/turboderp/exllama#results-so-far

[2] https://github.com/aigoopy/llm-jeopardy

[3] https://github.com/Troyanovsky/Local-LLM-comparison/tree/mai...

[4] https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

Re: GPT-4 API General Availability

#447

Post by pantulis »

jerrygenser wrote: post #134
pantulis wrote: post #67

Earlier quoted context omitted.

You would need to have a local copy of the GPT model, which are not exactly OpenAI's plans.

For embeddings, you can use smaller transformers/llms or sentence2vec and often get good enough results. You don't need very large models to generate usable embeddings.

You are correct, I assumed parent was referring to specific embeddings generated by OpenAI LLMs.

Re: GPT-4 API General Availability

#448

Post by szundi »

rileymat2 wrote: post #397
Art9681 wrote: post #370

Earlier quoted context omitted.

I'm going to play devil's advocate and state that a lot of what you mentioned will be relevant to a tiny part of the world that has the means to enforce this. The law will be forced to change as a response to AI. Many debates will be had. Many crap laws will be made by people grasping at straws but it's too late. Putting red tape around this technology puts that nation at a technological disadvantage. I would go as f…

> How long have they tried to take down The Pirate Bay? Pretty much every permutation of every software that's ever been banned can be found and ran with impunity if you have the technical knowledge to do so. No law exists that can prevent that. Forms of this argument get tossed out a lot. Laws don’t prevent, they hopefully limit. Murder has been illegal for a long time, it still happens.

You missed the point: these laws are not limiting other countries, only those who introduce them. Self-limiting, giving advantage to others.

Re: GPT-4 API General Availability

#449

Post by dagaci »

Obscurity4340 wrote: post #434
PostOnce wrote: post #197

Promote and proliferate local LLMs. If you use GPT, you're giving OpenAI money to lobby the government so they'll have no competitors, ultimately screwing yourself, your wallet, and the rest of us too. OpenAI has no moat, unless you give them money to write legislation. I can currently run some scary smart and fast LLMs on a 5 year old laptop with no GPU. The future is, at least, interesting.

Where can we aquire or access these local LLMs? How much space and specs does it actually require?

https://gpt4all.io/index.html is a good place to start, you can literally download one of the many recommended models.

https://github.com/imartinez/privateGPT is great if you want do it with code.

Re: GPT-4 API General Availability

#450

Post by lhl »

Rastonbury wrote: post #423
wokwokwok wrote: post #419

Earlier quoted context omitted.

Whaaaaat, how are you getting 100ms per token on an 5 year old potato without a graphics card? Like, not vaguely hand wavey stuff, specifically, what model and what inference code? I get nothing like that performance for the 7B models, forget the larger models, using llama.cpp on a pc without an nvidia GPU.

This is a exactly a case in point why people decide to pay OpenAI instead of rolling their own. I'm non-technical but have setup an image gen app based custom SD model using diffusers, so not entirely clueless. But for LLM I have no where idea where to start quickly. Finding a model on a leaderboard, download and setup then customising it and benchmarking is way too much time for me, I'll just pay for GPT4 if ever ne…

I've found https://gpt4all.io/ to be the fastest way to get started. I've also started moving my notes to https://llm-tracker.info/ which should help make it easier for people getting started: https://llm-tracker.info/books/howto-guides/page/getting-sta...
Post reply on HN