Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 API General Availability

openai.com

Post reply on HN
Print view View original

551–560 of 562 posts

Re: GPT-4 API General Availability

#551

Post by bredren »

ignoramous wrote: post #409
bredren wrote: post #385

Earlier quoted context omitted.

Running an LLM locally and paying for access to OpenAI are two separate concerns. But to address both: is it very relevant what LLM you use right now? Local or hosted, openAI or other? It seems like the interface has converged around chat-based prompts. New ideas for tuning or improving the efficiency of foundational models are published almost every week. If one wants to build a product on top of of generative AI, w…

> Couldn’t someone say prototype the entire product on some lower-quality LLM and occasionally pass requests to GPT4 to validate behavior? This, infact, might be a better way to do inference anyway: https://twitter.com/Francis_YAO_/status/1675967988925710338 > So, to return to the point, can’t people just build their product with openAI or other and plan to move away based on the cost and fit for their circumstances?…

Asking because I have not implemented these yet: is there anything unique about the syntax that it can't just be copied?

Re: GPT-4 API General Availability

#552

Post by pbmonster »

staticman2 wrote: post #529
pbmonster wrote: post #432

Earlier quoted context omitted.

I'm running TheBlokes wizard-vicuna-13b-superhot-8k.ggmlv3 with 4-bit quantization on a Ryzen 5 that's probably older than OPs laptop. I get around 5 tokens a second using the webui that comes with oogabooga using default settings. If I understand correctly, this does not get me 8k context length yet, because oogabooga doesn't have NTK-aware scaled RoPE implemented yet. Using the same model with the newest kobold.cpp…

"Note that this model is great at creative writing" Could you elaorate on what you mean by that, like, are you telling it to write you a short story and it does a good job? My experiments with using these models for creative writing have not been particularly inspiring.

Yes, having the model write an entire short story or chapter is not very good. It excels if you interact closely with it.

I tested it to create NPCs for fantasy role playing games. I think its the primary reason cobold.cpp exists (hence the name).

You give it a (ideally long, detailed) prompt describing the character traits of the NPCs you want, and maybe even add back and forth dialogue with other characters to the prompt.

And then you just talk to those characters in the scene you set.

There's also "story mode", where you and the model take turns writing a complete story, not only dialogue. So both of you can also provide exposition and events, and the model usually only creates ~10 sentences at a time.

There's communities online providing extremely complex starting prompts and objectives (escape prison, assassin someone at a party and get away, ect.) for the player, and for me, the antagonistic ones (the models has control over NPCs that don't like you) are surprisingly fun.

Note that one of the main drivers of having uncensored open source LLMs is people wanting to role-play erotica with the model. That's why the model that first had scaled RoPE for 8k context length is called "superhot" - and the reason it has 8K context is that people wanted to roleplay longer scenes.

Re: GPT-4 API General Availability

#553

Post by juliensalinas »

ignoramous wrote: post #549
lhl wrote: post #446

Earlier quoted context omitted.

In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs. You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I th…

> https://github.com/turboderp/exllama Is exllama an alternative to llama.cpp?

llama.cpp focuses on optimizing inference on a CPU, while exllama is for inference on a GPU.

Re: GPT-4 API General Availability

#554

Post by ignoramous »

juliensalinas wrote: post #553
ignoramous wrote: post #549

Earlier quoted context omitted.

> https://github.com/turboderp/exllama Is exllama an alternative to llama.cpp?

llama.cpp focuses on optimizing inference on a CPU, while exllama is for inference on a GPU.

Thanks. I thought llama.cpp got CUDA capabilities a while ago? https://github.com/ggerganov/llama.cpp/pull/1827

Re: GPT-4 API General Availability

#555

Post by ignoramous »

bredren wrote: post #551
ignoramous wrote: post #409

Earlier quoted context omitted.

> Couldn’t someone say prototype the entire product on some lower-quality LLM and occasionally pass requests to GPT4 to validate behavior? This, infact, might be a better way to do inference anyway: https://twitter.com/Francis_YAO_/status/1675967988925710338 > So, to return to the point, can’t people just build their product with openAI or other and plan to move away based on the cost and fit for their circumstances?…

Asking because I have not implemented these yet: is there anything unique about the syntax that it can't just be copied?

Some (not all) projects are indeed "copying" the OpenAI APIs; ex: https://github.com/go-skynet/LocalAI/issues/588

Re: GPT-4 API General Availability

#556

Post by juliensalinas »

ignoramous wrote: post #554
juliensalinas wrote: post #553

Earlier quoted context omitted.

llama.cpp focuses on optimizing inference on a CPU, while exllama is for inference on a GPU.

Thanks. I thought llama.cpp got CUDA capabilities a while ago? https://github.com/ggerganov/llama.cpp/pull/1827

Oh it seems you're right, I had missed that.

As far as I can see llama.cpp with CUDA is still a bit slower than ExLLaMA but I never had the chance to do the comparison by myself, and maybe it will change soon as these projects are evolving very quickly. Also I am not exactly sure whether the quality of the output is the same with these 2 implementations.

Re: GPT-4 API General Availability

#557

Post by lhl »

juliensalinas wrote: post #556
ignoramous wrote: post #554

Earlier quoted context omitted.

Thanks. I thought llama.cpp got CUDA capabilities a while ago? https://github.com/ggerganov/llama.cpp/pull/1827

Oh it seems you're right, I had missed that. As far as I can see llama.cpp with CUDA is still a bit slower than ExLLaMA but I never had the chance to do the comparison by myself, and maybe it will change soon as these projects are evolving very quickly. Also I am not exactly sure whether the quality of the output is the same with these 2 implementations.

Until recently, exllama was significantly faster, but they're about on par now (with llama.cpp pulling ahead on certain hardware or with certain compile-time optimizations now even).

There are a couple big difference as I see it. llama.cpp uses `ggml` encoding for their models. There were a few weeks where they kept making breaking revisions which was annoying, but it seems to have stabilized and now also supports more flexible quantization w/ k-quants. exllamma was built for 4-bit GPTQ quants (compatible w/ GPTQ-for-LLaMA, AutoGPTQ) exclusively. exllama still had an advantage w/ the best multi-GPU scaling out there, but as you say, the projects are evolving quickly, so it's hard to say. It has a smaller focus/community than llama.cpp, which also has its pros and cons.

It's good to have multiple viable options though, especially if you're trying to find something that works best w/ your environment/hardware and I'd recommend anyone to HEAD checkouts a try for both and see which one works best for them.

Re: GPT-4 API General Availability

#558

Post by juliensalinas »

lhl wrote: post #557
juliensalinas wrote: post #556

Earlier quoted context omitted.

Oh it seems you're right, I had missed that. As far as I can see llama.cpp with CUDA is still a bit slower than ExLLaMA but I never had the chance to do the comparison by myself, and maybe it will change soon as these projects are evolving very quickly. Also I am not exactly sure whether the quality of the output is the same with these 2 implementations.

Until recently, exllama was significantly faster, but they're about on par now (with llama.cpp pulling ahead on certain hardware or with certain compile-time optimizations now even). There are a couple big difference as I see it. llama.cpp uses `ggml` encoding for their models. There were a few weeks where they kept making breaking revisions which was annoying, but it seems to have stabilized and now also supports mo…

Thank you for the update! Do you happen to know if there are quality comparisons somewhere, between llama.cpp and exllama? Also, in terms of VRAM consumption, are they equivalent?

Re: GPT-4 API General Availability

#559

Post by abhinavkulkarni »

juliensalinas wrote: post #558
lhl wrote: post #557

Earlier quoted context omitted.

Until recently, exllama was significantly faster, but they're about on par now (with llama.cpp pulling ahead on certain hardware or with certain compile-time optimizations now even). There are a couple big difference as I see it. llama.cpp uses `ggml` encoding for their models. There were a few weeks where they kept making breaking revisions which was annoying, but it seems to have stabilized and now also supports mo…

Thank you for the update! Do you happen to know if there are quality comparisons somewhere, between llama.cpp and exllama? Also, in terms of VRAM consumption, are they equivalent?

Here's one: https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...
Post reply on HN