Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

11–20 of 175 posts

Re: OpenLLM

#11

Post by rmbyrro »

comfypotato wrote: post #7
visarga wrote: post #2

Fine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.

What exactly do you mean here that the smaller models can compete with the the larger once they are fine-tuned? What about once the larger models are fine-tuned? Are they then out of reach of the fine-tuned smaller models?

They're probably referring to fine-tuning on private/proprietary data that is specific to a use case. Say a history of conversation transcripts in a call center.

Larger models, like OpenAI's GPT, don't have access to this by default.

Re: OpenLLM

#12

Post by aarnphm »

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Re: OpenLLM

#14

Post by chaoyu »

comfypotato wrote: post #7
visarga wrote: post #2

Fine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.

What exactly do you mean here that the smaller models can compete with the the larger once they are fine-tuned? What about once the larger models are fine-tuned? Are they then out of reach of the fine-tuned smaller models?

Smaller models are likely more efficient to run inference and doesn't necessarily need the latest GPU. Larger language model trend to have better performance over more different type of tasks. But for a specific enterprise use case, either distilling a large model or use large model to help with training a smaller model can be quite helpful in getting things to production - where you may need cost-efficiency and lower latency.

Re: OpenLLM

#15

Post by chaoyu »

SimFG wrote: post #13

Looks great! I'm planning to integrate it into my new project(to-chatgpt: https://github.com/SimFG/to-chatgpt ), which will provide users of the ChatGPT applications with a wider range of LLM service options.

Looking forward to it!

OpenLLM is adding a OpenAI-compatible API layer, which will make it even easier to migrate LLM apps built around OpenAI's API spec. Feel free to join our Discord community and discuss more!

Re: OpenLLM

#16

Post by moffkalast »

mjburgess wrote: post #10
moffkalast wrote: post #9

Earlier quoted context omitted.

Well it's not a completely meaningless metric as it immediately tells you roughly how much memory you need to load it, which is kind of important?

If you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.

Well then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds?

Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.

Re: OpenLLM

#17

Post by moffkalast »

aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Very cool, btw it's not mentioned in the readme so I assume it's only for running full precision models or do quantized GGML/GPTQ/etc. also work with it?

Re: OpenLLM

#18

Post by aarnphm »

moffkalast wrote: post #17
aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Very cool, btw it's not mentioned in the readme so I assume it's only for running full precision models or do quantized GGML/GPTQ/etc. also work with it?

Hi there, 8bit and 4bit is currently supported on main. GPTQ is working in progress, as well as GGML

Re: OpenLLM

#19

Post by boomskats »

moffkalast wrote: post #16
mjburgess wrote: post #10

Earlier quoted context omitted.

If you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.

Well then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds? Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.

FWIW I easily interpreted these as '100s of petabytes' and '0.5 terabytes' without having to give it too much thought. The original comment explicitly specified 'bytes' as the unit being suggested.

Re: OpenLLM

#20

Post by quickthrower2 »

rmbyrro wrote: post #11
comfypotato wrote: post #7

Earlier quoted context omitted.

What exactly do you mean here that the smaller models can compete with the the larger once they are fine-tuned? What about once the larger models are fine-tuned? Are they then out of reach of the fine-tuned smaller models?

They're probably referring to fine-tuning on private/proprietary data that is specific to a use case. Say a history of conversation transcripts in a call center. Larger models, like OpenAI's GPT, don't have access to this by default.

OpenAI’s API has fine tuning options for older GPT models: davinci, curie, babbage, and ada
Post reply on HN