Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

21–30 of 175 posts

Re: OpenLLM

#21

Post by mjburgess »

moffkalast wrote: post #16
mjburgess wrote: post #10

Earlier quoted context omitted.

If you look at my suggestion, it's to state exactly that memory -- rather than to estimate based on bits/parameter.

Well then do explain a bit further, I still don't fully grasp what "100s PT in 0.5T" means exactly. 100 petatokens in half a trillion? Half a terrabyte? 100 seconds? Plus afaik base model training tokens don't have the same effect as fine tuning tokens, so there would need to be a way to specify each of those separately.

I edited to be TB,PB --- I was thinking of these as prefixes on bytes

Re: OpenLLM

#22

Post by quickthrower2 »

uniqueuid wrote: post #4
visarga wrote: post #2

Fine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.

Yes but fine tuning requires a lot more gpu memory and is thus much more expensive, complicated and out of reach of most people. To fine tune a >10B model you still need multiple A100 / H100. Let’s hope that changes with quantized fine tuning, forward pass only etc.

What is the $ cost of a fine tune though? $500?

Re: OpenLLM

#23

Post by Havoc »

This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead

Re: OpenLLM

#24

Post by YetAnotherNick »

mjburgess wrote: post #6

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression rati…

> 100s PB

Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire training set of ChatGPT is almost surely less than a TB or two with compression which is 5 orders of magnitude lower. I believe that such efficient representation of text is one of the biggest reason why text models are working so well but image understanding models are not.

Re: OpenLLM

#25

Post by bjornsing »

mjburgess wrote: post #6

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression rati…

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

Re: OpenLLM

#26

Post by user432678 »

aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Thanks for the great project! Any chance, your team might consider more open platform than Discord for posting updates? I personally find Discord hard to use, and there’s no way to have sensible subscription (like RSS). Discord is usually muted.

Re: OpenLLM

#27

Post by mjburgess »

YetAnotherNick wrote: post #24
mjburgess wrote: post #6

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression rati…

> 100s PB Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire training set of ChatGPT is almost surely less than a TB or two with compression which is 5 orders of magnitude lower. I believe that such efficient representation of text is one of the biggest reason why text models are working so well but image understanding mode…

Perhaps the process was an initial c. 1PB then sampled down to the TBs.

Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB.

This is one of the illusions of text-generative NNs: a 0.5TB weight set is basically enough to store every book. Making claims to "out-sample generalisation" extremely suspicious, and indeed, fairly obviously false.

eg., Ask ChatGPT to write tic-tak-toe in javascript and you get a working game; as it to write duck-hunt and you dont.

Re: OpenLLM

#28

Post by logicchains »

Havoc wrote: post #23

This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead

In my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.

Re: OpenLLM

#29

Post by nologic01 »

user432678 wrote: post #26
aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

Thanks for the great project! Any chance, your team might consider more open platform than Discord for posting updates? I personally find Discord hard to use, and there’s no way to have sensible subscription (like RSS). Discord is usually muted.

s/rd/urse/g

Re: OpenLLM

#30

Post by manojlds »

aarnphm wrote: post #12

Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!

How can we stay tuned if we can't do tuning? :P
Post reply on HN