Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

51–60 of 175 posts

Re: OpenLLM

#51

Post by decide1000 »

belugacat wrote: post #48
bsaul wrote: post #45

Earlier quoted context omitted.

Side question : why are people working on open source project communicating through discord a lot noawadays ? are discord conversations persisted and indexed on search engines ?

short answer: because it's one of the options with least friction to get running a lot of people who are into tech stuff already have a discord account making joining the community a one click process, the instant nature of it seems to appeal to younger users more than async forums, it's a fairly mature platform so it has a bunch of moderation/customization/integration features you might want, etc. > are discord conv…

Isn't it a generation thing? If I had the choice everyone would be on IRC still.

Re: OpenLLM

#52

Post by treprinum »

Havoc wrote: post #23

This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead

Unfortunately, for transformer-based LLMs the magic starts only when they are trained by more that 10^22 TFlops (preferably 10^24) so smaller models might not cut it even for fine-tuned tasks.

Re: OpenLLM

#53

Post by simonw »

decide1000 wrote: post #51
belugacat wrote: post #48

Earlier quoted context omitted.

short answer: because it's one of the options with least friction to get running a lot of people who are into tech stuff already have a discord account making joining the community a one click process, the instant nature of it seems to appeal to younger users more than async forums, it's a fairly mature platform so it has a bunch of moderation/customization/integration features you might want, etc. > are discord conv…

Isn't it a generation thing? If I had the choice everyone would be on IRC still.

Fue problem with IRC is that it's crucial to have really robust read state synchronization across desktop and mobile these days.

Slack was the first to really get that right, and Discord effectively emulated them and made it available for free.

IRC users could get there with bouncers, but those were always a lot harder to get going with.

Re: OpenLLM

#55

Post by piker »

bjornsing wrote: post #25
mjburgess wrote: post #6

Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression rati…

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.

Re: OpenLLM

#56

Post by d4rkp4ttern »

Suppose I’ve written code that calls the OpenAI API. Is there some library that helps me easily switch to a local/other LLM. I.e a library that (ideally) provides the same OpenAI interface for several models, or if not then at least the same interface.

Re: OpenLLM

#57

Post by visarga »

treprinum wrote: post #52
Havoc wrote: post #23

This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead

Unfortunately, for transformer-based LLMs the magic starts only when they are trained by more that 10^22 TFlops (preferably 10^24) so smaller models might not cut it even for fine-tuned tasks.

At medium size (13B) Microsoft Orca demonstrated you can trade off size with larger fine-tuning dataset.

Re: OpenLLM

#58

Post by monkeydust »

d4rkp4ttern wrote: post #56

Suppose I’ve written code that calls the OpenAI API. Is there some library that helps me easily switch to a local/other LLM. I.e a library that (ideally) provides the same OpenAI interface for several models, or if not then at least the same interface.

Langchain might be what you need.

Re: OpenLLM

#59

Post by visarga »

uniqueuid wrote: post #4
visarga wrote: post #2

Fine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.

Yes but fine tuning requires a lot more gpu memory and is thus much more expensive, complicated and out of reach of most people. To fine tune a >10B model you still need multiple A100 / H100. Let’s hope that changes with quantized fine tuning, forward pass only etc.

You can fine-tune medium models 3..60B on a single GPU with QLoRA

Re: OpenLLM

#60

Post by chrisan »

decide1000 wrote: post #51
belugacat wrote: post #48

Earlier quoted context omitted.

short answer: because it's one of the options with least friction to get running a lot of people who are into tech stuff already have a discord account making joining the community a one click process, the instant nature of it seems to appeal to younger users more than async forums, it's a fairly mature platform so it has a bunch of moderation/customization/integration features you might want, etc. > are discord conv…

Isn't it a generation thing? If I had the choice everyone would be on IRC still.

Practically all of my friends grew up with IRC, we are in our late 30s, 40s, early 50s.

We might reminisce about irc but we all prefer discord.

Even the searchability of indexed irc has been surpassed by other knowledge sites. It would have to be something extremely niche these days where the only source of info is in an irc chat log

Post reply on HN