Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

81–90 of 175 posts

Re: OpenLLM

#81

Post by falcor84 »

pmontra wrote: post #63
cinntaile wrote: post #47

Earlier quoted context omitted.

Because it's easy, free and it just works. Very few people actually care about indexing the conversations.

So all knowledge is lost and questions have to be asked and answered again and again?

Isn't there something really nice about it though? It seems to me that most every community gradually evolves into one where every new message from a new-ish member is answered by something like "Duplicate, please search first!". And this in turn makes those newcomers either go away, become passive lurkers, or become part of the "hive-mind" (as only likeminded questions get answered).

On the other hand, if people have to actually converse to get an answer to their questions (like back in the real world), newcomers can more rapidly become part of the community, and help make it more diverse.

Re: OpenLLM

#82

Post by Zak »

falcor84 wrote: post #81
pmontra wrote: post #63

Earlier quoted context omitted.

So all knowledge is lost and questions have to be asked and answered again and again?

Isn't there something really nice about it though? It seems to me that most every community gradually evolves into one where every new message from a new-ish member is answered by something like "Duplicate, please search first!". And this in turn makes those newcomers either go away, become passive lurkers, or become part of the "hive-mind" (as only likeminded questions get answered). On the other hand, if people hav…

I just recently saw a post where someone said something similar about Reddit versus traditional forums.

There's a balance between engaging with new members and not turning it into a time sink for older members. This is probably a good use case for LLMs.

Re: OpenLLM

#83

Post by DrNosferatu »

The project seems great!

However, newcomers (like me) are pretty blind about minimum system requirements.

Could you please add them to the models list?

For example: what minimum hardware do I need to run Falcon-40b?

PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

Re: OpenLLM

#84

Post by hyperhopper »

decide1000 wrote: post #51
belugacat wrote: post #48

Earlier quoted context omitted.

short answer: because it's one of the options with least friction to get running a lot of people who are into tech stuff already have a discord account making joining the community a one click process, the instant nature of it seems to appeal to younger users more than async forums, it's a fairly mature platform so it has a bunch of moderation/customization/integration features you might want, etc. > are discord conv…

Isn't it a generation thing? If I had the choice everyone would be on IRC still.

IRC doesn't even have history, one of the most basic requirements for a modern rudimentary chat app. It's ridiculous to suggest using it in 2023 when it doesn't have features a freshman homework assignment chat app has.

Re: OpenLLM

#85

Post by Zak »

decide1000 wrote: post #51
belugacat wrote: post #48

Earlier quoted context omitted.

short answer: because it's one of the options with least friction to get running a lot of people who are into tech stuff already have a discord account making joining the community a one click process, the instant nature of it seems to appeal to younger users more than async forums, it's a fairly mature platform so it has a bunch of moderation/customization/integration features you might want, etc. > are discord conv…

Isn't it a generation thing? If I had the choice everyone would be on IRC still.

I've used IRC for a long time and still do, but I do think Discord has a nicer UX for most use cases. In particular, building communities around clusters of channels ("servers") and support for rich media (yes, some old people might call that a downside) increase the appeal for most people. It's also a lot more work to have a persistent connection on IRC (bouncers).

My main problem with Discord is that it's someone else's centralized, for-profit company and has no apparent barriers to enshittification[0]. As Reddit recently demonstrated, it's probably a mistake to build communities on top of something like that.

Matrix is a good candidate for a modern successor to IRC. It's not quite as slick a UX as Discord, but it addresses the main advantages Discord has over IRC.

[0] https://pluralistic.net/2023/01/21/potemkin-ai/#hey-guys

Re: OpenLLM

#87

Post by leetharris »

DrNosferatu wrote: post #83

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

For falcon 40b you probably need an A100 40gb or so.

Every model is drastically different.

If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

Re: OpenLLM

#88

Post by moneywoes »

visarga wrote: post #2

Fine-tuning is the most important part, but it is under intense research today, things change fast. I hope they can streamline this process because these smaller models can only compete with big models when they are fine-tuned.

Is there a good checklist or framework for fine tuning vs using a vector db to increase the context size

Re: OpenLLM

#90

Post by philoko »

DrNosferatu wrote: post #83

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.
Post reply on HN