Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

161–170 of 175 posts

Re: OpenLLM

#161

Post by micromacrofoot »

andsoitis wrote: post #92
micromacrofoot wrote: post #89

Earlier quoted context omitted.

have you used google lately? might as well not be indexed with all the seo spam you get as top results

> have you used google lately? might as well not be indexed with all the seo spam you get as top results I just googled "how to use openllm" as an example to test your thesis, and the results look very relevant to me. https://www.google.com/search?client=safari&rls=en&q=how+to+...

when I click this google gives me results for “how to use openlm” a commercial product, they literally change your search term if there’s a product that fits

Re: OpenLLM

#162

Post by thecal »

asylteltine wrote: post #93

What does it mean to “serve a model”? Where exactly does the request go and how does it interface with the model?

Think of the model as a gigantic compiled binary where you send in strings in a certain format and get back a response. This is a web API wrapper for that so you only need an HTTP client instead of having to run something like llama.cpp yourself.

Re: OpenLLM

#163

Post by throwaway675309 »

falcor84 wrote: post #116
scubbo wrote: post #106

Earlier quoted context omitted.

The best of both worlds - a friendly community that welcomes newbies, with a searchable archive - is possible. Limiting to only chat-based support means that support is bottle-necked by the folks who are available and engaged at the time of the question, and that knowledge will "drop out" of the community as people forget it.

Apologies for my skepticism, but is it just "possible", or do you actually have an example of a long-lived community that remained fully welcoming to newbies while utilizing a searchable archive? In any case, I'm not arguing that it's impossible, but rather that the more comprehensive the archive, the less welcoming the community would tend to be, all other things being equal. To take it to the extreme, I'll posit th…

Hard disagree. If anything you'll find that the most knowledgeable members get burned out answering the same questions over and over again, so they begin to simplify their answers until they just become copy pasta.

You can still have channels open to welcoming new people while at the same time having a large archive of answered questions so that over time a reservoir gets built.

Saying that the same questions getting asked over and over again by new people is somehow a more welcoming community, is like saying that there's any meaningful interaction happening when two people say "What's up?" followed by the response "not much". It's a handshake protocol equivalent without actual depth.

Re: OpenLLM

#164

Post by scubbo »

falcor84 wrote: post #116
scubbo wrote: post #106

Earlier quoted context omitted.

The best of both worlds - a friendly community that welcomes newbies, with a searchable archive - is possible. Limiting to only chat-based support means that support is bottle-necked by the folks who are available and engaged at the time of the question, and that knowledge will "drop out" of the community as people forget it.

Apologies for my skepticism, but is it just "possible", or do you actually have an example of a long-lived community that remained fully welcoming to newbies while utilizing a searchable archive? In any case, I'm not arguing that it's impossible, but rather that the more comprehensive the archive, the less welcoming the community would tend to be, all other things being equal. To take it to the extreme, I'll posit th…

A very reasonable question, and I'll admit that I'm not deeply-entrenched in enough technical communities to give you an actual example. But yeah, intuitively I do agree with the sibling commenter - a well-curated archive is a tool of technology which allows skilled respondents to preserve their time and energy for new and interesting questions. A pointer to search is not necessarily dismissive - there is a world of difference between the following _technically_ equivalent responses:

* FFS, read the fuckin' archive noob and stop wasting our time

* Hey there, thanks for asking! This is actually a pretty common question, and we have guides written up for just this case. Try entering some of your search terms here [link], and come back with a follow-up question if that doesn't help you!

But yes, in fairness, I'll certainly agree that a community which _chooses_ to respond as the former will stagnate and die.

Re: OpenLLM

#165

Post by shmolf »

khimaros wrote: post #114
user432678 wrote: post #26

Earlier quoted context omitted.

Thanks for the great project! Any chance, your team might consider more open platform than Discord for posting updates? I personally find Discord hard to use, and there’s no way to have sensible subscription (like RSS). Discord is usually muted.

plugging the open source and self hostable https://revolt.chat which i've found to have great UX and be very performant compared to discord.

I'm liking revolt. Thanks for the suggestion.

Re: OpenLLM

#166

Post by darkwater »

MatthiasPortzel wrote: post #142
darkwater wrote: post #78

Earlier quoted context omitted.

> That didn't stop IRC being popular in the 1990s. IRC chats, especially in opensource projects channels, could and would be archived, published over the web and indexed by search engines.

In my experience, I don’t think I’ve ever seen an IRC log in a search result. #haskell on Libra is publicly logged, but I couldn’t get Google to return a quoted phrase from a message a few weeks ago. Many people on IRC don’t enjoy being in logged channels. I’ve also heard that there are GDPR implications to publicly logging people’s messages without their consent. Discussion of the difficulty and downsides of IRC log…

No, it's not. If you work on an opensource / open development project, it totally makes sense to avoid walled gardens for the community chat/forum (a few years ago it was public Slack instance, nowadays it's Discord servers).

Re: OpenLLM

#167

Post by falcor84 »

throwaway675309 wrote: post #163
falcor84 wrote: post #116

Earlier quoted context omitted.

Apologies for my skepticism, but is it just "possible", or do you actually have an example of a long-lived community that remained fully welcoming to newbies while utilizing a searchable archive? In any case, I'm not arguing that it's impossible, but rather that the more comprehensive the archive, the less welcoming the community would tend to be, all other things being equal. To take it to the extreme, I'll posit th…

Hard disagree. If anything you'll find that the most knowledgeable members get burned out answering the same questions over and over again, so they begin to simplify their answers until they just become copy pasta. You can still have channels open to welcoming new people while at the same time having a large archive of answered questions so that over time a reservoir gets built. Saying that the same questions getting…

    I see friends shaking hands
    Saying, "How do you do?"
    They're really saying
    I love you.

Re: OpenLLM

#168

Post by rmbyrro »

ben_w wrote: post #80
sorenjan wrote: post #74

What kind of hardware do I need to run something small scale (1 user concurrently) and get reasonable result? Are we talking about Raspberry Pi, Core i5, Geforce 4090?

I think that strongly depends on what you count as "reasonable": smaller models take less memory, so there's a trade-off between quality and speed depending on if you can fit it all in graphics VRAM, or system RAM, or virtual memory… Just keep in mind the speed differences between the types of memory. If you've got a 170bn 4-bit parameter model, and you're on virtual memory on a 400 Mbps port, a naive calculation say…

Would you mind sharing the calculation memory that reaches at 28 minutes (!) per token?

Re: OpenLLM

#169

Post by ben_w »

rmbyrro wrote: post #168
ben_w wrote: post #80

Earlier quoted context omitted.

I think that strongly depends on what you count as "reasonable": smaller models take less memory, so there's a trade-off between quality and speed depending on if you can fit it all in graphics VRAM, or system RAM, or virtual memory… Just keep in mind the speed differences between the types of memory. If you've got a 170bn 4-bit parameter model, and you're on virtual memory on a 400 Mbps port, a naive calculation say…

Would you mind sharing the calculation memory that reaches at 28 minutes (!) per token?

If your model is 170bn 4-bit parameters, you have 85 Gbytes that has to be loaded into the CPU or GPU; if those parameters are on the other side of a 400 Mbps port, that takes 85 Gbyte / 400 Mbps = 1700 seconds = 28 minutes and 20 seconds.

If you don't have sufficient real RAM or VRAM, the entire model has to be re-loaded for each step of the process.

Assuming no looping (looping makes it longer) and no compartmentalisation in the network structure (if you can make it so an entire fragment might be not-activated, you have the possibility of skipping loading that section; I've not heard of any architecture that does this, it would be analogous to dark silicon or to humans not using every brain cell at the same time (outside of seizures)).

Re: OpenLLM

#170

Post by utbabya »

joot82 wrote: post #131
pmoriarty wrote: post #72

Earlier quoted context omitted.

Discord is a black hole where information goes to die. Its search and scrollback is awful. It's awful at being an archive, as finding anything that was asked more than a day or two ago is impractical. To use Discord in good faith and with open eyes, you have to prioritize communication in the present, and give up hope of archiving anything that was said for people who might need the information in the future.

Discord is just a rich IRC replacement. You can log and search in IRC too but nobody seriously tries to archive information for research later. And big difference is it's all closed and operated by one entity that can change conditions at will. Don't even try to use it for anything else than real time chat.

Greping through IRC logs has a 10x better UX
Post reply on HN