Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenLLM

github.com

Post reply on HN
Print view View original

91–100 of 175 posts

Re: OpenLLM

#91

Post by luckystarr »

DrNosferatu wrote: post #83

The project seems great! However, newcomers (like me) are pretty blind about minimum system requirements. Could you please add them to the models list? For example: what minimum hardware do I need to run Falcon-40b? PS: If you only have a few setups "known to work" (or just one), listing that would be helpful too.

From the HuggingFace page:

> You will need at least 85-100GB of memory to swiftly run inference with Falcon-40B.

So it may be possible with less (swap around method), though not as efficiently and also slower.

Re: OpenLLM

#92

Post by andsoitis »

micromacrofoot wrote: post #89
geysersam wrote: post #77

Earlier quoted context omitted.

The issue is it's not indexed by Google

have you used google lately? might as well not be indexed with all the seo spam you get as top results

> have you used google lately? might as well not be indexed with all the seo spam you get as top results

I just googled "how to use openllm" as an example to test your thesis, and the results look very relevant to me.

https://www.google.com/search?client=safari&rls=en&q=how+to+...

Re: OpenLLM

#94

Post by xpe »

piker wrote: post #55
bjornsing wrote: post #25

Earlier quoted context omitted.

I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.

Bytes does imply a level of precision, however, which affects the mathematical meaning. Perhaps there’s a metric that captures both.

Allocating more bits for each parameter increases precision, by definition. But that doesn’t come for free.* So it is useful to optimize network performance for a given number of total parameter bytes.

I haven’t done a recent literature review, but my hand-wavy guessplanation is that a NN (as a whole) can adapt to relatively low precision parameters. Up to a point.

* In general. Given actual hardware designs, there are places where you have slack in the system. So adding some extra parameters, e.g. to fully utilize a GPU’s core’s threads (e.g. 32), might actually cost you nothing.

Re: OpenLLM

#95

Post by DragonStrength »

andsoitis wrote: post #92
micromacrofoot wrote: post #89

Earlier quoted context omitted.

have you used google lately? might as well not be indexed with all the seo spam you get as top results

> have you used google lately? might as well not be indexed with all the seo spam you get as top results I just googled "how to use openllm" as an example to test your thesis, and the results look very relevant to me. https://www.google.com/search?client=safari&rls=en&q=how+to+...

You might want to glance again because all of those results are for a different product.

Re: OpenLLM

#96

Post by sputr »

Question: for someone that wants to play around with self-hosted text generation but has a crap laptop – are there any hosting providers (like a VPS) where I can run open source models?

Re: OpenLLM

#97

Post by andsoitis »

DragonStrength wrote: post #95
andsoitis wrote: post #92

Earlier quoted context omitted.

> have you used google lately? might as well not be indexed with all the seo spam you get as top results I just googled "how to use openllm" as an example to test your thesis, and the results look very relevant to me. https://www.google.com/search?client=safari&rls=en&q=how+to+...

You might want to glance again because all of those results are for a different product.

Top of the results page says:

"Showing results for how to use openlm

Search instead for how to use openllm"

Re: OpenLLM

#98

Post by Ajedi32 »

I like the idea of having a standard API for interacting with LLMs over the network. Many models need to run on beefy hardware and would benefit from offloading to a remote (possibly self-hosted) server, and I think makes logical sense to separate the code for running LLMs from the UI for accessing them.

Re: OpenLLM

#99

Post by tomschwiha »

sputr wrote: post #96

Question: for someone that wants to play around with self-hosted text generation but has a crap laptop – are there any hosting providers (like a VPS) where I can run open source models?

You can rent pretty easy a server with a GPU using runpod, vast.ai or datacrunch. Or maybe even use something Like Google Colab.

Re: OpenLLM

#100

Post by ekianjo »

cinntaile wrote: post #47
bsaul wrote: post #45

Earlier quoted context omitted.

Side question : why are people working on open source project communicating through discord a lot noawadays ? are discord conversations persisted and indexed on search engines ?

Because it's easy, free and it just works. Very few people actually care about indexing the conversations.

indexing conversations is secondary for gaming but primary for FOSS projects and Discord sucks at that. its like wiping your ass with a fork.
Post reply on HN