Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RamaLama

github.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: RamaLama

#13

Post by Y_Y »

So it's a replacement for Ollama?

The killer features of Ollama for me right now are the nice library of quantized models and the ability to automatically start and stop serving models in response to incoming requests and timeouts. The first send to be solved by reusing the Ollama models, but I can't see if the service is possible from my cursory look.

Re: RamaLama

#14

Post by jniles »

glitchc wrote: post #11

Great, finally an alternative to ollama's convenience.

It sounds like this project isn't addressing the user convenience aspect of ollama, but rather the developer convenience.

Hopefully both will be easy for users to play around with, but RamaLama should be easier to get your PR merged as a developer and swap out different registries. Vendor lock-in is rarely a good thing in the world of open source.

Re: RamaLama

#15

Post by ericyd »

eigenvalue wrote: post #4

This is the point of it: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

I wish this were on the readme. Or if it already is, I wish it were significantly higher up.

Re: RamaLama

#16

Post by maxamillion »

Y_Y wrote: post #13

So it's a replacement for Ollama? The killer features of Ollama for me right now are the nice library of quantized models and the ability to automatically start and stop serving models in response to incoming requests and timeouts. The first send to be solved by reusing the Ollama models, but I can't see if the service is possible from my cursory look.

ramalama can just pull (almost) any arbitrary model off huggingface and run it ... you're not limited to just what ollama has repackaged into their non-standard format

Re: RamaLama

#17

Post by maxamillion »

esafak wrote: post #2

Is this useful? Can someone help me see the value add here?

ramalama can just pull (almost) any arbitrary model off huggingface and run it ... you're not limited to just what ollama has repackaged into their non-standard format

Re: RamaLama

#18

Post by ggerganov »

mckirk wrote: post #9

This looks great! While we're at it, is there already some kind of standardized local storage location/scheme for LLM models? If not, this project could potentially be a great place to set an example that others can follow, if they want. I've been playing with different runtimes (Ollama, vLLM) the last days, and I really would have appreciated better interoperability in terms of shared model storage, instead of every…

The llama.cpp tools and examples download the models by default to a OS-specific cache folder [0]. We try to follow the HF standard (as discussed in the linked thread), though the layout of the llama.cpp cache is not the same atm. Not sure about the plans for RamaLama, but it might be something worth to consider.

[0] https://github.com/ggerganov/llama.cpp/issues/7252

Re: RamaLama

#19

Post by mohsen1 »

eigenvalue wrote: post #4

This is the point of it: https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...

I see! Now I understand why I need to create those useless `Modelfile` files...

I'm glad there is a more open source alternative to Ollama now.

Re: RamaLama

#20

Post by baron-bourbon »

Does this provide a Ollama compatible API endpoint? I've got at least one other project running that only supports Ollama's API or OpenAI's hosted solution (ie. the API endpoint isn't configurable to use llama.cpp and friends)
Post reply on HN