211–220 of 234 posts
Earlier quoted context omitted.
The compilation toolchain may be reliable but then you get kernel panics at runtime.
I've heard geohot is upset about that. I haven't tortured any of my AMD cards enough to run into that issue yet. Do you know how to make it happen?
Earlier quoted context omitted.
> It's not meant for production deployments. I am probably not the demographics you expect. I don’t do “production” in that sense, but I have ollama running quite often when I am working, as I use it for RAG and as a fancy knowledge extraction engine. It is incredibly useful: - I can test a lot of models by just pulling them (very useful as progress is very fast), - using their command line is trivial, - the fact tha…
> I use it for RAG and as a fancy knowledge extraction engine Curious, can you share more details about your usecase?
I have a setup that takes pdfs and does some OCR and layout detection with Amazon, and then bunch them with some internal reports. Then, I have a pipeline to write summaries of each document and another one to slice them into chunks, get embeddings and set up a vector store for a RAG chat bot. At the moment it’s using Mixtral and the command line. But I like being able to swap LLMs to experiments with different models and quantisation without hassle, and I more or less plan to set this up on a remote server to free some resources on my workstation so the web UI could come in handy. Running this locally is a must for confidentiality reasons. I’d like to get rid of Textract as well, but unfortunately I haven’t found a solution that’s even close. Tesseract in particular was very disappointing.
Earlier quoted context omitted.
It should be fairly obvious that one can find alternative models and use them in the above command too. Look, I’m not arguing that a prebuilt binary that handles model downloading has no value over a source build and manually pulling down gguf files. I just want to dispel some of the mystery. Local LLM execution doesn’t require some mysterious voodoo that can only be done by installing and running a server runtime. I…
There are 5 commands in that README two comments up, 4 can reasonably fail (I'll give cd high marks for reliability). `make` especially is a minefield and usually involves a half-hour of searching the internet and figuring out which dependencies are a problem today. And that is all assuming someone is comfortable with compiled languages. I'd hazard most devs these days are from JS land and don't know how to debug mak…
Earlier quoted context omitted.
The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like: git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp make wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128
This will likely build a version without GPU acceleration, I think?
Earlier quoted context omitted.
I'm curious about both: - what's special about the memory allocation, and how might it help me? - what are you now using instead of ollama?
Ollama does a nice job of looking at how much VRAM the card has and tuning the number of gpu layers offloaded. Before that, I mainly just had to guess. It's still a heuristic, but I thought that was neat. I'm mainly just using llama.cpp as a native library now, mainly for the direct access to more of llama's data structures, and because I have a sort of unique sampler setup.
Earlier quoted context omitted.
Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt. And they could have very, very, easily, there's a server, sitting right there. They chose not to. That's fine. But it's a choice. The rest I chalk it up to inexperience and being busy.
OP never said "contributions to llama.cpp", the comment just said just "contributions" which I read as "own developments".
Earlier quoted context omitted.
Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt. And they could have very, very, easily, there's a server, sitting right there. They chose not to. That's fine. But it's a choice. The rest I chalk it up to inexperience and being busy.
OP never said "contributions to llama.cpp", the comment just said just "contributions" which I read as "own developments".
Earlier quoted context omitted.
Not angry. Having a discussion :-). It just amazes me how the HN crowd is more than happy with just trying out a model on their machine and calling it a day and not seeing the real picture ahead. Let ignore perf concerns for a moment. Let's say I want to run it on a shared server in the enterprise network so that any application can make use of it. Each application might want to use a model of their choosing. Ollama…
> Not sure if folks here are realizing this :-) I’m not sure you’re capable of understanding that your needs and requirements are just that, yours.
Earlier quoted context omitted.
OP never said "contributions to llama.cpp", the comment just said just "contributions" which I read as "own developments".
Fair. I didn't want to assume the worst, that it was just rhetorical slight of hand where "providing release packaging around an open source? you should credit it, at some point, somewhere." is implied as ridiculous, like asking llama.cpp to credit Google Brain. (Presumably, the implication is, for transformers / the Attention is All You Need paper)