261–270 of 436 posts
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Earlier quoted context omitted.
Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…
> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back
Having said that, I've deployed two large Django projects on the web with tons of customers and it runs and scales just fine, and it's a DREAM to maintain and develop for than for example Java.. I would go so far as to say the opposite, if you haven't used Python for web deployment you've been missing out! (you lose some efficiency I'm sure but you gain other things)
I might be missing something but I actually couldn't reproduce. I purposefully chose a computer with 16GiB RAM to run the 30B model. Performance was extremely slow, and the process was clearly not CPU-limited, unlike when it's running the 13B model. It's clearly swapping a lot.
> mmap-ed memory pages backed by a file that aren't dirty aren't counted in an process's RSS usage, only kernel page cache. The mmap-ed regions of virtual memory does get counted in VSZ (virtual memory) but that is just virtual and can be larger than RAM+swap.
Earlier quoted context omitted.
Python is the 2nd best language for everything. It doesn’t excel at anything, but anything a software can do, it can be done in Python somehow. So, a great pick when you’ve got no idea where you’re going to, when you’re prototyping, when you don’t care about performance or perfection. I agree that for large scale systems when you already know what you’re doing, Python shows its limits quite soon (and we should add th…
> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back
Maybe you haven't noticed but Lisp is now a tiny niche and most new languages aren't homoiconic either..
Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)? From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data. Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
Earlier quoted context omitted.
It's not the easiest syntax? It's the easiest among most popular languages. It uses the least amount of symbols, parenthesis and braces only for values. Some people don't like the significant whitespace, but that helps readability.
> It uses the least amount of symbols, parenthesis and braces only for values. is there any evidence that this makes it easier? people learn python as beginners because it has a reputation for being easy for beginners I don't see anything about the syntax that makes it inherently easier
I liked the one way of doing most things philosophy, coming off working on a large C++ code base.
"The recent change also means you can run multiple LLaMA ./main processes at the same time, and they'll all share the same memory resources." So this could have a main and multiple sub-worker llm processes possibly collaborating while sharing same memory footprint?
There are other bottlenecks than CPU cores though, it might not be very useful to run multiple in parallel..
https://huggingface.co/Pi3141/alpaca-lora-30B-ggml/tree/main
Earlier quoted context omitted.
who will be entrusted to benefit from this? recall that OpenAI did begin as an open source project. and that they chose to go the capitalist route, despite initially explicitly stating on their site they were a non-profit
OpenAI specifically cited scaling costs as a reason for why they switched their org structure from non-profit to "capped profit"[0]. You could potentially crowdfund this, though I should point out that this was already tried and Kickstarter shut it down. The effort in question, "Unstable Diffusion", was kinda sketchy, promising a model specifically tuned for NSFW work. What you'd want is an organization that's respon…
Does anyone know how/why this change decreases memory consumption (and isn't a bug in the inference code)? From my understanding of the issue, mmap'ing the file is showing that inference is only accessing a fraction of the weight data. Doesn't the forward pass necessitate accessing all the weights and not a fraction of them?
It's not a bug, but it's misreading the htop output as mmap doesn't show up as a resident set size there. The pages are RO and not dirty so it's "on the OS" to count it and the OP had lots of RAM on the computer so the model just resides in his page cache instead.