271–280 of 436 posts
Earlier quoted context omitted.
Where are you going to download that from?
Hey if Meta can see a leak, then a former non-profit like OpenAI definitely can.
Earlier quoted context omitted.
I think the counter on those arguments is that LLM owners want to avoid arguing that the model is a derivative work of the training data. If the LLM is a specific arrangement of the copyrighted works, it's very clearly a derivative work of them
I was not suggesting that an LLM itself consists of an arrangement of the copyrighted works comprising the training data, but that the specific selection of the copyrighted works comprising the training data is part of what differentiates one LLM from another. A strained but useful analogy might be to think of the styles of painting an artist is trained in and/or exposed to prior to creating their own art. Obvious or…
If you ever come up with more hypothetical arguments in favor of NNs being copyrightable, please let me know. Or post them somewhere.
Author here. For additional context, please read https://github.com/ggerganov/llama.cpp/discussions/638#discu... The loading time performance has been a huge win for usability, and folks have been having the most wonderful reactions after using this change. But we don't have a compelling enough theory yet to explain the RAM usage miracle. So please don't get too excited just yet! Yes things are getting more awesome,…
Didn't expect to see two titans today: ggerganov AND jart. Can ya'll slow down you make us mortals look bad :') Seeing such clever use of mmap makes me dread to imagine how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions. Perhaps SWE is dead after all, but LLMs didn't kill it...
Though in practice, in many cases, mmap won't be faster, it can be even slower than open+read.
are these made available somewhere?
Earlier quoted context omitted.
It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.
Yeah, I've been using python for the first time in a while to try out some of the llm stuff and I can't believe how bad the dependency hell is. It's probably particularly bad due to the pace of change in this field. But I spend an hour getting dependencies fixed every time I touch anything. 80% of the Google Collabs I find are just outright broken. I wish there were other viable non python options to try out these th…
ML libraries are particularly bad, most other stuff works well.
Friends don't let friends install pip into /usr/lib.
Earlier quoted context omitted.
> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back
Well for starters, web deployment isn't "everything". Python is the de-facto go-to language for research or general prototyping, where not everyone is a programming wiz keeping track of the latest trendy new compiled language. Not everyone can compile stuff even.. :) Having said that, I've deployed two large Django projects on the web with tons of customers and it runs and scales just fine, and it's a DREAM to mainta…
Earlier quoted context omitted.
> Python is the 2nd best language for everything. Huh? Why? You can barely deploy it to Web. it doesn't scale perfoance wise you can't built robust abstractions The REPL is merely OK You can barely ship working code without containers the syntax is hard to manipulate programmatically Python has inertia but it's holding us back
You have good points but "the syntax is hard to manipulate programmatically"?? Maybe you haven't noticed but Lisp is now a tiny niche and most new languages aren't homoiconic either..
Earlier quoted context omitted.
> It uses the least amount of symbols, parenthesis and braces only for values. is there any evidence that this makes it easier? people learn python as beginners because it has a reputation for being easy for beginners I don't see anything about the syntax that makes it inherently easier
Anecdata, but I learned Python many years ago precisely because I found the syntax was clear. I liked the one way of doing most things philosophy, coming off working on a large C++ code base.
Earlier quoted context omitted.
Is the title misleading here ? 30B quantized requires 19.5 GB, not 6GB; Otherwise severe swapping to disk model original size quantized size (4-bit) 7B 13 GB 3.9 GB 13B 24 GB 7.8 GB 30B 60 GB 19.5 GB 65B 120 GB 38.5 GB
Now it's clear that there was a bug in the measurement. The author used a machine with lots of RAM, so I guess most of us are still stuck with quantized 13B. Still, the improvement hopefully translates, and I hope that 30B will run with 3 bit quantization in a few days.