Revert for jart’s llama.cpp MMAP miracles
1–10 of 96 posts
Re: Revert for jart’s llama.cpp MMAP miracles
#2Can someone in the know describe what the hullaballoo is about?
Seems like ego-driven optimization that breaks compatibility?
Re: Revert for jart’s llama.cpp MMAP miracles
#3Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?
What's ego-driven optimization?
Re: Revert for jart’s llama.cpp MMAP miracles
#4Why reverting it instead of adding upon it? The author of the revert could easily start working on reintroducing the previous format behind a flag. AFAIK llama.cpp is not even v1 yet. I see this revert PR as unnecessary
Re: Revert for jart’s llama.cpp MMAP miracles
#5I really dislike giving HN exposure to this kind of issue; it only brings us the forbidden pleasure of voyeurism while not helping the maintainers & contributors in the slightest – and can even crystallize conflicts while we eat popcorn.
Let us let them take their time, wash their dirty laundry among themselves, and take the time they need to go forward on the project.
Re: Revert for jart’s llama.cpp MMAP miracles
#6Why reverting it instead of adding upon it? The author of the revert could easily start working on reintroducing the previous format behind a flag. AFAIK llama.cpp is not even v1 yet. I see this revert PR as unnecessary
Feature flags are great but they should not be used as a crutch to leave unfinished or non-working code in the codebase. IMO they should be used to rapidly pull back the change before the slower revert can take place.
Re: Revert for jart’s llama.cpp MMAP miracles
#7Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?
The tl;dr as I understand it is that jart had a misunderstanding of how what was actually happening and the benefits of the map optimization… the claims of actually being able to shrink the model size from 20GB > 6GB were just completely false, and while there was a model loading time improvement, actual memory required and used did not change.
A number of people saw this and said that making a breaking change to the repo that a lot of people are using and have forked for other models was a bad idea, thus this new PR.
Re: Revert for jart’s llama.cpp MMAP miracles
#8Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?
Is not ego-driven, the requirements for running any model dropped by more than half, you can know run the largest model, 30b, on domestic over the shelf computers. The change is very welcome in the community
Re: Revert for jart’s llama.cpp MMAP miracles
#9Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?
What's ego-driven optimization?
Pushing optimizations for personal glory that look good with some benchmarks but may have unthought of or hidden regressions on other aspects of the code/user base.
On the same hand this may be an ego driven revert. from reading the bug it seems like some people might be salty that jart gets a lot of publicity for a few changes where other people which bigger contributions to the project don't get.
Re: Revert for jart’s llama.cpp MMAP miracles
#10Can someone in the know describe what the hullaballoo is about? Seems like ego-driven optimization that breaks compatibility?
mmap breaks previous (not guaranteed) compatibility and (few) people demand option to turn it off by reverting all of the commits and throwing hands up by demanding more testing, documenting, and more optional arguments.
While I agree with some of the premises, it’s up to ggerganov in the end. If he wants this to be the default, so be it. Throwing a (polite) tantrum because you should please all, while not offering to do any of the work, is entitled demand.