Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-v3.1

api-docs.deepseek.com

Post reply on HN
Print view View original

131–140 of 273 posts

Re: DeepSeek-v3.1

#131

Post by lucideer »

mFixman wrote: post #113
danielhanchen wrote: post #88

Earlier quoted context omitted.

As an update, I pushed https://github.com/unslothai/unsloth-zoo/commit/ae675a0a2d20... (1) Removed and disabled sudo (2) Installing via apt-get will ask user's input() for permission (3) Added an error if failed llama.cpp and provides instructions to manual compile llama.cpp

Maybe it's a personal preference, but I don't want external programs to ever touch my package manager, even with permission. Besides, this will fail loudly for systems that don't use `apt-get`. I would just ask the user to install the package, and _maybe_ show the command line to install it (but never run it).

I don't think this should be a personal preference, I think it should be a standard*.

That said, it does at least seem like these recent changes are a large step in the right direction.

---

* in terms of what the standard approach should be, we live in an imperfect world and package management has been done "wrong" in many ecosystems, but in an ideal world I think the "correct" solution here should be:

(1) If it's an end user tool it should be a self contained binary or it should be a system package installed via the package manager (which will manage any ancillary dependencies for you)

(2) If it's a dev tool (which, if you're cloning a cpp repo & building binaries, it is), it should not touch anything systemwide. Whatsoever.

This often results in a README with manual instructions to install deps, but there are many good automated ways to approach this. E.g. for CPP this is a solved problem with Conan Profiles. However that might incur significant maintenace overhead for the Unsloth guys if it's not something the ggml guys support. A dockerised build is another potential option here, though that would still require the user to have some kind of container engine installed, so still not 100% ideal.

Re: DeepSeek-v3.1

#132

Post by mkl »

danielhanchen wrote: post #85
pxc wrote: post #81

Earlier quoted context omitted.

It seems Unsloth is useful and popular, and you seem responsive and helpful. I'd be down to try to improve this and maybe package Unsloth for Nix as well, if you're up for reviewing and answering questions; seems fun. Imo it's best to just depend on the required fork of llama.cpp at build time (or not) according to some configuration. Installing things at runtime is nuts (especially if it means modifying the existing…

Thanks for the suggestions! Apologies again I'm pretty bad at packaging, so hence the current setup. 1. So I added a `check_llama_cpp` which checks if llama.cpp does exist and it'll use the prebuilt one https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_z... 2. Yes I like the idea of determining distro 3. Agreed on bailing - I was also thinking if doing a Python input() with a 30 second waiting period for apt…

It shouldn't install any packages itself. Just print out a message about the missing packages and your guess of the command to install them, then exit. That way users can run the command themselves if it's appropriate or add the packages to their container build or whatever. People set up machines in a lot of different ways, and automatically installing things is going to mess that up.

Re: DeepSeek-v3.1

#133

Post by ffsm8 »

danielhanchen wrote: post #127
rfoo wrote: post #126

Earlier quoted context omitted.

IMO the correct thing to do to make these people happy, while being sane, is - do not build llama.cpp on their system. Instead, bundle a portable llama.cpp binary along with unsloth, so that when they install unsloth with `pip` (or `uv`) they get it. Some people may prefer using whatever llama.cpp in $PATH, it's okay to support that, though I'd say doing so may lead to more confused noob users spam - they may just ha…

Oh yes I was working on providing binaries together with pip - currently we're relying on pyproject.toml, but once we utilize setup.py (I think), using binaries gets much simpler I'm still working on it, but sadly I'm not a packaging person so progress has been nearly zero :(

I think you misunderstood rfoos suggestion slightly.

From how I interpreted it, he meant you could create a new python package, this would effectively be the binary you need.

In your current package, you could depend on the new one, and through that - pull in the binary.

This would let you easily decouple your package from the binary,too - so it'd be easy to update the binary to latest even without pushing a new version of your original package

I've maintained release pipelines before and handled packaging in a previous job, but I'm not particularly into the python ecosystem, so take this with a grain of salt: an approach would be

Pip Packages :

    * Unsloth: current package, prefers using unsloth-llama, and uses path llama-cpp as fallback (with error msg as final fallback if neither exist, promoting install for unsloth-llama)
    * Unsloth-llama: new package which only bundles the llama cpp binary

Re: DeepSeek-v3.1

#134

Post by Leynos »

xmichael909 wrote: post #40

Seems to hallucinate more than any model I've ever worked with in the past 6 months.

DeepSeek is bad for hallucinations in my experience. I wouldn't trust its output for anything serious without heavy grounding. It's great for fantastical fiction though. It also excels at giving characters "agency".

Re: DeepSeek-v3.1

#135

Post by pshirshov »

danielhanchen wrote: post #68
pshirshov wrote: post #60

Earlier quoted context omitted.

By the way, I'm wondering why unsloth (a goddamn python library) tries to run apt-get with sudo (and fails on my nixos). Like how tf we are supposed to use that?

Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…

It won't work well if you deal with non ubuntu+cuda combination. Better just fail with a reasonable message.

Re: DeepSeek-v3.1

#137

Post by danielhanchen »

ffsm8 wrote: post #133
danielhanchen wrote: post #127

Earlier quoted context omitted.

Oh yes I was working on providing binaries together with pip - currently we're relying on pyproject.toml, but once we utilize setup.py (I think), using binaries gets much simpler I'm still working on it, but sadly I'm not a packaging person so progress has been nearly zero :(

I think you misunderstood rfoos suggestion slightly. From how I interpreted it, he meant you could create a new python package, this would effectively be the binary you need. In your current package, you could depend on the new one, and through that - pull in the binary. This would let you easily decouple your package from the binary,too - so it'd be easy to update the binary to latest even without pushing a new vers…

Oh ok sorry maybe I misunderstood sorry! I actually found my partial work I did for precompiled binaries! https://huggingface.co/datasets/unsloth/precompiled_llama_cp...

I was trying to see if I could pre-compile some llama.cpp binaries then save them as a zip file (I'm a noob sorry) - but I definitely need to investigate further on how to do python pip binaries

Re: DeepSeek-v3.1

#138

Post by danielhanchen »

pshirshov wrote: post #135
danielhanchen wrote: post #68

Earlier quoted context omitted.

Oh hey I'm assuming this is for conversion to GGUF after a finetune? If you need to quantize to GGUF Q4_K_M, we have to compile llama.cpp, hence apt-get and compiling llama.cpp within a Python shell. There is a way to convert to Q8_0, BF16, F16 without compiling llama.cpp, and it's enabled if you use `FastModel` and not on `FastLanguageModel` Essentially I try to do `sudo apt-get` if it fails then `apt-get` and if al…

It won't work well if you deal with non ubuntu+cuda combination. Better just fail with a reasonable message.

For now I'm re-directly people to our docs https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-...

But I'm working on more cross platform docs as well!

Re: DeepSeek-v3.1

#139

Post by danielhanchen »

mkl wrote: post #132
danielhanchen wrote: post #85

Earlier quoted context omitted.

Thanks for the suggestions! Apologies again I'm pretty bad at packaging, so hence the current setup. 1. So I added a `check_llama_cpp` which checks if llama.cpp does exist and it'll use the prebuilt one https://github.com/unslothai/unsloth-zoo/blob/main/unsloth_z... 2. Yes I like the idea of determining distro 3. Agreed on bailing - I was also thinking if doing a Python input() with a 30 second waiting period for apt…

It shouldn't install any packages itself. Just print out a message about the missing packages and your guess of the command to install them, then exit. That way users can run the command themselves if it's appropriate or add the packages to their container build or whatever. People set up machines in a lot of different ways, and automatically installing things is going to mess that up.

Hmmm so I should get rid of the asking / permissions message?

Re: DeepSeek-v3.1

#140

Post by danielhanchen »

lucideer wrote: post #131
mFixman wrote: post #113

Earlier quoted context omitted.

Maybe it's a personal preference, but I don't want external programs to ever touch my package manager, even with permission. Besides, this will fail loudly for systems that don't use `apt-get`. I would just ask the user to install the package, and _maybe_ show the command line to install it (but never run it).

I don't think this should be a personal preference, I think it should be a standard*. That said, it does at least seem like these recent changes are a large step in the right direction. --- * in terms of what the standard approach should be, we live in an imperfect world and package management has been done "wrong" in many ecosystems, but in an ideal world I think the "correct" solution here should be: (1) If it's an…

I would like to be in (1) but I'm not a packaging person so I'll need to investigate more :(

(2) I might make the message on installing llama.cpp maybe more informative - ie instead of re-directing people to the docs on manual compilation ie https://docs.unsloth.ai/basics/troubleshooting-and-faqs#how-..., I might actually print out a longer message in the Python cell entirely

Yes we're working on Docker! https://hub.docker.com/r/unsloth/unsloth

Post reply on HN