Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What AI assistants are already bundled for Linux?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Ask HN: What AI assistants are already bundled for Linux?

#31

Post by diggan »

zajio1am wrote: post #29
diggan wrote: post #26

Earlier quoted context omitted.

> One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and Huh? That's not at all true. It's only using processing power (CPU) while it actually generates text, otherwise it sits and wait. Although yes it occupies memory (RAM or VRAM) if you don't unload it, but you can configure it to startup when you need it…

If one uses llama.cpp in CPU mode, then models are mmaped, so they do not really occupy memory when not used (they are just in reclaimable page cache).

Do anyone actually use the CPU for anything besides testing? Last time I tried it, it was horribly slow compared to GPU that there wasn't really any point to use for me, besides getting access to more memory.

Re: Ask HN: What AI assistants are already bundled for Linux?

#32

Post by dartharva »

brucethemoose2 wrote: post #15

One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes. Running one as a background desktop assistant is whole different animal than calling a Microsoft API.

You don't need the larger/smarter models for most "assistant" use cases. Small language models like Phi-2 can be enough.

Re: Ask HN: What AI assistants are already bundled for Linux?

#35

Post by number6 »

AlienRobot wrote: post #23
HKH2 wrote: post #21

Earlier quoted context omitted.

Not OP, but when searching for files, spelling something wrong, or using the wrong synonym is a big problem. We're just used to computers being inflexible.

I mean, I see, but do you need an entire AI for this? This sounds like a specific problem that would be best solved by a specific algorithm.

You can use fzf for this

Re: Ask HN: What AI assistants are already bundled for Linux?

#36

Post by ingenieroariel »

diggan wrote: post #31
zajio1am wrote: post #29

Earlier quoted context omitted.

If one uses llama.cpp in CPU mode, then models are mmaped, so they do not really occupy memory when not used (they are just in reclaimable page cache).

Do anyone actually use the CPU for anything besides testing? Last time I tried it, it was horribly slow compared to GPU that there wasn't really any point to use for me, besides getting access to more memory.

I have.

On a Mac Studio with NixOS based Asahi Linux and 128Gb of RAM, mixtral 8x7b uses 49GB of RAM. At the same time I load airflow tasks that deal with world wide datasets (using ~60GB on 16 parallel streams with the performance cores) format is parquet and also mmaped.

Computer still has 8 efficiency cores and the whole GPU for visualizing the maps using lonboard / browsing / etc.

The computer uses 8-10W when idle, ~100W when running jobs or actively using the LLM and around ~200W when really using the GPU.

This makes it very efficient energy wise in my book compared to the beast of keeping a modern CPU and nvidia GPU on when idle. My electricity bill is unaffected.

Re: Ask HN: What AI assistants are already bundled for Linux?

#37

Post by yjftsjthsd-h »

diggan wrote: post #31
zajio1am wrote: post #29

Earlier quoted context omitted.

If one uses llama.cpp in CPU mode, then models are mmaped, so they do not really occupy memory when not used (they are just in reclaimable page cache).

Do anyone actually use the CPU for anything besides testing? Last time I tried it, it was horribly slow compared to GPU that there wasn't really any point to use for me, besides getting access to more memory.

Some of us like to experiment with new technology but don't physically own the kind of a hardware that is ideal for it. So yes, I've actually gotten passable results running on CPU (on a 2019 laptop at that)

Re: Ask HN: What AI assistants are already bundled for Linux?

#39

Post by luke-stanley »

dartharva wrote: post #32
brucethemoose2 wrote: post #15

One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes. Running one as a background desktop assistant is whole different animal than calling a Microsoft API.

You don't need the larger/smarter models for most "assistant" use cases. Small language models like Phi-2 can be enough.

I'd love this to be true, and it might be for some specific well tested situations with a narrow set of data that you can be confident about. But that's a bit wishful, isn't it?

Re: Ask HN: What AI assistants are already bundled for Linux?

#40

Post by yjftsjthsd-h »

bossyTeacher wrote: post #17

Why? They take lots of space and lots of computing power. Linux has always been about lightweight and a bundle only containing essential things. You can always install one if you need it but as it stands right now LLMs are not useful enough to warrant their bundling in a distro. Just my 2 cents

> Linux has always been about lightweight and a bundle only containing essential things.

I really don't think that's true. There have always been distros that are based on being tiny, of course, but I think most of the normal distros are concerned with hitting a happy medium of size and features. Otherwise I can't imagine why anything would be shipping GNOME or KDE over LXDE, or why libreoffice would be installed by default. So the question is more where LLMs are on cost/benefit... which granted, may not be there yet, but I could easily see it turning into a checkbox at install time - "this machine has 16+GB of RAM; add SomeLLM?"

Post reply on HN