Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running LLMs in one line of Python without Docker

lepton.ai

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Show HN: Running LLMs in one line of Python without Docker

#21

Post by jiayq84 »

brucethemoose2 wrote: post #19
jiayq84 wrote: post #17

Earlier quoted context omitted.

Thanks - we definitely agree that llama.cpp is great. Big fan of their optimizations. We are more or less orthogonal to the engines though - in the sense that we serve as the infra/platform to run and manage those implementations easily. For example, we support running a wider range of models - for example sdxl is one single line too: lep photon run -n sdxl -m hf:stabilityai/stable-diffusion-xl-base-1.0 --local It's…

SDXL is indeed a monster to install and setup. The UIs are even worse. IDK if the GPL license is compatible with your business, but I wonder if you could package Fooocus or Fooocus-MRE into a window? Its a hairy monster to install and run, but I've never gotten such consistently amazing results from a single prompt box + style dropdown box (including native HF diffusers and other diffusers-based frontends). The autom…

Oh wow yeah, that is a beast. Let me give it a shot.

Re: Show HN: Running LLMs in one line of Python without Docker

#22

Post by [deleted] »

rw2 wrote: post #5

Is not having to build a dockers image worth $100 a month? I do find server setup to be a pain but I think if I will use a model for a year, I can take the time(3-4 hours) to set it up. Only with constant switching of models would I use a service like this. I never set up bigger models like LLAMA on servers. Other hacker news people can chime in.

[deleted]

Re: Show HN: Running LLMs in one line of Python without Docker

#24

Post by bddppq »

brucethemoose2 wrote: post #15

llama.cpp (and derivative projects) is quickly becoming SOTA for many use cases, and it basically has zero dependencies. Kobold.cpp, for example, provides an entire web UI and API with python, and 3 python packages (numpy, sentencepiece, and gguf which is the llama.cpp library). The llm itself is a single file you can get with curl or whatever. It takes less than a minute to compile against the native CPU/acclerator…

lepton is at a different layer comparing to llama.cpp, in fact for LLM model files that are of GGUF format, it's using llama.cpp (ctransformers to be precise) as the execution engine

Re: Show HN: Running LLMs in one line of Python without Docker

#25

Post by ideal0227 »

hiatus wrote: post #23

How does this compare to ollama.ai

Oollama.ai focuses on making it as easy as possible to run models locally. We aim to provide a seamless experience that feels the same whether you're developing locally or deploying remotely for production.

declaimer: work at Lepton AI.

Re: Show HN: Running LLMs in one line of Python without Docker

#26

Post by hiatus »

ideal0227 wrote: post #25
hiatus wrote: post #23

How does this compare to ollama.ai

Oollama.ai focuses on making it as easy as possible to run models locally. We aim to provide a seamless experience that feels the same whether you're developing locally or deploying remotely for production. declaimer: work at Lepton AI.

I've been using a remote ollama server with a local jupyter notebook. The langchain configuration allows me to specify the ollama host. So I can develop locally with remote models. I guess I still don't see the difference. Does lepton decouple the HTTP server from the model backend?

Re: Show HN: Running LLMs in one line of Python without Docker

#27

Post by sbierwagen »

jiayq84 wrote: post #12
moffkalast wrote: post #11

Earlier quoted context omitted.

Hardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.

In theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.

Llama.cpp can run quantized Falcon on a top end Mac Studio, which is only five grand: https://twitter.com/ggerganov/status/1699791226780975439

If I'm paying a third party a hundred bucks a month, I'd at least want them to be able to match the capacities of consumer hardware.

Post reply on HN