Llama 70B but no Falcon 180B?
Are there some examples(prompts) that Falcon 180B is performing better than Llama 70B?
Hardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.
Show HN: Running LLMs in one line of Python without Docker
11–20 of 32 posts
Re: Show HN: Running LLMs in one line of Python without Docker
#12Earlier quoted context omitted.
Are there some examples(prompts) that Falcon 180B is performing better than Llama 70B?
Hardly anyone can even run a 70B model, let alone 180B. Any anecdata will be extremely rare.
In theory one can have 640G = 8 * 80G A100s memory and launch it. 180B Falcon with fp16 will be 360G, so there would be enough memory. It's definitely going to be very expensive indeed.
Re: Show HN: Running LLMs in one line of Python without Docker
#13[deleted]
Re: Show HN: Running LLMs in one line of Python without Docker
#14How is data secured and protected?
Is the company collecting all my prompts and responses?
I don’t see a privacy policy or anything like that linked on the main page.
Re: Show HN: Running LLMs in one line of Python without Docker
#15llama.cpp (and derivative projects) is quickly becoming SOTA for many use cases, and it basically has zero dependencies.
Kobold.cpp, for example, provides an entire web UI and API with python, and 3 python packages (numpy, sentencepiece, and gguf which is the llama.cpp library). The llm itself is a single file you can get with curl or whatever. It takes less than a minute to compile against the native CPU/acclerator architecture, with nothing but the GPU libs themself, which nets better performance than a generic binary distribution.
...Its not "one line" I guess, but I can hardly imagine a simpler setup. It doesn't really need docker or a fancy container.
Re: Show HN: Running LLMs in one line of Python without Docker
#16How is data secured and protected? Is the company collecting all my prompts and responses? I don’t see a privacy policy or anything like that linked on the main page.
Thanks - the policies are listed here: https://www.lepton.ai/policies
we'll put a link on our homepage.
In short - we do not collect, record, or log any of your prompts and responses. They are computed in memory, returned and discarded on the fly.
Re: Show HN: Running LLMs in one line of Python without Docker
#17llama.cpp (and derivative projects) is quickly becoming SOTA for many use cases, and it basically has zero dependencies. Kobold.cpp, for example, provides an entire web UI and API with python, and 3 python packages (numpy, sentencepiece, and gguf which is the llama.cpp library). The llm itself is a single file you can get with curl or whatever. It takes less than a minute to compile against the native CPU/acclerator…
Thanks - we definitely agree that llama.cpp is great. Big fan of their optimizations. We are more or less orthogonal to the engines though - in the sense that we serve as the infra/platform to run and manage those implementations easily. For example, we support running a wider range of models - for example sdxl is one single line too:
lep photon run -n sdxl -m hf:stabilityai/stable-diffusion-xl-base-1.0 --local
It's really about how to productize a wide range of models as easy as possible.
Re: Show HN: Running LLMs in one line of Python without Docker
#18congrats yangqing et al! i was really impressed by your llama2 demo https://llama2.lepton.run/ where you showed that you were the "fastest llama runners" (https://twitter.com/swyx/status/1695183902770614724). definitely needed for model hosting infra.
Re: Show HN: Running LLMs in one line of Python without Docker
#19llama.cpp (and derivative projects) is quickly becoming SOTA for many use cases, and it basically has zero dependencies. Kobold.cpp, for example, provides an entire web UI and API with python, and 3 python packages (numpy, sentencepiece, and gguf which is the llama.cpp library). The llm itself is a single file you can get with curl or whatever. It takes less than a minute to compile against the native CPU/acclerator…
Thanks - we definitely agree that llama.cpp is great. Big fan of their optimizations. We are more or less orthogonal to the engines though - in the sense that we serve as the infra/platform to run and manage those implementations easily. For example, we support running a wider range of models - for example sdxl is one single line too: lep photon run -n sdxl -m hf:stabilityai/stable-diffusion-xl-base-1.0 --local It's…
SDXL is indeed a monster to install and setup. The UIs are even worse.
IDK if the GPL license is compatible with your business, but I wonder if you could package Fooocus or Fooocus-MRE into a window? Its a hairy monster to install and run, but I've never gotten such consistently amazing results from a single prompt box + style dropdown box (including native HF diffusers and other diffusers-based frontends). The automatic augmentations to the SDXL pipine are amazing:
Re: Show HN: Running LLMs in one line of Python without Docker
#20congrats yangqing et al! i was really impressed by your llama2 demo https://llama2.lepton.run/ where you showed that you were the "fastest llama runners" ( https://twitter.com/swyx/status/1695183902770614724 ). definitely needed for model hosting infra.
Thanks so much for the warm words!