111–120 of 184 posts
Earlier quoted context omitted.
Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).
Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.
Earlier quoted context omitted.
Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.
Have you tried this yourself? Curious to know how well this works for an LLM home lab.
Nvidia has significant dominance in the AI space because of their work on software and the overall platform.
With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power.
The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere.
Just don’t do it.
Earlier quoted context omitted.
Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.
Have you tried this yourself? Curious to know how well this works for an LLM home lab.
[1] https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jets...
Earlier quoted context omitted.
Did you do anything special to make that work? Is it useful? Or just a toy?
I've had less luck with Mixtral, but I run Yi 34B finetunes for general personal use, including quick queries for work. Its kinda like GPT 3.5, with no internet access and slightly less reliable responses, but unrestrained, much faster and with a huge (up to 75K on my Nvidia 3090) usable context. Mixtral is extremely fast though, at least at a batch size of 1.
If you have used gpt4 and then use mistral, it’s like looking at a Retina display and then have to go back to a low res screen. You are always thinking “but GPT4 could do this though”
Earlier quoted context omitted.
What is the definition of a submarine comment? Google fails and ChatGPT says: > A "submarine comment" on social media refers to a comment that is made on an old post or thread, long after the conversation has died down. This term derives from the idea of a submarine which remains submerged and out of sight for long periods before suddenly surfacing. In the context of social media, it's when someone delves deep into s…
I think GP is trying to coin a term for stealth marketing Hacker News comments, except the analogy doesn't really make sense.
Earlier quoted context omitted.
Using TheBloke nous-hermes-2-mixtral-8x7b-dpo.Q5_K_M.gguf, llam.cpp with oobabooga, on Ryzen 9 7950x and RTX 2070 super, offloading 4 layers to the GPU. 27,631MiB system memory used. 7358/8192MiB usage on the GPU. I'm getting 6.32 tokens/second. You: You are Hunter S. Thompson, write a 450 words essay on the LHC project. The writing must be in Thompson's style AI: Well, buckle up, kiddos, because we're about to dive…
I am using the exact same model. Ryzen 5600G w/32GB and an Nvidia P40 w/24GB VRAM 20/33 layers offloaded to GPU, 4K context. Uses 25GB system RAM and all 24GB VRAM. 5-7 tokens per second.
Thanks for making me feel better about investing in tht motherboard + CPU + RAM upgrade and deferring the GPU upgrade.
Unlinkable subscription: If you sign up for Leo Premium, you’re issued unlinkable tokens that validate your subscription when using Leo. This means that Brave can never connect your purchase details with your usage of the product, an extra step that ensures your activity is private to you and only you. The email you used to create your account is unlinkable to your day-to-day use of Leo, making this a uniquely private credentialing experience.
Interesting, I must have missed the first Leo announcement. I really like how privacy conscious it is. They don’t store any chat record which is what I want.
Earlier quoted context omitted.
Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.
Have you tried this yourself? Curious to know how well this works for an LLM home lab.
Maybe I will try to get a 32 GB+ LLM running one of those days.