Earlier quoted context omitted.
True but try to find a 96GB GPU.
H100 NVL is easily available. It’s just that it’s close to $20k.
It is actually much more expensive than 20k. I can find them between 25-30k only.
Forget ChatGPT: why researchers now run small AIs on their laptops
291–300 of 385 posts
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#292Earlier quoted context omitted.
This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.
You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper. That's how I'm writing this message to you. Learning to use these speech-to-text systems will be a new kind of literacy. I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from s…
It would be really amazing if you could expand your workflow a little, especially how you have stitched everything together.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#293Earlier quoted context omitted.
You basically need as much RAM as the size of the model.
You actually need a lot less than that if you use the mmap option, because then only activations need to be stored in RAM, the model itself can be read from disk.
You missed the part where this is slow as hell.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#294Earlier quoted context omitted.
Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l... Admittedly it's slow (3.5 token/sec)
Approximately, how many tokens per second would the (edited) >~ $ 40k x 8 >=~ $320k version process? Would this result in a >~32x boost in performance compared to other setups? Thanks!
If you really want to know an exact number for a specific use case, you can rent an 8xH100 node on RunPod and benchmark it.
You should expect somewhere around 30t/s for a single response, if running the FP8 rowwise quant that would typically be used on such a node, with TensorRT-LLM. Massively more in total with batching.
That quant is twice the size as the 4.5bpw one used on the Mac though. A lower quality one would be faster.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#295Earlier quoted context omitted.
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Here is someone running 405b on 12x3090 (4.5bpw). Total cost around $10k. https://www.reddit.com/r/LocalLLaMA/comments/1ej9uzh/local_l... Admittedly it's slow (3.5 token/sec)
This is really suboptimal. They should buy 4 more 3090s so they can run it massively faster with tensor parallelism.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#296Earlier quoted context omitted.
This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.
You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper. That's how I'm writing this message to you. Learning to use these speech-to-text systems will be a new kind of literacy. I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from s…
Cool. Not sure why downvoted.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#297Earlier quoted context omitted.
You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).
I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves f…
There are leaderboards [1] that can provide a rough estimate of the relative capabilities of different models.
[1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#298Earlier quoted context omitted.
> However, the telemetry of VSCode is non-personal metrics I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent. > some of the most popular extensions are only available with VSCode This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd…
I dropped VSCode when I found out that the remote editing and language server extensions were both proprietary. Back to vim and sorry I strayed.
Jetbrains is pretty ok on this front. I've been enjoying using my beefy computer to do work from my potato laptop.
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#299Earlier quoted context omitted.
If I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.
True but try to find a 96GB GPU.
This drives me insane.
PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now.
For $680 you can get 96 GB of very fast RAM.
How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
Re: Forget ChatGPT: why researchers now run small AIs on their laptops
#300Earlier quoted context omitted.
How is it bypassed?
There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
Good to know. I never got too far with VSCodium because of this limitation.