331–340 of 385 posts
Earlier quoted context omitted.
True but try to find a 96GB GPU.
This drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
I have a three year old M1 Max, 32gb RAM. Llama 8bn runs at 25 tokens/sec, that’s fast enough, and covers 80% of what I need. On my ryzen 5600h machine, I get about 10 tokens/second, which is slow enough to be annoying. If I get stuck on a problem, switch to chat gpt or phind.com and see what that gives. Sometimes, it’s not the LLM that helps, but changing the context and rewriting the question. However I cannot use…
A small model necessarily is missing many facts. The large model is the one that has memorized the whole internet, the small one is just trained to mimic the big one. You simply cannot compress the whole internet under 10gb without throwing out a lot of information. Please be careful about what you take as fact coming from the local model output. Small models are better suited to summarization.
This is totally wrong and a potentially dangerous way to think about LLMs. They have no clue about what's factual knowledge and what is not, per design.
What's the current cost of building a DIY bare-bones machine setup to run the top LLaMA 3.1 models? I understand that two nodes are typically required for this. Has anyone built something similar recently, and what hardware specs would you recommend for optimal performance? Also, do you suggest waiting for any upcoming hardware releases before making a purchase?
405B is beyond homelab-scale. I recently obtained a 4x4090 rig, and I am comfortable running 70B and occasionally 128B-class models. For 405B, you need 8xH100 or better. A single H100 costs around $40k.
Earlier quoted context omitted.
Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there…
> Button-toggled voice notes in the iPhone Notes app Is this a physical button or on-screen? I’ve been rewatching Twin Peaks recently and would love a high-tech implementation of Cooper’s tape recorder.
Check out the new voice transcription feature in iOS 18. On my SE 2022 (very much not high-end), I have a good-size record/pause button on screen.
After you’re done with the voice recording, it gives you a transcription of what you spoke.
Earlier quoted context omitted.
It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.
>the telemetry of VSCode is non-personal metrics But I don't want it. I want my software to work for me, not against me. >and some of the most popular extensions are only available with VSCode, not with Codium. I'll manage without them. What's especially annoying is that this restriction is completely artificial. Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect,…
How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?
Earlier quoted context omitted.
While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.
>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions. There are same issues with GPT API, 1. non reproducible is there in the API 2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff…
With regard to DALLE - that's a fair complaint I didn't realize they don't have a seed for their API. You should really try switching to an open model if you can. You'll have complete control. I recommend flux-schnell or flux-dev.
Earlier quoted context omitted.
This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.
You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper. That's how I'm writing this message to you. Learning to use these speech-to-text systems will be a new kind of literacy. I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from s…
If it matters, I managed to install Jan AI on my Linux Mint and able to use the Mistral model. I use an Android phone if it helps. Thanks.
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Thanks for your recommendation! I just ran Llamafile for the first time with a custom prompt on my Windows machine (i5-13600KF, RX6600) and found that it performed extremely slowly and wasn't as smart as ChatGPT. It doesn't seem suitable for productive writing. Did I do something wrong, or is there a way to improve its writing performance?
I've found that you can get faster performance by choosing a smaller model and/or by using a smaller quantization. You can use other models with llamafile as well. They have some prebuilt ones:
https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
You can also search for other llamafiles for other models on HuggingFace by using the llamafile tag.
https://huggingface.co/models?library=llamafile&sort=trendin...
And you can download model weights directly and use them by providing an -m flag to llamafile but that's getting a bit less straightforward.
https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...
Earlier quoted context omitted.
It’s mostly how would you solve this programming problem, or reminders on syntax, scaffolding a configuration file etc. Often it’s a form of rubber duck programming, with a smarter rubber duck.
All of this can be solved with a 3-20MB PDF file, a 10kb snippet/template file, and a whiteboard.