Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

211–220 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#211

Post by simion314 »

throwaway314155 wrote: post #55
simion314 wrote: post #8

OpenAI APIs for GPT and Dalle have issues like non determnism, and their special prompt injection where they add stuff or modify your prompt (with no option to turn that off. Makes it impossible to do research or to debug as a developer variations of things.

While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.

>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.

There are same issues with GPT API,

1. non reproducible is there in the API

2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff then I should not pay for it's problems.

3. dalle gives no seed so no reproducible, and no option to opt out on their GPT modifying the prompt , so images are sometimes absurdly enhanced with extreme amount of details or extreme diversity, so you need to fight against their GPT enhancing.

What extra option we have with the APIs that is useful ?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#212

Post by inciampati »

jxcl wrote: post #205
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.

You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper.

That's how I'm writing this message to you.

Learning to use these speech-to-text systems will be a new kind of literacy.

I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from speech to text.

By doing this we can all basically type at 150-200 words a minute.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#213

Post by RevEng »

I'm currently working on an LLM-based product for a large company that's used in circuit design. Our customers have very strict confidentiality requirements since the field is very competitive and they all have trade secret technologies that give them significant competitive advantages. Using something public like ChatGPT is simply out of the question. Their design environments are often completely disconnected from the public internet, so our tools need to run local models. Llama 3 has worked well for us so far and we're looking at other models too. We also prefer not being locked in to a specific vendor like OpenAI, since our reliance on the model puts us in a poor position to negotiate and the future of AI companies isn't guaranteed.

For my personal use, I also prefer to use local models. I'm not a fan of OpenAI's shenanigans and Google already abuses its customers data. I also want the ability to make queries on my own local files without having to upload all my information to a third party cloud service.

Finally, fine tuning is very valuable for improving performance in niche domains where public data isn't generally available. While online providers do support fine tuning through their services, this results in significant lock in as you have to pay them to do the tuning in their servers, you have to provide them with all your confidential data, and they own the resulting model which you can only use through their service. It might be convenient at first, but it's a significant business risk.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#214

Post by credit_guy »

narrator wrote: post #192
threecheese wrote: post #174

Earlier quoted context omitted.

Have you tried a Llamafile? Not sure what platform you are using. From their readme: > … by combining llama.cpp with Cosmopolitan Libc into one framework that collapses all the complexity of LLMs down to a single-file executable (called a "llamafile") that runs locally on most computers, with no installation. Low cost to experiment IMO. I am personally using MacOS with an M1 chip and 64gb memory and it works perfectl…

With 64GB can you run the 70B size llama models well?

No, you can't. I have 128 GB and a 70B llamafile is unusable.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#215

Post by distances »

zelphirkalt wrote: post #166
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?

I'm using Ollama with an AMD GPU (7800, 16GB) on Linux. Works out of the box. Another question is then if I get much value out of these local models.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#216

Post by mnky9800n »

evbogue wrote: post #207
AlienRobot wrote: post #202

Earlier quoted context omitted.

2000: Javascript is webpages. 2010: Javascript is webservers. 2020: Javascript is desktop applications. 2024: Javascript is AI.

From this data we must conclude that within our lifetimes all matter in the universe will eventually be reprogrammed in JavaScript.

I'm not sure I want to live in that reality.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#217

Post by behnamoh »

toddmorey wrote: post #178
alyandon wrote: post #105

Earlier quoted context omitted.

I would be greatly interested in knowing how you set all that up if you felt like sharing the specifics.

My hope is to make this easy with a GH repo or at least detailed instructions. I'm on a Mac and I found the easiest way to run & use local models is Ollama as it has a rest interface: https://github.com/ollama/ollama/blob/main/docs/api.md I just have a local script that pulls the audio file from Voice Memos (after it syncs from my iPhone), runs it through openai's whisper (really the best at voice to speech; excellen…

You can record your voice messages and send them to yourself in Telegram. They're saved on-device. You can then create a bot to do things to stuff as they come in, like "transcribe new ogg files and write back the text as a message after the voice memo".

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#218

Post by twh270 »

privacyis1mp wrote: post #203
twh270 wrote: post #198

Earlier quoted context omitted.

I'm interested, but I can't find any documentation for it. Can I give it local content (documents, spreadsheets, code, etc.) and ask questions?

> Can I give it local content (documents, spreadsheets, code, etc.) It's coming roughly in December (may be sooner). Roadmap is following: - October - private remote AI (when you need smarter AI than your machine can handle, but don't want your data to be logged or stored anywhere) - November - Web search capabilities (so the AI will be capable of doing websearch out of the box) - December - PDF, docs, code embedding…

Oh awesome, thank you! I will check back in December.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#219

Post by actualwitch »

atemerev wrote: post #39
mrfinn wrote: post #10

It's kinda funny how nowadays an AI with 8 billion parameters is something "small". Specially when just two years back entire racks were needed to run something giving way worst performance.

IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.

A solid chunk of python is just wrappers around C/C++, most tensor frameworks included.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#220

Post by valval »

lukan wrote: post #188
flimflamm wrote: post #155

Earlier quoted context omitted.

That's just meat CoT (chain of thought) - right?

I do not understand?

GP is making a joke about speaking to oneself really just being the human version of Chain of Thought, which in my understanding is an architecural decision in LLMs to have it write out intermediate steps in problem solving and evaluate the validity of them as it goes.
Post reply on HN