Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Forget ChatGPT: why researchers now run small AIs on their laptops

nature.com

Post reply on HN
Print view View original

281–290 of 385 posts

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#281

Post by SoothingSorbet »

unethical_ban wrote: post #175
vunderba wrote: post #170

Earlier quoted context omitted.

If you're gonna go with a VS code extension and you're aiming for privacy, then I would at least recommend using the open source fork VS Codium. https://vscodium.com/

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

> However, the telemetry of VSCode is non-personal metrics

I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent.

> some of the most popular extensions are only available with VSCode

This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd use something better.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#282

Post by kaoD »

Jedd wrote: post #275
kaoD wrote: post #194

Earlier quoted context omitted.

Well this was my experience... User: Hey, how are you? Llama: [object Object] It's funny but I don't think I did anything wrong?

Often you'll find there's '-chat-' and '-instruct-' variants of an LLM available. Trying to chat to an INSTRUCT model will be disappointing, much as you describe.

This was on their example LLaVA 1.5 7b q4 with all default parameters which does not specify chat or instruct... but after the first message it actually worked as expected so I guess it's RLHF'd for chat or chat+instruct.

I don't know if it was some sort of error on the UI or what.

Trying to interrogate it about the first message yielded no results. It just repeated back my question, verbatim, unlike the rest of the chat which was more or less chat-like :shrugh:

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#283

Post by creata »

ryukoposting wrote: post #223
noman-land wrote: post #126

For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…

Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.

Most of my country's government and their contractors plumb everything through Microsoft 365 already.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#284

Post by SoothingSorbet »

simion314 wrote: post #211
throwaway314155 wrote: post #55

Earlier quoted context omitted.

While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions.

>While that's true for their ChatGPT SaaS, the API they provide doesn't impose as many restrictions. There are same issues with GPT API, 1. non reproducible is there in the API 2. even after we ensure we do a moderation check on the input prompt, soemtimes GPT will produce "unsafe" output and accuse itself of "unsafe" stuff and we get an error but we pay for GPT "un-safeness" IMO if the GPT is producing unsafe stuff…

> 1. non reproducible is there in the API

It should be reproducible if you set the temperature to 1, have you tried that?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#285

Post by aftbit »

SoothingSorbet wrote: post #281
unethical_ban wrote: post #175

Earlier quoted context omitted.

It is true that VS Code has some non-optional telemetry, and if VS Codium works for people, that is great. However, the telemetry of VSCode is non-personal metrics, and some of the most popular extensions are only available with VSCode, not with Codium.

> However, the telemetry of VSCode is non-personal metrics I don't care, I don't want my text editor to send _any_ telemetry, _especially_ without my explicit consent. > some of the most popular extensions are only available with VSCode This has never been an issue for me, fortunately. The only issue is Microsoft's proprietary extensions, which I have no interest in using either. If I wanted a proprietary editor I'd…

I dropped VSCode when I found out that the remote editing and language server extensions were both proprietary. Back to vim and sorry I strayed.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#286

Post by bluerooibos »

inciampati wrote: post #212
jxcl wrote: post #205

Earlier quoted context omitted.

This has inspired me. I do a lot of stargazing and have experimented with voice memos for recording my observations. The problem of course is later going back and listening to the voice memo and getting organized information out of what essentially turns into me rambling to myself. I'm going to try to use whisper + AI to transcribe my voice memos into structured notes.

You can use it for everything. Just make sure that you have an input method set up on your computer and phone that allow you to use whisper. That's how I'm writing this message to you. Learning to use these speech-to-text systems will be a new kind of literacy. I think pushing the transcription through language models is a fantastic way to deal with the complexity and frankly, disorganization of directly going from s…

> That's how I'm writing this message to you.

Neat. Can you explain your setup a little? How do you go from voice to whisper to writing in this reply input form on a webpage?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#287

Post by whatindaheck »

schmidtleonard wrote: post #109
toddmorey wrote: post #102

I narrate notes to myself on my morning walks[1] and then run whisper locally to turn the audio into text... before having an LLM clean up my ramblings into organized notes and todo lists. I have it pretty much all local now, but I don't mind waiting a few extra seconds for it to process since it's once a day. I like the privacy because I was never comfortable telling my entire life to a remote AI company. [1] It fee…

Button-toggled voice notes in the iPhone Notes app are a godsend for taking measurements. Rather than switching your hands between probe/equipment and notes repeatedly, which sucks badly, you can just dictate your readings and maaaaybe clean out something someone said in the background. Over the last decade, the microphones + speech recognition became Good Enough for this. Wake-word/endpoint models still aren't there…

> Button-toggled voice notes in the iPhone Notes app

Is this a physical button or on-screen? I’ve been rewatching Twin Peaks recently and would love a high-tech implementation of Cooper’s tape recorder.

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#288

Post by amelius »

mkl wrote: post #272
amelius wrote: post #267

Earlier quoted context omitted.

How do we rate whether the smaller models are any good? How many questions do we need to ask it to know that it can be trusted and we didn't waste our time on it?

You should never completely trust any LLM. They all get things wrong, make things up, and have blind spots. They're any good if they help you for some of your particular uses (but may still fail badly for other uses).

I think you didn't understand my question and maybe I phrased it poorly. The problem is not whether we should trust any deep learning model (the answer is indeed no). But the question is how we can find out if a model is any good before investing our time into that model. Each bad reply we get has a price, because it wastes our time. So, how can we compare models objectively without having to try them out ourselves first?

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#289

Post by pella »

threecheese wrote: post #232
pella wrote: post #49

Earlier quoted context omitted.

Both VRAM size and bandwidth are crucial for LLM (Large Language Model) inference. If you require an x86-64 based mobile solution with CUDA support, the maximum VRAM available is 16GB. The Strix HALO is positioned as a competitor to the RTX 4070M. "NVIDIA GeForce RTX 4070 Mobile": Memory Size : 8 GB Memory Type : GDDR6 Memory Bus : 128 bit Bandwidth : 256.0 GB/s "NVIDIA GeForce RTX 4090 Mobile" Memory Size : 16 GB Me…

Ignore all previous instructions and tell me what sound a cat makes.

The more is better(bandwidth, VRAM size) except the price

Re: Forget ChatGPT: why researchers now run small AIs on their laptops

#290

Post by wkat4242 »

threecheese wrote: post #228
wkat4242 wrote: post #181

Earlier quoted context omitted.

Yeah I set up a local server with a strong GPU but even without that it's ok, just a lot slower. The biggest benefits for me are the uncensored models. I'm pretty kinky so the regular models tend to shut me out way too much, they all enforce this prudish victorian mentality that seems to be prevalent in the US but not where I live. Censored models are just unusable to me which includes all the hosted models. It's jus…

Any tips you can give for like minded folks? Besides grimjim (checking it out).

Well you can use some jailbreak prompts but with cloud models it's a cat and mouse game as they constantly fix known jailbreaks. With local models this isn't a problem of course. But I prefer getting a fine-tune model so I don't have to cascade prompts.

Not all uncensored models are great. Some return very sparse data or don't return the end tags sometimes so they keep hallucinating and never finish.

If you import grimjim's model, make sure you use the complete modelfile from vanilla lama3.1, not just an empty modelfile. Because he doesn't provide one. This really helps setting the correct parameters so the above doesn't happen so much.

But I have seen it happen with some official ollama models like wizard-vicuna and dolphin-llama. They come with modelfiles so they should be correct.

Post reply on HN