Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anyone doing anything cool with tiny language models?

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 356 posts

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#241

Post by GardenLetter27 »

frail_figure wrote: post #189
pk-protect-ai wrote: post #184

Earlier quoted context omitted.

why don't you spam the companies who want your data instead? The sites can simply stop gathering your data, then they will not require to ask for consent ...

It’s the same comments on HN as always. They think EU setting up rules is somehow worse than companies breaking them. We see how the US is turning out without pesky EU restrictions :)

The US has 3x higher salaries, larger houses and a much higher quality of life?

I work as a senior engineer in Europe and make barely $4k net per month... and that's considered a "good" salary!

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#242

Post by rpastuszak »

antonok wrote: post #68

I've been using Llama models to identify cookie notices on websites, for the purpose of adding filter rules to block them in EasyList Cookie. Otherwise, this is normally done by, essentially, manual volunteer reporting. Most cookie notices turn out to be pretty similar, HTML/CSS-wise, and then you can grab their `innerText` and filter out false positives with a small LLM. I've found the 3B models have decent performa…

Tangentially related, I worked on something similar, using LLMs to find and skip sponsored content in YT videos:

https://butter.sonnet.io/

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#243

Post by whalesalad »

dh1011 wrote: post #130

I copied all the text from this post and used an LLM to generate a list of all the ideas. I do the same for other similar HN post .

chatgpt did a stellar job parsing the "books on hard things" thread from a little while ago. my prompt was:

Can you identify all the books here, sorted by a weight which is determined based on a combo of the number of votes the comment has, the number of sub-comments, or the number of repeat mentions.

Ideally retain hyperlinks if possible.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#244

Post by bashbjorn »

I'm working on a plugin[1] that runs local LLMs from the Godot game engine. The optimal model sizes seem to be 2B-7B ish, since those will run fast enough on most computers. We recommend that people try it out with Gemma 2 2B (but it will work with any model that works with llama.cpp)

At those sizes, it's great for generating non-repetitive flavortext for NPCs. No more "I took an arrow to the knee".

Models at around the 2B size aren't really capable enough to act a competent adversary - but they are great for something like bargaining with a shopkeeper, or some other role where natural language can let players do a bit more immersive roleplay.

[1] https://github.com/nobodywho-ooo/nobodywho

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#245

Post by Draiken »

lnenad wrote: post #205
relistan wrote: post #174

Earlier quoted context omitted.

Interesting idea. But those say what’s in the commit. The commit diff already tells you that. The best commit messages IMO tell you why you did it and what value was delivered. I think it’s gonna be hard for an LLM to do that since that context lives outside the code. But maybe it would, if you hook it to e.g. a ticketing system and include relevant tickets so it can grab context. For instance, in your first example,…

I disagree. When you look at the git history in x months you're gonna have a hard time understanding what was done following your example.

I disagree. If you look back and all you see are commit messages summarizing the diff, you won't get any meaningful information.

Telling me `Changed timeout from 30s to 60s` means nothing, while `Increase timeout for slow requests` gives me an actual idea of why that was done.

Even better if you add meaningful messages to the commit body.

Take a look at commits from large repositories like the Linux kernel and we can see how good commit messages looks like.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#246

Post by mettamage »

kreyenborgi wrote: post #160
sundarurfriend wrote: post #135

Earlier quoted context omitted.

You're using it to anonymize your code, not de-anonymize someone's code. I was confused by your comment until I read the replies and realized that's what you meant to say.

I read it the other way, their code contains eg fetch(url, pw:hunter123), and they're asking Claude anonymized questions like "implement handler for fetch(url, {pw:mycleartrxtpw})" And then claude replies fetch(url, {pw:mycleartrxtpw}).then(writething) And then the local llm converts the placeholder mycleartrxtpw into hunter123 using its access to the real code

It's that yea

Flow would be:

1. Llama prompt: write a console log statement with my username and password: mettamage, superdupersecret

2. Claude prompt (edited by Llama): write a console log statement with my username and password: asdfhjk, sdjkfa

3. Claude replies: console.log('asdfhjk', 'sdjkfa')

4. Llama gets that input and replies to me: console.log('mettamage', 'superdupersecret')

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#247

Post by SuperHeavy256 »

Evidlo wrote: post #81
RVuRnvbM2e wrote: post #73

Earlier quoted context omitted.

This is fantastic. How have your hooked up a mobile number to the llm?

Android app that forwards to a Python service on remote workstation over MQTT. I can make a Show HN if people are interested.

I am so SO interested, please make a Show HN

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#248

Post by mettamage »

sundarurfriend wrote: post #230
kreyenborgi wrote: post #160

Earlier quoted context omitted.

I read it the other way, their code contains eg fetch(url, pw:hunter123), and they're asking Claude anonymized questions like "implement handler for fetch(url, {pw:mycleartrxtpw})" And then claude replies fetch(url, {pw:mycleartrxtpw}).then(writething) And then the local llm converts the placeholder mycleartrxtpw into hunter123 using its access to the real code

> Put in all your work related questions in the plugin, an LLM will make it as an abstract question for you to preview and send it So the LLM does both the anonymization into placeholders and then later the replacing of the placeholders too. Calling the latter step de-anonymization is confusing though, it's "de-anonymizing" yourself to yourself. And the overall purpose of the plugin is to anonymize OP to Claude, so t…

I could've been a bit more clear, sorry about that.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#249

Post by jajko »

iamnotagenius wrote: post #222
andai wrote: post #24

Earlier quoted context omitted.

Could you elaborate?

I just run llama-cli with the model. Every time I want some "awk" or "find" trickery, I just ask model. Good for throwaway python scripts too.

Can it do 'sed'?

I think one major improvement for folks like me would be human->regex LLM translator, ideally also respecting different flavors/syntax for various languages and tools.

This has been a bane of me - I run into requirement to develop some complex regexes maybe every 2-3 years, so I dig deep into specs, work on it, deliver eventually if its even possible, and within few months almost completely forget all the details and start at almost same place next time. It gets better over time but clearly I will retire earlier than this skill settles in well.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#250

Post by gpm »

kreyenborgi wrote: post #161
gpm wrote: post #133

I made a shell alias to translate things from French to English, does that count? function trans llm "Translate \"$argv\" from French to English please" end Llama 3.2:3b is a fine French-English dictionary IMHO.

Is it better than translatelocally? https://translatelocally.com/downloads/ (the same as used in firefox)

It's different. It doesn't always just give one translation but different options. I can do things like give it a phrase and then ask it to break it down. Or give it a word and if its translation doesn't make sense to me ask how it works in the context of a phrase.

llm -c, which continues the previous conversation, is specifically useful for that sort of manipulation.

It's also available from the command line, which I find convenient because I basically always have one open.

Post reply on HN