Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is anyone doing anything cool with tiny language models?

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 356 posts

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#291

Post by computers3333 »

sainib wrote: post #218
tinco wrote: post #201

Earlier quoted context omitted.

That's cool, I really like it. One piece of feedback: I am usually more interested in the HN comments than in the original article. If you'd include a link to the comments then I might switch to GopherSignal as a replacement for the HN frontpage. My flow is generally: Look at the title and the amount of upvotes to decide if I'm interested in the article. Then view the comments to see if there's interesting discussion…

May be even rate each post on the comments activity level.

Great call! That’s a really solid idea—using the LLMs to rate posts based on comment activity could totally work and would be fun.

Were you thinking something like a “DramaLlama,” deciding if it’s a slow day or a meltdown-worthy soap opera in the comments? Or maybe something more valuable, like an “Insight Index” that uses the LLM to analyze comments for links, explanations, or phrases that add context or insight—basically gauging how constructive or meaningful the discussion is?

I also saw an idea in another post on this thread about an LLM that constantly listens to conversations and declares a winner. That could be fun to adapt for spicier posts—like the LLM picking a “winner” in the comments. Make the argument GopherSignal official lol. If it helps bring in another user, I’m all in!

Appreciate the feedback.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#292

Post by computers3333 »

goodklopp wrote: post #288
tinco wrote: post #201

Earlier quoted context omitted.

That's cool, I really like it. One piece of feedback: I am usually more interested in the HN comments than in the original article. If you'd include a link to the comments then I might switch to GopherSignal as a replacement for the HN frontpage. My flow is generally: Look at the title and the amount of upvotes to decide if I'm interested in the article. Then view the comments to see if there's interesting discussion…

I would love this feature. Regardless, what you have built is really cool

Hey thanks a ton for checking out GopherSignal! From the feedback I’m getting, it seems like comments and upvotes are the secret sauce I’ve been missing—appreciate you helping me get that through my thick skull lol. The pressure’s on now—I’ll do my best to deliver.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#293

Post by lacoolj »

Evidlo wrote: post #69

I have ollama responding to SMS spam texts. I told it to feign interest in whatever the spammer is selling/buying. Each number gets its own persona, like a millennial gymbro or 19th century British gentleman. http://files.widloski.com/image10%20(1).png http://files.widloski.com/image11.png

Most spam are just verifying you exist as a person, then from there you become an actual "target" if you respond.

This feels like an in-between that both wastes their time and adds you to extra lists.

Send the results somewhere! Not sure if "law enforcement" is applicable (as in, would be able/willing to act on the info) but if so, that's a great use of this data :)

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#294

Post by mystified5016 »

nozzlegear wrote: post #61

I have a small fish script I use to prompt a model to generate three commit messages based off of my current git diff. I'm still playing around with which model comes up with the best messages, but usually I only use it to give me some ideas when my brain isn't working. All the models accomplish that task pretty well. Here's the script: https://github.com/nozzlegear/dotfiles/blob/master/fish-func... And for this chan…

That's actually pretty useful. This could be a big help in betting back into the groove when you leave uncommitted changes over the weekend.

A summary of changes like this might be just enough to spark your memory on what you were actually doing with the changes. I'll have to give it a shot!

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#295

Post by bripkens »

Evidlo wrote: post #69

I have ollama responding to SMS spam texts. I told it to feign interest in whatever the spammer is selling/buying. Each number gets its own persona, like a millennial gymbro or 19th century British gentleman. http://files.widloski.com/image10%20(1).png http://files.widloski.com/image11.png

You should put all these interactions on the web. For education purposes ofc.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#296

Post by aDyslecticCrow »

coder543 wrote: post #132
azhenley wrote: post #12

Microsoft published a paper on their FLAME model (60M parameters) for Excel formula repair/completion which outperformed much larger models (>100B parameters). https://arxiv.org/abs/2301.13779

That paper is from over a year ago, and it compared against codex-davinci... which was basically GPT-3, from what I understand. Saying >100B makes it sound a lot more impressive than it is in today's context... 100B models today are a lot more capable. The researchers also compared against a couple of other ancient(/irrelevant today), small models that don't give me much insight. FLAME seems like a fun little model,…

I would like to disagree with its being irrelevant. If anything, the 100B models are irrelevant in the context and should be seen as a "fun inclusion" rather than a serious addition worth comparing against. It out-performing a 100B model at the time becomes a fun bragging point, but it's not the core value of the method or paper.

Running a prompt against every single cell of a 10k row document was never gonna happen with a large model. Even using a transformer model architecture in the first place can be seen as ludicrous overkill but feasible on modern machines.

So I'd say the paper is very relevant, and the top commenter in this very thread demonstrated their own homegrown version with a very nice use-case (paper abstract and title sorting for making a summary paper)

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#297

Post by aDyslecticCrow »

Tepix wrote: post #278
bashbjorn wrote: post #244

I'm working on a plugin[1] that runs local LLMs from the Godot game engine. The optimal model sizes seem to be 2B-7B ish, since those will run fast enough on most computers. We recommend that people try it out with Gemma 2 2B (but it will work with any model that works with llama.cpp) At those sizes, it's great for generating non-repetitive flavortext for NPCs. No more "I took an arrow to the knee". Models at around…

Cool. Are you aware of good games that use LLMs like this?

I have not seen much myself, but it's one of the earliest use cases I thought about when they started showing up.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#298

Post by coder543 »

aDyslecticCrow wrote: post #296
coder543 wrote: post #132

Earlier quoted context omitted.

That paper is from over a year ago, and it compared against codex-davinci... which was basically GPT-3, from what I understand. Saying >100B makes it sound a lot more impressive than it is in today's context... 100B models today are a lot more capable. The researchers also compared against a couple of other ancient(/irrelevant today), small models that don't give me much insight. FLAME seems like a fun little model,…

I would like to disagree with its being irrelevant. If anything, the 100B models are irrelevant in the context and should be seen as a "fun inclusion" rather than a serious addition worth comparing against. It out-performing a 100B model at the time becomes a fun bragging point, but it's not the core value of the method or paper. Running a prompt against every single cell of a 10k row document was never gonna happen…

> Running a prompt against every single cell of a 10k row document was never gonna happen with a large model

That isn’t the main point of FLAME, as I understood it. The main point was to help you when you’re editing a particular cell. codex-davinci was used for real time Copilot tab completions for a long time, I believe, and editing within a single formula in a spreadsheet is far less demanding than editing code in a large document.

After I posted my original comment, I realized I should have pointed out that I’m fairly sure we have 8B models that handily outperform codex-davinci these days… further driving home how irrelevant the claim of “>100B” was here (not talking about the paper). Plus, an off the shelf model like Qwen2.5-0.5B (a 494M model) could probably be fine tuned to compete with (or dominate) FLAME if you had access to the FLAME training data — there is probably no need to train a model from scratch, and a 0.5B model can easily run on any computer that can run the current version of Excel.

You may disagree, but my point was that claiming a 60M model outperforms a 100B model just means something entirely different today. Putting that in the original comment higher in the thread creates confusion, not clarity, since the models in question are very bad compared to what exists now. No one had clarified that the paper was over a year old until I commented… and FLAME was being tested against models that seemed to be over a year old even when the paper was published. I don’t understand why the researchers were testing against such old models even back then.

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#299

Post by potatoman22 »

Evidlo wrote: post #69

I have ollama responding to SMS spam texts. I told it to feign interest in whatever the spammer is selling/buying. Each number gets its own persona, like a millennial gymbro or 19th century British gentleman. http://files.widloski.com/image10%20(1).png http://files.widloski.com/image11.png

You probably just get more spam texts since you're replying. Maybe that's a good thing tbh

Re: Ask HN: Is anyone doing anything cool with tiny language models?

#300

Post by vvillena »

GardenLetter27 wrote: post #240
antonok wrote: post #68

I've been using Llama models to identify cookie notices on websites, for the purpose of adding filter rules to block them in EasyList Cookie. Otherwise, this is normally done by, essentially, manual volunteer reporting. Most cookie notices turn out to be pretty similar, HTML/CSS-wise, and then you can grab their `innerText` and filter out false positives with a small LLM. I've found the 3B models have decent performa…

It's funny that this is even necessary though - that great EU innovation at work.

Bear in mind, those arcane cookie forms are probably not compliant with EU laws. If there's not a "reject" button next to the "accept" button, the form is almost definitely not to spec.
Post reply on HN