Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

github.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#111

Post by giancarlostoro »

fho wrote: post #88
giancarlostoro wrote: post #78

This is something I've been wondering about myself. What's the "Minimally Viable LLM" that can have simple conversations. Then my next question is, how much can we push it so it can learn from looking up data externally, can we build a tiny model with an insanely larger context window? I have to assume I'm not the only one who has asked or thought of these things. Ultimately, if you can build an ultra tiny model that…

You might be interested in RWKV: https://www.rwkv.com/ Not exactly "minimal viable", but a "what if RNNs where good for LLMs" case study. -> insanely fast on CPUs

My personal idea revolves around "can I run it on a basic smartphone, with whatever the 'floor' for basic smartphones under lets say $300 is for memory (let's pretend RAM prices are normal).

Edit: The fact this runs on a Smartphone means it is highly relevant. My only thing is, how do we give such a model an "unlimited" context window, so it can digest as much as it needs. I know some models know multiple languages, I wouldnt be surprised if sticking to only English would reduce the model size / need for more hardware and make it even smaller / tighter.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#112

Post by Peteragain »

cwmoore wrote: post #90
nrhrjrjrjtntbt wrote: post #57

Earlier quoted context omitted.

MAYBE

Universally correct reply, although honestly a bit vague.

Fair. The background reading is the EMCA stuff - conversation analysis cf Sacks etc at, and Ethnomethods (Garfunkel). And Vygotsky cf Kozulin. People such as Robert Moore at IBM and Lemon at Herriot-Watt work in this space but there is no critical mass in the face of LLM mania.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#115

Post by dangus »

ben_w wrote: post #98
dangus wrote: post #84

Earlier quoted context omitted.

Group FaceTime calls didn’t exist at the time. That wasn’t added until 2018 and required iOS 12. Remember that Slack does simultaneous multiple participants screen sharing plus annotations plus HD video feeds from all participants plus the entirety of the rest of the app continues to function as if you weren’t on a call at all simultaneously. It’s an extremely powerful application when you really step back and think…

> Group FaceTime calls didn’t exist at the time. That wasn’t added until 2018 and required iOS 12. And CU-SeeMe did that in the early 90s with even worse hardware: https://en.wikipedia.org/wiki/File:CU-Schools.GIF Even more broadly, group calls were sufficiently widely implemented to get themselves standardised 29 years ago: https://en.wikipedia.org/wiki/H.323 > It’s an extremely powerful application when you really…

Why don’t you just go ahead and tell me what specs you think Slack should run on and link me to an example program that has 100% feature parity that stays within those specs?

Showing me a black and white Showing me that someone thought of a protocol is pointless. Just look at the history of HDTV. We wouldn’t really describe HDTV as being available to consumers despite it existing in the early 1990s.

I’d also like you to show me a laptop SKU sold in the last 10 years that is incapable of running Slack. If Slack is so inefficient you should be able to find me a computer that struggles with it.

Finally, I’ll remind you that Slack for mobile is a different application that isn’t running in the same way as the desktop app and uses fewer resources. The latest version of it will run on very old phone hardware, going all the way back to the iPhone 8 (2GB RAM), and that’s assuming you even need the latest version for it to function.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#116

Post by dangus »

fc417fc802 wrote: post #110
dangus wrote: post #84

Earlier quoted context omitted.

Group FaceTime calls didn’t exist at the time. That wasn’t added until 2018 and required iOS 12. Remember that Slack does simultaneous multiple participants screen sharing plus annotations plus HD video feeds from all participants plus the entirety of the rest of the app continues to function as if you weren’t on a call at all simultaneously. It’s an extremely powerful application when you really step back and think…

If these applications only hogged memory when under stress (outgoing screencap plus video, multiple streams incoming, display to 3+ monitors) you might have a point. But that's not the case so you don't. Meanwhile I can play back multiple 1080 videos on different monitors, run a high speed curl download, saturate my gigabit LAN with a bulk transfer, and run a brrfs scrub in the background all most likely without brea…

If you didn’t babysit your task manager would you know which program used more RAM or not?

This argument is just so endless and tiring.

Saturating my bandwidth or running a btrfs scrub isn’t accomplishing the business logic I need to do my job, that’s what my web browser is doing.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#117

Post by [deleted] »

andrepd wrote: post #95
dangus wrote: post #87

Earlier quoted context omitted.

I have great doubts that you were doing simultaneous screen sharing from multiple participants with group annotation plus HD video in your group calls, all while supporting chatting that allowed you to upload and view multiple animated gifs, videos, rich formatted text, reactions, slash command and application automation integrations, all simultaneously on your Pentium 3. I would be interested to know the name of the…

You're grasping at anything to justify the unjustifiable. Not only did I do most (not all, obviously) of those things in my Pentium 3, including video and voice chat, screenshare, and silly animated gifs and rich text formatting, but also: that's beside the point. Let's compare like with like then; how much memory does it take to have a group chat with a few people and do a voice/video in MSN messenger or the origina…

[deleted]

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#118

Post by GuB-42 »

MagicMoonlight wrote: post #91

What I really want is a game where each of the NPCs has a tiny model like this, so you can actually talk to them.

I thought about this, chatbots existed well before LLMs (Eliza: 1966!) and the only time I have seen a commercially successful game with a (very simple) chatbot was Quake III Arena!

Quake 3 is probably the last game where you would expect a chatbot, as there are few games where storytelling matters less and it is a very little known feature, but Quake 3 bots can react to what you say in the chat, in addition to the usual taunts.

But that's the thing, Quake 3 can do it because it is inconsequential, in a story-driven game like a RPG, NPCs have a well defined spot in the story and gameplay, they tell you exactly what you need to know, as to not disrupt the flow of the story. Tell you too much, and they will spoil the big reveal, tell you too little, and you don't know what to do, tell you irrelevant details and you get lost chasing them. It has to be concise and to the point, so that those who don't really care know what to do to advance the story, but with enough flavor to make the world alive. It is really hard to find the right balance, and if in addition, you have to incorporate a chatbot, it borders on impossible.

It looks like a good idea on the surface, but it most likely isn't, unless it is clearly not part of the main gameplay loop, as in Quake 3.

Some people had some success using a (big) LLM as a DM in D&D, which I think is easier since it can make up the story as it advances, it is much harder to make up game elements in a computer RPG that are not programmed in.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#119

Post by fc417fc802 »

dangus wrote: post #116
fc417fc802 wrote: post #110

Earlier quoted context omitted.

If these applications only hogged memory when under stress (outgoing screencap plus video, multiple streams incoming, display to 3+ monitors) you might have a point. But that's not the case so you don't. Meanwhile I can play back multiple 1080 videos on different monitors, run a high speed curl download, saturate my gigabit LAN with a bulk transfer, and run a brrfs scrub in the background all most likely without brea…

If you didn’t babysit your task manager would you know which program used more RAM or not? This argument is just so endless and tiring. Saturating my bandwidth or running a btrfs scrub isn’t accomplishing the business logic I need to do my job, that’s what my web browser is doing.

So is it the "business logic" or is it the multiple HD streams that are supposed to account for the resource consumption? You've changed your story. But do please explain how the "business logic" to handle the chat box, UI, and whatever else is supposed to justify the status quo.

People making excuses for poorly designed software is what's tiring.

Re: Show HN: Z80-μLM, a 'Conversational AI' That Fits in 40KB

#120

Post by numpad0 »

dangus wrote: post #84
ben_w wrote: post #65

Earlier quoted context omitted.

> Slack/Teams handles company-wide video calls and can render anything a web browser can, and they run an entire App Store of apps, all from a cross-platform application. The 4th Gen iPod touch had 256 meg of RAM and also did those things, with video calling via FaceTime (and probably others, but I don't care). Well, except "cross platform", what with it being the platform.

Group FaceTime calls didn’t exist at the time. That wasn’t added until 2018 and required iOS 12. Remember that Slack does simultaneous multiple participants screen sharing plus annotations plus HD video feeds from all participants plus the entirety of the rest of the app continues to function as if you weren’t on a call at all simultaneously. It’s an extremely powerful application when you really step back and think…

The problem with that kind of feature/benefit based thinking is that it won't correlate with code or computational footprints well. That's like justifying price of cars with seatback materials. That's not where the costs are.

Modern chat apps like Slack, Discord, Teams, etc. are extremely resource intense solely by being skinned Chrome showing overbloated HTMLs. That's it. Most of the "actual" engineering of it is outsourced and externalized to Google, NVIDIA/Intel/AMD, Microsoft/Apple, etc.

Post reply on HN