Live data from Hacker News

Can a MUD evaluate LLMs? A $99 proof of concept

cruciblebench.ai

81–88 of 88 posts

Re: Can a MUD evaluate LLMs? A $99 proof of concept

#81

I am having a hell of a lot of fun letting agents play and understand a (still alive, human populated) MUD, which I have also played for the last 30 years. It’s mostly my way to play with local llm inference (m5 64gb, gwen3.6 27). It’s amazing. They build maps, classify events (building a grammar for a parser), run experiments (to verify the grammar). They are now (given the correct tools/infrastructure) trying to fi…

How is the agent interfacing with it? Are you just manually copy pasting game output and doing the response or something more integrated?

in a container a daemon is managing the telnet connection/logins and throws into a parser, that takes grammaries (basically, lists of regexes) and produces events, which are appended into simple files as json blobs (with types: rooms, npc enters/exists, unknown, etc)

agents can tail live logs or parse in any sense: i built agents that try to classify (via grammar file edits) "what that unknown event is", build experiments and verify against live runs

surprisingly, a local 27b is quite smart

ultimately you need to run fast inference for combats (5 second rounds), for that i built a system to prepare fine-tuning runs over 3b models

overall, using container primitives for everything seems to be the biggest gain: in the end they mostly are coding models, so intuitively know how to interact without needing too many explanations

example: mud commands are small bash commands, the "game help" is provided via man for those commands, and hence when there are attempts at reasoning (which is done for knowledge building via grammar files) agents are great at using apropos to find out more about a specific topic

another example: setting a "pick all money from body" is a tail + grep, once the agent is shown it he picked up usage quite fast

to run agents i use pi and simple setup where a main 27b agent acts as an orchestrator and delegates runs to agents (orchestrator also build prompts), usually as "experiments" "if i classify event x than what should happen in-game"?

i do not really keep it running all time, it's mostly about me putting up challenges, see what happens, and using that as a learning experience

Re: Can a MUD evaluate LLMs? A $99 proof of concept

#84

Earlier quoted context omitted.

I agree with you. You've also got more / easier competition. IIRC, MUDs peaked in the mid-late 90s. MMOs were really the death knell, then the explosion of other types of games, mobile games, then to your point things like Discord. One adjacent project we're exploring for the future is if you took all of the current understanding of game design and mechanics with modern AI functionality, knowledge bases, etc, and bui…

Would be kind of a cool experiment for a mud-style interface, where each npc/avatar was backed by a lower-cost LLM... similar rules for the npcs in the game, but a background feed, and history of interactions with other players as background. Maybe limiting npc's to only a certain number of moves that aren't a response to other users per day... It could be a lot of fun.

I'm building this exact thing, actually. Intelligence determines which model powers a mob/npc. Getting neat results so far!

Re: Can a MUD evaluate LLMs? A $99 proof of concept

#85

Earlier quoted context omitted.

There was a distinct split between people there for the grind and people there for the RP. I was there for the RP and I was mostly on MUCKs, MUSHes, and the occasional MOOs (multi-user chat kingdoms, multi-user shared hallucinations, and mud (object-oriented) ). They all had a combat system lurking somewhere in their code but those routines rarely got used.

Firstly, "multi-user chat kingdom" is a backronym that was not envisioned by the original authors of TinyMUCK, but perhaps by the users or developers of some future fork. TinyMUCK is notable for having a "Fuzzball" edition or fork, that powered FurryMUCK itself, as well as related worlds which hosted and catered to people in furry fandom. Secondly, exactly none of these had a "combat system" of any kind. These were a…

Thanks for the corrections :)

I sure spent a lot of time on Furry. I think the penultimate time I logged on there was something like a decade ago; the last time was to attend Tugrik's wake.

Re: Can a MUD evaluate LLMs? A $99 proof of concept

#87

I don't understand this article. In fact I even read the entire whitepaper and I don't understand it. You didn't use "a MUD"; you used a very limited "MUD-style environment". This is not a MUD. Your headline, your article, your whitepaper is a lie. Your "MUD" didn't originate in the 1970s; you coulnd't even be arsed to include its source code! I was at first mystified when you couldn't be clear about what genre or sp…

Why would the code need to originate in the 70s?

This is some sparkling wine vs Champagne nonsense.

Re: Can a MUD evaluate LLMs? A $99 proof of concept

#88
post #9

Earlier quoted context omitted.

> even what can be done. I think this could be solved with bridge bots to make your MUD multiplatform. You could have a dedicated web app, an IRC bridge, a Discord bridge, etc. Discord-brained users can still use your MUD and contribute to network effects, and you can provide an off-ramp ("I just want to play VarelionScape, maybe I'll just open the dedicated website on my phone instead of going on Discord and digging…

That's a good idea, thanks! We'd only considered the web app + using a client.

Can I ask what your MUD is called?
Post reply on HN