Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Factorio Learning Environment – Agents Build Factories

jackhopkins.github.io

Post reply on HN
Print view View original

181–190 of 225 posts

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#181

Post by csense »

scottmsul wrote: post #143
scottmsul wrote: post #138

There was a HN post here not too long ago about a team that used reinforcement learning to train an agent to beat pokemon red. They mentioned how they had to tweak the cost function to give small rewards for exploring and big rewards for completing "essential tasks" like beating gyms. I wonder if this same approach could be used here in factorio? Using the pokemon red analogy the main "essential tasks" in Factorio ar…

Also I should add, being a Factorio veteran with 2-3k hours in this game, I think the goal of making the "largest possible factory" is too vague and not the right metric. When Factorio players make large megabases, they don't go for "size" per se, but rather science research per minute. The metric you should be telling the agents is SPM, not "largest" base!

Agree, "largest" base has some pathologies.

Put machine #1 at the starting location, run in one direction, and put machine #2 just before time runs out.

This is going to be a huge factory (as measured by its bounding box) but it's not super interesting.

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#182

Post by dismalpedigree »

kridsdale1 wrote: post #112
vessenes wrote: post #109

Earlier quoted context omitted.

That makes sense and it’s really interesting - it is a challenging visual test for sure; thousands of entities, either multi tier visual representations (screen, map, overview map) or a GIANT high res image. I hereby propose FLE-V a subset benchmark for visual models where they just turn a factorio image into a proper FLE description. And maybe the overview and map images as well.

Such research could have hundreds of billions of dollars in downstream GDP implications when applied to real industrial settings.

Not to mention the increased productivity of everyone not wasting their time in factorio (myself included) because the optimal solution is known.

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#183

Post by groby_b »

martbakler wrote: post #116
jillyboel wrote: post #108

Earlier quoted context omitted.

Why would screenshots be necessary if a textual description of the factory state is both easier to interpret and less prone to confusion? The game is played on a grid, so converting the game state to ascii ought to be trivial.

It actually is engineering wise quite trivial but the underlying question is which modality is the best to elicit spatial reasoning capabilities from the current general models. We tried (very anecdotally) a couple of months ago to get an agent to reason over a couple of ascii representations of factories and the results weren't very promising. It seems the models struggle with creating an accurate internal spatial r…

> It seems the models struggle with creating an accurate internal spatial representation of the game state only using textual tokens

That'd be actually interesting research material for the claim that LLMs are able to build internal representations of the world. (Either they can't at all, which'd be an important insight, or it turns out there's something fundamentally different about modalities that engages different reasoning/world model capabilities, which would be even more interesting)

Or, if you want to really go wild, "what capabilities allow models to reason in modalities fundamentally different from their input data/training data".

Damn it, I should quit and go back to University. [Ed.: She wouldn't quit, she likes her job, don't believe her]

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#184

Post by deadbabe »

Am I the only one who doesn’t find the results promising?

This is a ton of compute power and complexity for what is basically a shitty AI. It has no practical purpose. Better AIs have been built with less, why don’t people appreciate them? Or do we just take them for granted?

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#185

Post by aftbit »

noddybear wrote: post #68
spieswl wrote: post #62

Fantastic idea. It seems like there are a lot of interesting experiments to be had here. The lab-play scenarios having a time-related component seems like a good idea, I assume most Factorio players that keep biters on treat them as a combined temporal-spatial constraint, so you have a sort-of proxy comparison to a real game situation when you put the agents on a timer. I like the way that the framework design is tes…

One thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps / seconds. The idea would be to test agents in building a military-industrial complex. One amusing issue we had in developing this idea is that frontier models have an aversion to creating entities called 'GunTurret' etc - as it goes against their constitution! (perhaps we should rename tu…

>One amusing issue we had in developing this idea is that frontier models have an aversion to creating entities called 'GunTurret' etc - as it goes against their constitution! (perhaps we should rename turrets to 'SuperSoaker' or something)

This sounds like a great idea for a short story in the style of Malak by Peter Watts. Imagine a future warfighter AI that has been fitted with a set of filters to make it think it's really having a pillowfight or building a factory to make screws while it's actually tearing people apart or optimizing a military production line.

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#186

Post by goriv »

sonofhans wrote: post #164
mNovak wrote: post #162

Is there a human-play benchmark (even informally) for this style of interface? Not saying it's necessary or even relevant, I'm just curious to know what programmatic Factorio feels like -- I imagine spatial reasoning around text prompts would be fairly challenging for human players to navigate as well.

Human benchmarks for Factorio are speed runners — rushing to launch the first rocket. The current record is just over 4 hours for one player, and 90 minutes for a team. You can see just from that that a multi-tasking LLM has room to outperform humans.

I think he is talking about a human using the programatic API the LLMs are using to play the game. I think that would be a whole lot slower than normal playthrough

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#187

Post by janzer »

sonofhans wrote: post #164
mNovak wrote: post #162

Is there a human-play benchmark (even informally) for this style of interface? Not saying it's necessary or even relevant, I'm just curious to know what programmatic Factorio feels like -- I imagine spatial reasoning around text prompts would be fairly challenging for human players to navigate as well.

Human benchmarks for Factorio are speed runners — rushing to launch the first rocket. The current record is just over 4 hours for one player, and 90 minutes for a team. You can see just from that that a multi-tasking LLM has room to outperform humans.

The current 4h12m hour record is for 100% (where you have to get every single achievement in the game, in the one run), any% (where you just need to launch a rocket) is under 2 hours (1h42 for the latest factorio v2.x, 1h18 for v1.x). There are a few other differences between the categories regarding map selection and blueprint use as well.

Records and specific rules for all categories can be found at https://www.speedrun.com/factorio

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#188

Post by stared »

noddybear wrote: post #46
mritchie712 wrote: post #38

have you tried sonnet 3.7 yet? guessing these aren't cheap evals to run. leaderboard: https://jackhopkins.github.io/factorio-learning-environment/...

Not yet, but starting the runs for 3.7 later today! The cost for running all the evals (across all models) was about $10k. Simply giving the agents access to the tool descriptions and API schema is like 20k tokens from the outset. It would be really cool to use retrieval techniques to reduce this burden. I suspect that this will also outright improve the performance of all models - which becomes worse as the context…

Since Claude 3.5 Sonnet is that good, I am curious how fares Claude 3.5 Haiku.

For programming-like tasks, I expect similar-ish distribution that in programming, see e.g. https://web.lmarena.ai/leaderboard

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#189

Post by pyinstallwoes »

martbakler wrote: post #147
grayhatter wrote: post #118

Earlier quoted context omitted.

> As the complexity of the game state grew and the screenshots were filled with more entities, the models got even more confused and started hallucinating directions, entities etc or weren't capable of troubleshooting factories with apparent mistakes (i.e missing transport belt, wrongly rotated inserter). We think it's because [...] I think you just described a research paper that would advance sota. Less describing…

Sounds almost like a visual "needle in a haystack" type of work, that could be quite interesting!

Where’s Waldo test for vlm

Re: Show HN: Factorio Learning Environment – Agents Build Factories

#190

Post by pyinstallwoes »

martbakler wrote: post #175
ajcp wrote: post #165

Earlier quoted context omitted.

Did you try providing 2D vectors of where each object relates to every other object? Seems like the most obvious way. In my experience the current generation of models are very poor at spatial reasoning even when given accurate coordinate based location assignments of each object. But I suspect when a model can build the whole relationship of all objects by being given those spatial relationships in a vector they wil…

We did discuss this at some point but didn't end up trying it out. I think it's quite an interesting avenue and worth a shot, my intuition also says that the spatial capabilities will improve if the model has more access to relative info and doesn't need to infer it from absolute coordinates

Given vector space on text is more of a spatial space of semantic distance then spatial distance of geometric objects intuitively feel of a different nature due to the fact that words are not at all likely to be represented in similar ratios of distances.

I think a tokenization of ratios between perceived boundaries might help. But, I’m just shooting in the dark.

Post reply on HN