[dead]
Show HN: Factorio Learning Environment – Agents Build Factories
201–210 of 225 posts
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#202Earlier quoted context omitted.
Human benchmarks for Factorio are speed runners — rushing to launch the first rocket. The current record is just over 4 hours for one player, and 90 minutes for a team. You can see just from that that a multi-tasking LLM has room to outperform humans.
I think he is talking about a human using the programatic API the LLMs are using to play the game. I think that would be a whole lot slower than normal playthrough
We were able to pass all the early lab tasks manually - although it took a lot longer than using the UI!
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#203I don't understand - were these models post-trained to play factorio? A) If so, how is that possible given that e.g. Claude doesn't have public weights? B) If not, how would the agent know what the API does? Even if it's "guessing" from the English meaning of the API commands (e.g. place_entity_next_to places entity next to something), how would it know what the recipes are? If it's trying and learning we go back to A).
Having read the pdf I don't think these models were post-trained, so how do we explain the questions in B)?
And if indeed there's no post-training and authors expected exploration of recipes to come from the context window.... I think that's way too short for RL-style improvement.
In short, I don't understand how they could've tested those models with post training, and without post training they all did unbelievably well.
If the authors read this: can you give us an idea how many API query and API pairs fit within the context window, on average? Follow up, do you get better results if you abbreviate the API call names, so that more response pairs fit within one context window?
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#204I have long dreamt of automating Factorio in the way that HDL and a PCB router works: just specify the ingredients and it produces a Factorio Blueprint. First MVP stupid designs, then optimized routing, and eventually usable ingame where it connects with provided in/outputs. Would be more fun to develop than to play obviously.. I liked the nilhouse mega base with that factory-train-blocks blueprints, its basically Fa…
I've wondered if automating Factorio would free me of the compulsion to play it.
You should learn more about FPGA's!
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#205I don't understand - were these models post-trained to play factorio? A) If so, how is that possible given that e.g. Claude doesn't have public weights? B) If not, how would the agent know what the API does? Even if it's "guessing" from the English meaning of the API commands (e.g. place_entity_next_to places entity next to something), how would it know what the recipes are? If it's trying and learning we go back to…
The way I read the footnotes about the authors, one works at Anthropic. I would guess that is some insider access.
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#206I don't understand - were these models post-trained to play factorio? A) If so, how is that possible given that e.g. Claude doesn't have public weights? B) If not, how would the agent know what the API does? Even if it's "guessing" from the English meaning of the API commands (e.g. place_entity_next_to places entity next to something), how would it know what the recipes are? If it's trying and learning we go back to…
These models were not post-trained - all off-the-shelf.
We can fit about 128 pairs maximum in the context, but this performed the same as 32, which we ultimately decided on (for cost, latency purposes).
Encoding the input/outputs to make them shorter degraded performance. It seems that descriptive names is helpful for pretrained models because they have an intuition on what they do.
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#207I don't understand - were these models post-trained to play factorio? A) If so, how is that possible given that e.g. Claude doesn't have public weights? B) If not, how would the agent know what the API does? Even if it's "guessing" from the English meaning of the API commands (e.g. place_entity_next_to places entity next to something), how would it know what the recipes are? If it's trying and learning we go back to…
The way I read the footnotes about the authors, one works at Anthropic. I would guess that is some insider access.
One of us works at Anthropic - but we had no insider access to any models or weights. All of our evals were on public models.
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#208>We evaluate six frontier language models across both settings: Claude 3.5-Sonnet, GPT-4o, GPT-4o-Mini, Deepseek-v3, Gemini-2-Flash, and Llama-3.3-70B-Instruct. While I appreciate the effort and creativity that went into this there are a lot of much simpler dynamic benchmarks that can let you saturate the planning capabilities of non-reasoning models. Something as simple as giving a list of flight connections between…
This is true - there are simpler benchmarks that can saturate planning for these models. We were motivated to create a broader spectrum eval, to test multiple capabilities at once and remain viable into the future.
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#209Earlier quoted context omitted.
One thing we've been talking about is creating tasks that are a bit more 'tower defence', where biters are released every X steps / seconds. The idea would be to test agents in building a military-industrial complex. One amusing issue we had in developing this idea is that frontier models have an aversion to creating entities called 'GunTurret' etc - as it goes against their constitution! (perhaps we should rename tu…
>One amusing issue we had in developing this idea is that frontier models have an aversion to creating entities called 'GunTurret' etc - as it goes against their constitution! (perhaps we should rename turrets to 'SuperSoaker' or something) This sounds like a great idea for a short story in the style of Malak by Peter Watts. Imagine a future warfighter AI that has been fitted with a set of filters to make it think it…
There was a black mirror episode about this too, I seem to remember! Soldiers imagining they were fighting monsters - while actually committing war crimes.
Re: Show HN: Factorio Learning Environment – Agents Build Factories
#210Earlier quoted context omitted.
I think generating the scenarios as you suggest (in text) is easy, but creating correct factory game states to start from is a lot harder. AFAIK it reduces into the same manual task of designing an init state and a task to complete.
Yes each scenario will need someone to design it, but you can get a lot of mileage out of each. E.g. consider the "place the missing power pole" scenario: manually build a factory with a few dozen machines connected to a couple steam engines with 20 power poles, then you can generate 400 playable puzzles/scenarios by deleting 1-2 power poles from the working starting point. Humans would find all of these to be equiva…
Oh super interesting! Create 10 scenarios containing working factories, and ‘drop out’ entities to break the factory in different ways. great idea.