Can someone ELI5 how LLMs are manipulating environments like this? I have a similar question for when you see the blender examples online.
AFAIU the models are multimodal, so you could screenshot factorio and let it come back with some manipulation like move west 5 tiles, do that and then take another screenshot, but that seems wildly inefficient so I'm sure i'm missing something
Simplest general way is literally asking the model for the X and Y coordinates to click, given an input image. Tends to work with most multi-modal models, although some have better accuracy than others, obviously.
More involved integration, would be to expose reading and writing to the game state or character controls via tools that the LLM can use (either in your own harness however you want, or via MCP). This is basically what most "AI tooling" connected to editors do today, one way or another, with fallback to "click on this UI element after taking a screenshot" often, or "simulate click on this widget with #id".
So given a goal, the LLM would first call check_inventory(), then craft_building(:building), then walk_to_coords(x,y), then build_building(:building) and so on, which tends to indeed be a lot faster than doing it by screenshots and inferring coordinates.
Ultimately I think the harness and prompt matters more than the model here, but I guess what they're exploring, is the shortest possible prompt to have it do it correctly. Otherwise, setting your harness to "Continue until you've launched a rocket in the game" together with a bunch of added rules to the prompts as you discover where it gets stuck, is an easy way to probably get most LLMs to be able to do this correctly and relatively fast. Some of them requires more handholding than others, which I guess might be another thing they're testing for here, it isn't 100% clear what the intention/motivation nor exact environment was/is.
Edit: As fitting as it is, I was just trying out the Unreal Engine Editor MCP/tools for the first time with Codex, which ended up also being a good example for how a "real turn" would look like, with a LLM connected to some other editor: https://gist.github.com/embedding-shapes/b16c50ed591569d3c70...
There are people much smarter than me to answer this, but the basic idea are that models have a harness around them that include something called "tools." Each tool is basically a text description of its functionality. So for something like this, for example, the model might have a "move" tool. All of the tools are added to the model's context. Then the basic idea is that when the model detects it needs to "move west 5 tiles" it will spit out a bunch of JSON to "call" the tool, and then at that point a normal program picks up that tool call output and actually performs the action. So in other words, the LLM isn't directly manipulating the environment, it's spitting out JSON that tells the harness it is in what to do.
As far as I understand, the best any AI was able to get to until now was being able to produce blue science which is like level 2 or 3 of a science tree like 10 levels deep.
To me who has maybe put 20-30 hours into Factorio, this is even more impressive stuff than some of the Blender videos I've seen... just mind-boggling.
Ultra is max reasoning level that splits the work and spawn subagents. Good for whole-project reviews (well, as long as the cybersecurity refusal doesn't appear for whatever reason), but unusable on the $20/mo tier
Can someone ELI5 how LLMs are manipulating environments like this? I have a similar question for when you see the blender examples online.
AFAIU the models are multimodal, so you could screenshot factorio and let it come back with some manipulation like move west 5 tiles, do that and then take another screenshot, but that seems wildly inefficient so I'm sure i'm missing something
Simplest general way is literally asking the model for the X and Y coordinates to click, given an input image. Tends to work with most multi-modal models, although some have better accuracy than others, obviously.
More involved integration, would be to expose reading and writing to the game state or character controls via tools that the LLM can use (either in your own harness however you want, or via MCP). This is basically what most "AI tooling" connected to editors do today, one way or another, with fallback to "click on this UI element after taking a screenshot" often, or "simulate click on this widget with #id".
So given a goal, the LLM would first call check_inventory(), then craft_building(:building), then walk_to_coords(x,y), then build_building(:building) and so on, which tends to indeed be a lot faster than doing it by screenshots and inferring coordinates.
Ultimately I think the harness and prompt matters more than the model here, but I guess what they're exploring, is the shortest possible prompt to have it do it correctly. Otherwise, setting your harness to "Continue until you've launched a rocket in the game" together with a bunch of added rules to the prompts as you discover where it gets stuck, is an easy way to probably get most LLMs to be able to do this correctly and relatively fast. Some of them requires more handholding than others, which I guess might be another thing they're testing for here, it isn't 100% clear what the intention/motivation nor exact environment was/is.
Edit: As fitting as it is, I was just trying out the Unreal Engine Editor MCP/tools for the first time with Codex, which ended up also being a good example for how a "real turn" would look like, with a LLM connected to some other editor: https://gist.github.com/embedding-shapes/b16c50ed591569d3c70...
There are people much smarter than me to answer this, but the basic idea are that models have a harness around them that include something called "tools." Each tool is basically a text description of its functionality. So for something like this, for example, the model might have a "move" tool. All of the tools are added to the model's context. Then the basic idea is that when the model detects it needs to "move west 5 tiles" it will spit out a bunch of JSON to "call" the tool, and then at that point a normal program picks up that tool call output and actually performs the action. So in other words, the LLM isn't directly manipulating the environment, it's spitting out JSON that tells the harness it is in what to do.
Lots of people in here asking how it was done. Heres what the linked page says:
>...made a mode in LUA for the game to expose an MCP server that allows the llm to play the game.
(Factorio has a very mature LUA based modding system )
As far as I understand, the best any AI was able to get to until now was being able to produce blue science which is like level 2 or 3 of a science tree like 10 levels deep.
To me who has maybe put 20-30 hours into Factorio, this is even more impressive stuff than some of the Blender videos I've seen... just mind-boggling.
Let’s see the factory layouts though. Release the blueprints!
There's vod: https://m.twitch.tv/videos/2867906622
The factory must grow, and now it can grow while you sleep, which is the one thing Factorio players never managed.
Sleep is a bottleneck
How is the AI controlling the video game? Is there special coding/harness that allows for this?
That’s pretty impressive. I wonder how they handled biter attacks and some of the real-time requirements.
Getting good coding results with Astra Ultra agent and Astra Low subagents.
What does Ultra offer in this context? Better code? Or is the problem being solved extremely complicated that requires Ultra?
Ultra is max reasoning level that splits the work and spawn subagents. Good for whole-project reviews (well, as long as the cybersecurity refusal doesn't appear for whatever reason), but unusable on the $20/mo tier