How to evaluate AI agents in Factorio learning environment

TL;DR

AI agents are evaluated via a Factorio based framework that supports unbounded and goal directed testing. The environment uses a code synthesis approach with Python to control factory actions, and two reward signals track production and milestone achievements to prevent trivial gaming of the score. Lab play tests specific capabilities and open play pushes agents to autonomously set and pursue longer term goals.

Transcript

Hey everyone, welcome back to a late inens lightning pod. This is Allesio, partner and CTO at Decible and Swix is running late, so we'll just get started and maybe you'll join. Today I'm joined by Jack and Mart, who were the two researchers behind the Factorial learning environment. Since we had multiple guests on the podcast before, bring up Facto... Read More

Key Insights

• FLE enables unbounded agent evaluation by scaling game complexity from basic mining to massive production.
• The harness uses a Python code generation approach so models can implement high level abstractions like classes and data structures for large factories.
• Two evaluation modes exist: lab play with structured tasks and open play where agents autonomously define goals and pursue them.
• Python is chosen because pre trained models are strong at Python and it allows flexible tooling abstraction for factory management.
• Scores come from production rates and milestone milestones to prevent models from merely mining resources without a meaningful objective.
• Lab play targets spatial reasoning and constrained environments to judge API usage and planning in a bounded setting.
• Open play reveals long term planning abilities as models must derive sub goals and orchestrate complex production chains.
• Deepseek shows a gap between lab play performance and open play results due to short term planning weaknesses.

Questions & Answers

Q: What is the Factorio Learning Environment and its purpose

The Factorio Learning Environment is a novel framework built on Factorio to enable unbounded agent evaluation. It provides infrastructure, API and metrics for assessing frontier language model agents in code generation, spatial reasoning and long term planning. It covers rapid scaling challenges from basic resource extraction to production chains that process millions of units per second, creating curricula for evaluating capable agents.

Q: How does the harness interact with Factorio

The harness uses the Factorio admin console over a network protocol called archon to execute actions remotely across many servers. This approach allows scaling the environment for training and evaluation, enabling researchers to shard workloads and run large experiments. It replaces traditional low level lures by compiling high level Python code that drives factory actions.

Q: Why is Python chosen for the control code

Python was chosen because pre trained models are good at Python, and it provides a flexible way to express high level actions like moving to a point or placing entities. Models write Python code that when executed interacts with the Factorio environment, allowing the creation of complex factory segments and encouraging the development of abstractions such as classes and functions to manage large scaled factories.

Q: What are the two reward signals used

The two reward signals are production statistics, indicating how much is produced per second, and milestones, indicating technologies or new items developed for the first time. This dual signal helps avoid degenerate behavior like over mining coal just to inflate production scores, ensuring the agent learns meaningful factory organization and progression.

Q: What is lab play and what does it measure

Lab play is a structured, goal oriented task with clear completion criteria. It measures the upper bound of performance on simpler, constrained tasks and specifically assesses spatial reasoning and the ability to use the API to build a factory piece by piece, following a defined progression and target count of entities.

Q: What is open play and what challenge does it present

Open play has no predetermined end state and tests the agent in unbounded evaluation. The agent must autonomously set and achieve increasingly complex goals, deriving sub goals like where to place drills or how to chain production steps, and handle long term planning across potentially large and distributed factories.

Q: What issue was observed with DeepSeek in open play

In open play, DeepSeek performed poorly by often creating many chests rather than integrating components into a coherent production line. This demonstrates a short term planning weakness where the model focuses on immediate artifacts instead of assembling an automated, scalable system, highlighting the need for long term objective planning.

Q: How do the authors describe the difference between lab and open play

Lab play provides a clear, structured testbed to quantify API usage and spatial reasoning under constrained conditions, while open play pushes the model to self organize and set long term objectives, revealing its ability to plan across multiple stages and scale production chains without predefined end goals. Together they evaluate capability and strategy.

Summary & Key Takeaways

• The Factorio Learning Environment provides infrastructure to evaluate frontier language model agents in code generation, spatial reasoning and long term planning within Factorio. It scales from basic resource extraction to production chains processing millions of units per second, creating a natural curriculum for assessing agents.

• Lab play offers structured tasks with clear completion criteria to measure targeted capabilities and upper bounds, while open play allows unbounded evaluation where agents must set sub goals and manage complex, scaled objectives.

• The framework uses an admin console based harness and Python based code generation to execute actions in game servers, with a dual reward signal system that balances production statistics and milestone achievements to avoid degenerate strategies.