How Do Coding Agents Use Tools to Build Software?

TL;DR

Coding agents work effectively when they can execute programs, edit files, browse the web, and use existing software libraries through code. OpenHands applies this approach to data analysis, new script creation, and iterative software improvement, while its interface summarizes actions in plain English and lets users inspect detailed commands or notebooks when needed.

Transcript

okay uh hi everyone so um I was given the task of talking about agents in 2024 and this is an impossible task because there are so many agents so many agents in 2024 so this is going to be strongly covered by like my personal experience and what I think is interesting and important but um I think it's an important topic so let let's go ahead so um ... Read More

Key Insights

• Coding agents can support data analysis, new software creation, and iterative improvement of existing applications. Neubig says he uses them five to ten times a day, showing that their practical role extends beyond generating isolated code snippets to completing varied software-development tasks.
• Concrete prompts improve agent performance. The demonstrations specify outputs, data sources, interfaces, and implementation details, including labeled scatter plots, CSV input, Jinja2 templates, API documentation, pie charts, and totals across a monitored period, giving each agent a clear operational target.
• Programmatic tool use can reduce repeated model calls. Instead of asking a model to make roughly 30 granular API calls, OpenHands lets the agent write and execute Python code that invokes exposed APIs together, observes errors, and revises the program when necessary.
• OpenHands relies on a small core tool set. Its agent can execute Bash programs and Jupyter notebook cells, browse and overwrite file sections, perform global search and replacement, and browse the web through actions such as scrolling, clicking, and entering text.
• Existing software libraries expand an agent's effective capabilities. Access to Python packages for HTTP requests, PDF-to-text conversion, and other tasks allows a coding agent to perform data visualization and related work without requiring a separately designed tool for every possible operation.
• GitHub integration enables agents to work within active development workflows. An agent can clone repositories, use the GitHub API, inspect issue comments, check GitHub Actions, respond when tagged, and attempt fixes such as repairing failing tests on a pull request.
• Human-agent interfaces must balance visibility with information overload. OpenHands presents an English description for each action, hides potentially large Bash commands or notebook contents by default, and lets users open those details when closer inspection becomes useful.
• Coding agents can complete useful work while a user handles another activity. During the presentation, an OpenHands agent analyzed the SWE-Bench repository and generated labeled plots while Neubig continued speaking, illustrating asynchronous assistance within a broader working session.

Questions & Answers

Q: What tasks can coding agents handle in everyday software work?

Coding agents can assist with data analysis, creating new software, and improving existing applications. The demonstrations include plotting SWE-Bench score increases over time, reading an email service's API documentation to create a CSV-driven sending script with Jinja2 templates, and adding pie charts plus total counts to software that monitors agent contributions to a repository.

Q: Why do concrete prompts help coding agents perform better?

Concrete prompts give an agent specific outputs, inputs, constraints, and sources to work with. Neubig explicitly says agents work better with somewhat concrete prompts. His examples identify the repository to analyze, the plots and labels to produce, the API documentation to read, the script interface to build, and the features to add to existing software.

Q: How does OpenHands reduce the number of agent tool calls?

OpenHands lets the coding agent write arbitrary Python code that calls exposed APIs. Rather than directing the model through about 30 individual tool calls for a multi-country comparison, the agent can write one program that performs the operations together. It can execute that program, receive errors, correct mistakes, and rerun the code to obtain the result.

Q: What tools does an OpenHands coding agent receive?

The core capabilities include executing Bash programs, running cells in Jupyter notebooks, editing files, performing global search and replacement, and browsing the web. File tools can expose selected parts of files and overwrite content. Web browsing includes underlying interactions such as scrolling, text input, and clicking, even though Neubig groups them broadly as one capability.

Q: How can software libraries extend a coding agent's abilities?

A coding agent can use libraries already available to human programmers instead of requiring a custom agent tool for every task. Neubig mentions Python libraries for HTTP requests and PDF-to-text conversion as examples. With access to this ecosystem, the agent can also perform data visualization, clone repositories, interact with APIs, and assemble solutions through executable code.

Q: How does OpenHands show users what an agent is doing?

OpenHands provides an English description whenever the agent takes an action, such as announcing that it will create an email-sending script. Large Bash commands and Jupyter notebook contents are not displayed in full by default. Users can expand those actions, open the notebook, and inspect detailed commands or outputs when they need more information.

Q: How can coding agents work inside GitHub workflows?

OpenHands offers a GitHub plugin that responds when users tag the agent in issues or pull requests. A developer can ask it to resolve an issue or fix failing tests on a pull request, and the agent can enter the repository workflow to attempt the task. Agents can also inspect issue comments, GitHub Actions, and other information through the GitHub API.

Q: What did the live SWE-Bench data analysis demonstrate?

The live task showed that a coding agent could analyze the SWE-Bench repository and create scatter plots of score improvement over time while Neubig continued his presentation. The output labeled the top three systems for each requested dataset, including SWE-Bench normal, verified, and light. Neubig presented this as a useful example for researchers who frequently perform data analysis.

Summary & Key Takeaways

• Graham Neubig demonstrates three everyday coding-agent tasks: analyzing SWE-Bench score changes, creating an email-sending script from API documentation, and extending an existing monitoring application. The data-analysis agent finishes during the presentation, producing scatter plots and labeling leading systems across several SWE-Bench datasets while Neubig continues speaking to the audience.

• OpenHands gives its agent a compact set of capabilities for executing Bash commands and Jupyter cells, editing files, performing global search and replacement, and browsing the web. The agent can then write Python programs that call APIs and libraries, allowing one program execution to replace a long sequence of separate model-directed tool calls.

• The human-agent interface aims to reveal enough information without overwhelming users. OpenHands describes each action in English, collapses lengthy commands by default, and permits deeper inspection of Bash activity and Jupyter notebooks. It also meets developers in existing workflows through chat, GitHub issue tagging, pull-request assistance, and a remote headless runtime.