How Does Surfer H Automate Browser Tasks?

TL;DR

Surfer H automates web tasks by using screenshots, a policy to propose actions, a localizer to identify click coordinates, and a validator to check completion. Paired with the open-weight Hollow-1 models, it reportedly achieved 92.2% accuracy on WebVoyager at 13 cents per task, while remaining compatible with other vision-language models.

Transcript

H company has just released a state-of-the-art browser use agent framework. Not only that, they open sourced the core models that are powering all of it and they even published a research paper going over exactly how they were able to achieve such great performance. That's what we're going to be talking about today. I partnered with H Company to te... Read More

Key Insights

• Surfer H is a visual web retrieval agent that operates from website screenshots without requiring access to the document object model or accessibility tree. It simulates human interactions such as clicking, scrolling, waiting, navigating backward, and entering information.
• Runner H is a beta product that lets users describe web tasks in natural language and watch agents perform them. Multiple agents can run in parallel, allowing several independent browser assignments to proceed at the same time.
• The Surfer H architecture is divided into three modules: a policy, a localizer, and a validator. These components respectively propose actions, identify interface coordinates, and determine whether the agent's final answer or completed work satisfies the request.
• Hollow-1 is an open-weight family of lightweight vision-language models specialized in web navigation, information extraction, action evaluation, and interface-element localization. The available models include Hollow-1 Navigation and Hollow-1 Localization, with downloadable weights that developers can extend or fine-tune.
• The localizer works by examining a screenshot and returning two-dimensional coordinates for a described interface element. When the policy requests interaction with something such as a button, the browser uses those coordinates to simulate a human click.
• The validator checks whether a proposed answer is suitable for the user and whether the overall task is complete. If validation fails, its feedback is stored in the agent's memory, and execution continues until completion or a time or cost limit is reached.
• Surfer H with Hollow-1 7B reportedly achieved 92.2% accuracy on WebVoyager at 13 cents per task. The cited comparisons report 84.3% at 71 cents with GPT-4 and 88% at 26 cents with GPT-4.1 Mini.
• Runner H supports adjustable levels of human involvement, including highly involved, moderately involved, and fully automated operation. It also accepts uploaded documents as context, retains generated files, and provides built-in service connections such as Google Sheets.

Questions & Answers

Q: How does Surfer H automate browser tasks?

Surfer H receives a task, stores it in memory, and creates a policy containing sequential browser actions such as opening a URL, scrolling, waiting, or clicking. When an action targets a page element, a localizer analyzes the screenshot and supplies its coordinates. A validator then checks the result and either approves completion or returns feedback for another attempt.

Q: What are the policy, localizer, and validator in Surfer H?

The policy decides which browser actions should occur in sequence. The localizer examines website screenshots and identifies the two-dimensional coordinates of interface elements described by the policy. The validator evaluates the answer and determines whether the requested task has been completed. If it rejects the result, its feedback enters memory so the agent can continue working.

Q: Does Surfer H need website code or APIs to work?

Surfer H does not require the website's document object model, accessibility tree, or a custom API integration. It uses screenshots to understand the visible interface, locates the coordinates of relevant controls, and simulates human browser actions. This approach is intended for services that lack standardized or well-documented APIs that agents could otherwise use directly.

Q: What are the Hollow-1 open-weight models designed to do?

Hollow-1 is a family of lightweight vision-language models specialized in web navigation, information extraction, action evaluation, and interface localization. Hollow-1 Navigation can receive a user-interface image and task, then produce navigation steps. Hollow-1 Localization determines where an interface element appears on the screen so the browser can click the appropriate coordinates.

Q: How accurate and cost-efficient is Surfer H with Hollow-1?

The presented research reports that Surfer H with Hollow-1 7B achieved 92.2% accuracy on the WebVoyager benchmark at a cost of 13 cents per task. The cited comparison lists Surfer H with GPT-4 at 84.3% and 71 cents, while the GPT-4.1 Mini configuration reached 88% and cost 26 cents per task.

Q: What happened in the Runner H eBay demonstration?

Runner H was instructed to search eBay for Pokémon cards, extract ten listings, and create a Google Sheet containing the cards and their eBay links. After the user authenticated the Google Sheets connection, the agent planned and performed the task autonomously. The resulting spreadsheet included the collected entries, although two links were unavailable because of capture issues.

Q: Can Runner H execute multiple tasks and use connected services?

Runner H can launch multiple browser agents in parallel, so users can start several independent tasks without waiting for each one to finish. It also offers built-in connections to external services, including Google Sheets. Authentication may require user approval, after which the agent can continue its workflow and create or update files through the connected service.

Q: How much human involvement can Runner H require?

Runner H provides three selectable levels of human participation: highly involved, moderately involved, and fully automated. This lets users choose how closely they want to supervise an agent's browser work. The product can also accept uploaded documents as context, display generated materials in a files area, and use authenticated service connections during task execution.

Summary & Key Takeaways

• Runner H lets users assign web tasks that its agents execute through browser interactions, including clicking, scrolling, collecting information, and creating files. The demonstration searches eBay for Pokémon cards, extracts ten listings, connects to Google Sheets after authentication, and autonomously creates a spreadsheet containing the results and available links.

• Surfer H consists of a policy, localizer, and validator. The policy proposes sequential actions, the localizer converts textual descriptions of interface elements into two-dimensional click coordinates, and the validator evaluates the resulting answer. Failed validation produces feedback that enters memory, allowing execution to continue within established time or cost limits.

• Hollow-1 is an open-weight family of lightweight vision-language models for web navigation, interface localization, and information extraction. The presented comparisons show strong click accuracy and cost efficiency. Surfer H paired with Hollow-1 7B reportedly reaches 92.2% accuracy for 13 cents per task, outperforming the cited alternative configurations.