How to Automate Browser Tasks With Codex

TL;DR

Use Codex browser control to test web apps, complete repetitive website workflows, and interact with services that lack an API. Describe the task in natural language, demonstrate or supervise the flow, save it as a reusable skill, and validate it repeatedly before scheduling it, especially when financial accounts, passwords, or two-factor authentication are involved.

Transcript

Codex's browser use is probably the best that I've ever tried before. It can literally do anything on a browser or on your actual local computer. And if you need to be signed in for something, you just sign in once and Codex will save it. So, the possibilities are endless. You can actually automate anything. Now, let me show you guys how to set it ... Read More

Key Insights

• Codex browser control is an alternative for workflows that cannot be handled through an API or an available plugin. It uses visual analysis and AI to understand what appears on a webpage, decide where to click, enter information, and proceed through multi-step interfaces.
• The in-app browser supports visual annotations that connect interface feedback directly to the Codex chat. A user can select a misaligned or overlapping element, describe the defect, add more annotations, and submit those observations as specific implementation requests.
• Automated browser testing can exercise more scenarios than a person may test manually in the same period. The demonstrated agent clicked controls, entered data, tested required fields, checked keyboard submission and restart behavior, switched to mobile view, and reported passed and failed checks.
• The demonstrated QA run found real data integrity and state problems. Invalid contact information passed through the form, and changing the country code to the United Kingdom reverted to the United States after returning to edit the entry.
• Headless browser use runs in the background so the user can continue foreground work, while headed browser use exposes the agent's visible actions. The visible mode lets users watch its pointer move, follow its choices, and supervise unfamiliar or sensitive workflows.
• Reusable skills can preserve a browser workflow described in natural language. After one successful Relay run, Codex saved the steps for opening accounts, locating statements, downloading CSV files for two accounts, and placing those files into a specified folder.
• Saved login sessions are convenient but are not guaranteed to persist. Relay returned to its login screen because the banking platform ended the session, showing that scheduled routines may still encounter authentication barriers even after an earlier successful sign-in.
• Sensitive automation requires supervision and repeated validation. The presenter advises watching a banking skill run at least ten times rather than trusting a single success, and notes that two-factor authentication may require manual action even when passwords can be filled automatically.

Questions & Answers

Q: How can Codex automate tasks on websites without APIs?

Codex can use its in-app browser to visually analyze a website, identify controls, click through pages, type information, and download files. The user can describe the desired sequence in natural language, such as opening accounts, finding statements, selecting two accounts, downloading CSV files, and saving them to a folder. Once completed successfully, that flow can be saved as a reusable skill.

Q: How can Codex test a web app for bugs?

Open the app in the Codex browser and ask the agent to test it aggressively, try varied inputs, and attempt to break the workflow. In the demonstrated form, Codex checked required fields, entered valid and invalid data, tested navigation, keyboard submission, restart behavior, and mobile responsiveness. It then reported which checks passed, which failed, and what needed to be fixed.

Q: What bugs did Codex find in the demonstrated form?

Codex found that obviously invalid contact information could pass through the form and that selecting the United Kingdom country code reverted to the United States after returning to edit the entry. It also found mobile responsiveness failures. The agent continued checking keyboard submission and restart behavior because those paths can reveal bypasses that ordinary clicking may miss.

Q: What is the difference between headed and headless browser automation?

Headless browser automation runs in the background, allowing the user to continue other foreground work without watching the browser. Headed automation displays the browser and the agent's actions, including its pointer movements, clicks, typing, and navigation. Headed mode is useful when a user wants to observe or supervise a workflow, while headless mode suits longer unattended checks.

Q: How do browser annotations help fix user interface problems?

Browser annotations let a user select a specific interface element and attach a precise description of the problem. In the demonstration, annotations identified an overlapping phone-number field and a section that was not aligned with neighboring sections. Those observations appeared in the Codex chat, where they could be submitted together as focused requests for code changes.

Q: How can Codex reuse and schedule a browser workflow?

After Codex completes a workflow, it can save the sequence as a skill. The Relay example preserved the process of navigating to accounts, opening statements, downloading CSV files for two accounts, and putting them in a folder. The user could then create a scheduled routine that invokes the skill for the most recent monthly statement, subject to login and authentication requirements.

Q: How can Codex sign in without placing passwords in chat?

Codex can use an in-app browser password manager instead of receiving credentials through the chat history. The demonstrated setup imported a CSV containing a name, URL, username, password, and optional notes. Codex then retrieved the Relay credentials from the password manager, filled the login form, signed in, and continued navigating toward the accounts and statements pages.

Q: When should sensitive Codex browser automations be supervised?

Sensitive workflows, especially those involving a bank account, should be watched and tested repeatedly before being trusted. The presenter recommends observing the skill perform the task at least ten times rather than relying on one successful run. Secure platforms may terminate saved sessions, and two-factor authentication may still require manual intervention even if the password manager fills the username and password.

Summary & Key Takeaways

• Codex can navigate websites, click controls, enter information, and inspect interfaces through its in-app browser. Developers can annotate specific visual elements, request code fixes, and ask the agent to test many workflows. The demonstrated form testing uncovered validation, state retention, keyboard submission, restart, and mobile responsiveness problems.

• Browser automation is especially useful when a service lacks an API. In the banking example, Codex navigated a signed-in Relay account, opened the statements area, downloaded CSV files for two accounts, and saved the workflow as a reusable skill that could later be attached to a scheduled monthly routine.

• Saved sessions and an in-app password manager can reduce repeated manual sign-ins, although secure services may expire sessions and require authentication again. Credentials can be imported from a CSV instead of being placed in chat history. Sensitive workflows should be watched and repeated successfully before users consider broader automation.