How to Build Interactive Apps Inside ChatGPT

TL;DR

Developers can build interactive apps inside ChatGPT using the Apps SDK, which connects data, triggers actions, renders web-based interfaces, and exposes app context to the model. Built on MCP and available in preview, it supports existing-product login, conversational discovery, inline and full-screen experiences, and access to hundreds of millions of ChatGPT users.

Transcript

GOOD MORNING AND WELCOME TO DEVDAY. THANKS FOR BEING HERE IN SAN FRANCISCO, THE CITY WHERE OPENAI STARTED AND WHERE WE'RE COMMITTED TO BUILDING THE FUTURE OF AI. IT'S BEEN ALMOST TWO YEARS SINCE OUR VERY FIRST DEVDAY. WE AND MOST IMPORTANTLY ALL OF YOU, HAVE COME A LONG WAY SINCE THEN. BACK IN 2023, WE HAD 2 MILLION DEVELOPERS AND 100 MILL... Read More

Key Insights

• • OpenAI reported that its developer community grew from 2 million developers in 2023 to 4 million, while weekly ChatGPT usage increased from 100 million people to more than 800 million and API processing rose from about 300 million to over 6 billion tokens per minute.
• • The Apps SDK is a preview toolkit for building applications directly inside ChatGPT. It provides a full stack that can connect developer data, trigger actions, render fully interactive interfaces, and support experiences that users can operate through natural-language conversation.
• • The Apps SDK is built on MCP, an open standard that lets developers retain control over backend logic and frontend interfaces. An existing MCP implementation can be enhanced by adding a resource that returns an HTML template for display wherever ChatGPT is distributed.
• • App discovery works through direct invocation and contextual recommendations. A user can request an app by name, such as Figma, or ChatGPT can suggest a relevant app during a conversation, such as Spotify when the user asks for help creating a party playlist.
• • Talking to apps works by exposing application context back to ChatGPT. This allows the model to understand what the user is currently viewing or interacting with, such as the exact part of a Coursera video or a selected Zillow property.
• • Apps can render anything supported on the web and can appear inline, in picture-in-picture mode, or in a full-screen layout. The Coursera demonstration showed a video pinned above the conversation so the user could keep watching while asking contextual questions.
• • Conversational creation can carry prior context into specialized apps. In the Canva demonstration, ideas developed during a dog-walking business conversation were turned into posters and then a pitch deck, with users able to request revisions and open the results in Canva.
• • Full-screen apps can update data without creating a new instance. The Zillow demonstration filtered an existing map to three-bedroom homes with a yard, exposed the selected property to ChatGPT, and enabled questions that combined Zillow context with other tools such as search.

Questions & Answers

Q: How can developers build apps inside ChatGPT?

Developers can use the Apps SDK, available in preview, to build interactive applications that operate inside ChatGPT conversations. The SDK can connect an app’s data, trigger actions, and render a complete interactive interface. It is built on MCP and gives developers control over their backend logic and frontend presentation while supporting distribution through ChatGPT on web and mobile.

Q: What can the ChatGPT Apps SDK do?

The Apps SDK provides a full stack for connecting data, initiating actions, and displaying interactive web interfaces within ChatGPT. It supports inline content, picture-in-picture experiences, and full-screen layouts. Apps can also expose their current context to ChatGPT, allowing the model to understand what a user is viewing and respond to follow-up questions or requests accordingly.

Q: How are apps discovered inside ChatGPT conversations?

Apps can be discovered in two ways described during the keynote. A user can explicitly ask for an app by name, such as requesting Figma to transform a sketch into a workable diagram. ChatGPT can also suggest an app when it is relevant to the conversation, such as recommending Spotify when someone asks for help building a party playlist.

Q: How does the Apps SDK use MCP?

The Apps SDK is built on MCP, which the keynote describes as an open standard. Developers keep control over their backend logic and frontend interface, and the standard has been published so others can integrate it. Developers with an existing MCP implementation can enhance it by adding a resource that returns an HTML template for the app interface.

Q: What does talking to apps mean in ChatGPT?

Talking to apps refers to an Apps SDK capability that exposes application context back to ChatGPT. The model can therefore recognize what the user is currently interacting with instead of requiring a separate explanation. In the Coursera demonstration, ChatGPT understood the active part of a video and provided more detail about the data-preparation steps being discussed.

Q: What did the Coursera and Canva demonstrations show?

The Coursera demonstration showed educational video embedded within a conversation, with consent-based connection, pinned playback, and contextual questions about the current content. The Canva demonstration used ideas from an existing dog-walking business conversation to generate colorful posters and convert a selected design into a pitch deck, while allowing follow-up edits and opening the results in Canva.

Q: How did the Zillow app work inside ChatGPT?

The Zillow app fetched current property data and displayed an interactive map inside ChatGPT. A user could open a property in full screen, access expected Zillow actions such as requesting a tour, and ask ChatGPT to filter the map to three-bedroom homes with a yard. ChatGPT could also recognize the selected home and answer contextual questions about it.

Q: How large was OpenAI’s developer and ChatGPT usage in the keynote?

OpenAI reported 4 million developers, more than 800 million weekly ChatGPT users, and over 6 billion tokens processed per minute through the API. For comparison, the keynote said that in 2023 there were 2 million developers, 100 million weekly ChatGPT users, and about 300 million API tokens processed per minute.

Summary & Key Takeaways

• OpenAI introduced the Apps SDK in preview so developers can build conversational applications directly inside ChatGPT. The SDK provides tools for connecting data, initiating actions, and rendering interactive web interfaces. Built on MCP, it gives developers control over backend logic and frontend presentation while supporting ChatGPT distribution across web and mobile.

• The demonstrations showed apps being invoked by name or suggested when relevant to a conversation. Coursera combined educational video with contextual questions, while Canva generated posters and a pitch deck from prior brainstorming. These examples illustrated how apps can use conversation history, display rich media, and accept follow-up requests without leaving ChatGPT.

• The Zillow demonstration combined current property data, an interactive map, filtering, full-screen browsing, and contextual questions about a selected home. It showed how an app can update an existing interface and share the user’s current context with ChatGPT, which can then combine that information with other available tools such as search.