How Did T3 Chat's Database Migration Cause an Outage?

TL;DR

T3 Chat became almost entirely unusable for a couple of hours because Convex's WebSocket connection layer failed, preventing chats from loading and making new chat creation unreliable. The outage followed a migration from PlanetScale MySQL and browser-based IndexedDB storage to Convex, a change that effectively rewrote the application and replaced manual synchronization with live query subscriptions.

Transcript

If you all have watched my channel for a while, there's two things you almost certainly know. One is that I've been changing databases in T3 Chat too much. And the other is that I really, really, really don't like downtime. As such, this is kind of a sad video to make because we had our first major outage with T3 Chat. In the past, we've had blips ... Read More

Key Insights

• T3 Chat's first major outage made the site almost entirely unusable for a couple of hours. Chats did not load, creating new chats barely worked, and the broader application experienced severe lag during the incident.
• The immediate failure occurred in the WebSocket connection layer used by Convex. T3 Chat clients depend on that layer to establish query subscriptions, receive initial records, and obtain subsequent updates when application data changes.
• The migration was driven by the burden of rebuilding synchronization features on top of PlanetScale MySQL. Maintaining that custom system was not practical for a team described as having roughly two and a half to three and a half people.
• The database change was effectively a rewrite of T3 Chat. The application contained about 25,000 lines of code, while the migration pull request changed 10,000 lines and introduced a substantially different codebase and user experience.
• The previous architecture retrieved all of a user's threads and messages through a Next.js and tRPC backend. The client then persisted the returned information in IndexedDB, a browser storage system that repeatedly caused problems across browsers, Safari, and multiple tabs.
• Convex works through client subscriptions to TypeScript query functions. A query reads the database and returns initial results, while an active WebSocket connection delivers updated data after changes such as a renamed thread.
• Real-time synchronization simplified difficult state-management cases. The earlier architecture had to coordinate the client, backend, database, and multiple devices while handling deletions, unfinished generated messages, interrupted clients, and the risk of deleted information reappearing.
• The rollout strategy used a hard cutoff because the new implementation was not merely an interchangeable data store. The plan was to move everyone to the new deployment on a chosen day and import each user's existing data after that user signed in or opened the page.

Questions & Answers

Q: What caused the major T3 Chat outage?

The immediate cause was a failure in the WebSocket connection layer used to connect T3 Chat clients with Convex. Because Convex delivers query results and later changes through subscriptions over that connection, the failure prevented chats from loading normally. New chats barely worked, and the rest of the application became extremely slow, leaving the site almost entirely unusable for a couple of hours.

Q: How does T3 Chat use Convex to synchronize data?

T3 Chat clients query thread data by calling TypeScript functions running through Convex. Those functions read the Convex database and return the requested records through a subscription. If something later changes, such as a backend update that renames a thread, the WebSocket connection sends the updated thread data to the user without requiring the client to poll the backend for changes.

Q: Why did T3 Chat migrate away from PlanetScale MySQL?

PlanetScale itself was not presented as the fundamental problem. The difficulty was that T3 Chat was effectively rebuilding Convex-like synchronization behavior on top of its MySQL database. That custom work was not maintainable for a team described as having roughly two and a half to three and a half people, while Convex already had a larger team focused specifically on data synchronization.

Q: Why was the Convex migration considered an application rewrite?

The migration changed far more than the location where records were stored. T3 Chat had about 25,000 lines of code, and the migration pull request changed 10,000 of them. It introduced a different architecture, deployment, and user experience, so the old and new implementations could not simply share a standard feature flag that redirected database writes between two interchangeable systems.

Q: How did T3 Chat store and retrieve data before Convex?

The previous system placed a Next.js and tRPC backend between the client and a traditional MySQL database hosted on PlanetScale. When a user requested synchronized data, the backend used a database call to retrieve information, commonly all threads and messages for that user. It sent the records back to the browser, where they were persisted locally in IndexedDB.

Q: Why did IndexedDB become a problem for T3 Chat?

IndexedDB became a persistent source of complexity because T3 Chat had to make browser-local data behave consistently across different environments. The team encountered numerous issues involving browser differences, Safari, and multiple tabs. Although the Dexie library made IndexedDB easier to use, the underlying storage approach still created enough operational and synchronization pain that it was no longer suitable.

Q: What synchronization problems was Convex intended to solve?

Convex was intended to reduce the number of manual synchronization paths connecting users, the backend, the database, and multiple devices. The earlier design struggled with cases such as deleting a thread while the server was generating a response, opening deleted content on another device, deleted messages reappearing, and generated messages stopping partway through without restarting properly.

Q: How was the T3 Chat migration planned for users?

The team planned a hard cutoff rather than gradually switching database traffic behind a normal feature flag. On a selected day, everyone would begin using the new codebase. When each user signed in or opened the page for the first time, the system would pull that user's existing data from the old MySQL database into the Convex synchronization engine.

Summary & Key Takeaways

• T3 Chat experienced its first major outage on a Sunday, leaving the service almost entirely unusable for a couple of hours. Chats failed to load, new chats barely worked, and the application lagged heavily because the WebSocket connection layer used to connect clients with Convex had fallen over.

• The migration replaced a traditional MySQL database hosted on PlanetScale and a browser-based IndexedDB cache with Convex's synchronization engine. Previously, a backend retrieved threads and messages, returned the data to users, and stored it locally. Browser differences, Safari behavior, and synchronization across multiple tabs made that architecture difficult to maintain.

• Moving to Convex was effectively an application rewrite rather than a simple database replacement. T3 Chat contained about 25,000 lines of code, and the migration pull request changed 10,000 lines. Because the old and new systems produced different user experiences, the team planned a hard cutoff instead of a conventional database feature flag.