Why Traditional Logging Fails at Scale (and How to Fix It)

TL;DR

Traditional logging breaks down at scale because logs are interleaved by time, not grouped by request, so once thousands of concurrent users hit a system it becomes impossible to tell which log line belongs to which request. String-searching by user ID fails because not every log includes the ID and formats are inconsistent. Logs were designed for a single-server, monolith era that no longer matches how distributed systems work.

Transcript

If you ever ask my co-workers what it's like to work with me, they'll have a lot to say. But if you specifically ask for any weird things or bad habits I have, there's one in particular almost every engineer will bring up after working with me. I leave a lot of logs around, like far too many. In the T3 chat codebase alone, I think we had to clean u... Read More

Key Insights

• Traditional logging is fundamentally broken because it was designed for an era of monoliths, single servers, and problems you could reproduce locally, while today a single user request might touch 15 services, three databases, two caches, and a message queue.
• Logs fail you when something goes wrong because they are missing the one thing you actually need: context. They give timestamps and vague messages instead of a coherent story of what happened for a specific request.
• At scale, logs become unreadable because they arrive in the order events happen. With just 13 concurrent users it is already impossible to tell which user did what, and at 10,000 concurrent users you get roughly 130,000 log lines per second, most saying nothing useful.
• String search does not solve the problem because you only find logs where you remembered to attach the user ID, and even those appear in inconsistent formats like 'user [id]', 'userId=id', and 'user: userId', so grepping misses many relevant lines.
• OpenTelemetry does not fix these problems for the speaker; he says hotel only helps with looking at a trace on one box and legitimately prefers console.log to OpenTelemetry for most things.
• Distributed systems make log correlation hard because different parts run on different boxes. A gateway sits in front while inventory runs on a separate Lambda, and getting logs from there back to a main box to be grouped as one cohesive request is extremely difficult.
• Real debugging complexity comes from state scattered across many places. The T3 Chat wrapped feature spanned UI state, a PostHog feature flag that ad blockers could silently block, the Convex database, its query layer, and its workflows, making it hard to know what happened where.
• Production logs look worse than dev logs because concurrent requests get intermingled. When two things happen at the same time they log as they happen, so you cannot cleanly see what occurred for a given single request.

Questions & Answers

Q: Why is traditional logging considered fundamentally broken?

Logging as commonly practiced was designed for a different era of monoliths, single servers, and problems you could reproduce locally. Today a single user request might touch 15 services, three databases, two caches, and a message queue, but logs still act like it is 2005. They produce timestamps and vague messages while missing the context you actually need when something goes wrong, so they fail exactly when you rely on them.

Q: What happens to logs when many users hit a system at the same time?

Logs arrive in the order events happen, so concurrent requests get intermingled. With just 13 users it becomes impossible to see which user did what, because the logs come in as things happen rather than grouped by request. Scaled to 10,000 concurrent users you get roughly 130,000 log lines per second, most saying nothing useful, and production logs end up looking far worse than dev logs because requests are mixed together.

Q: Why does searching logs by user ID often fail?

When a user reports they cannot complete a purchase, the instinct is to search logs for their email or user ID. But you only get back the logs where you remembered to attach the user ID, and any logs without it are missed. On top of that, the IDs appear in inconsistent formats such as 'user [id]', 'userId=id', and 'user: userId', so a single string search cannot reliably catch every relevant line.

Q: Does OpenTelemetry fix the problems with logging?

According to the presenter, no. He says slapping OpenTelemetry on your codebase will not magically fix logging, and he has many issues with it. He states he legitimately prefers console.log to OpenTelemetry for most things. He argues it does not solve the core problems other than letting you look at a trace on one box, which does not help when different parts of a request run on different machines.

Q: Why is logging harder in distributed systems than on a single server?

In distributed systems different parts of a request run on different boxes. For example, a gateway sits as the layer in front, but inventory might be a separate system running on a Lambda. Getting the logs from there back to the main box to be grouped as one cohesive thing is very difficult. Logs still behave as though everything runs on one server with the database on top of it, which is no longer how anything works.

Q: What made the T3 Chat wrapped feature so difficult to debug?

The feature had state scattered across many places. There was UI state showing what the user saw, a feature flag from PostHog that would be blocked if you had an ad blocker on, the Convex database where data is stored, the Convex query layer, and Convex workflows running the generations on top of PostHog data. With data spread across roughly five parts of Convex, the front end, Vercel, and PostHog, it was hard to know what was happening anywhere.

Q: Why do production logs look worse than development logs?

In production, many different requests get intermingled because logging happens as events occur in real time. If two things happen at the same moment, both are logged when they happen, so you cannot clearly see what happened for a given single request. In development you usually have one request at a time, so the flow reads cleanly, but production interleaving destroys that clarity and makes step-by-step reading useless.

Q: What does a single-user logging flow look like in the chaos simulator?

For one user hitting process order, the logs are easy to follow step by step. It receives the checkout request, parses the body, validates the JWT token and confirms it is valid, creates the order, validates the cart, checks item stock, processes the payment by calling the Stripe API, gets declined for insufficient funds, attempts again successfully, confirms the order, and sends the response. This linear sequence makes sense until concurrency is introduced.

Summary & Key Takeaways

• The presenter admits to logging far too aggressively, noting they had to clean up over a thousand log statements in the T3 Chat codebase alone, yet defends the habit because random logs repeatedly made debugging unexpected behavior far easier.

• The core argument, drawn from Boris Tane's article at loggingsucks.com, is that logging as commonly practiced is broken because it assumes a single-server monolith era, while modern requests span many services, databases, caches, and queues that logs cannot coherently represent.

• A log chaos simulator demonstrates the failure: one user's checkout flow is easy to follow, but bumping to 13 concurrent users makes it impossible to tell which log belongs to which request, and string-searching by user ID fails due to missing IDs and inconsistent formats.