How Does Vercel's Active CPU Billing Work?

TL;DR

Vercel's active CPU billing better matches the cost profile of AI applications by distinguishing active computation from long periods spent waiting on streamed model responses. The broader release also introduces queues, sandboxed execution for user-submitted or AI-generated code, BotID for bot protection, and an AI gateway, addressing several practical infrastructure needs in modern web applications.

Transcript

It happened. Verscel actually shipped. I feel like it's been a while since Verscell dropped a bunch of things at once, like many years by now. The last two Next.js comps were relatively low in terms of new features both for Next and for Versell. That changed today. They dropped a ton of things I've been personally waiting for forever. From a new pr... Read More

Key Insights

• Active CPU billing is designed to charge more closely for time spent computing instead of treating an entire long-running request as equally intensive. This matters for AI endpoints that remain open while receiving and forwarding streamed tokens but use extraordinarily little CPU during much of that period.
• Traditional server hosting is based on provisioned capacity with a fixed monthly cost, whether the server receives no users or substantial traffic. When concurrent demand exceeds that server's capacity, operators must add servers, manage startup and shutdown timing, or risk delaying and losing traffic.
• Serverless infrastructure works by creating a separate instance for each request, keeping it alive only as long as the request needs, and removing it afterward. This model is well suited to short web operations because applications pay for accumulated request duration without continuously provisioning peak capacity.
• Gigabyte-hour billing measures how many gigabytes of memory remain allocated over a given duration. That approach can be economical for requests lasting roughly 100 to 200 milliseconds, but it becomes less attractive when an application keeps many low-compute connections open for much longer periods.
• Workload cost depends on both duration and CPU intensity, not duration alone. Server-side rendering has medium processor usage and low duration, while image re-encoding has high usage and medium duration. A thousand requests waiting for database responses may require less computation than 10 simultaneous image conversions.
• Large language model proxy endpoints are high-duration, low-CPU workloads because they assemble user context, call an external model API, receive its stream, and forward that stream to the user. The actual model inference occurs elsewhere, so the application endpoint mostly waits and transfers data.
• Cloudflare Workers are presented as infrastructure already suited to measuring active CPU rather than total elapsed request time. The transcript claims CPU-bound rendering tasks can take two to four times longer there than on Vercel or Lambda with Node, illustrating a tradeoff between processor performance and billing structure.
• Vercel's broader release addresses several distinct application needs through queues, sandboxed execution, BotID, and an AI gateway. Sandboxing targets user-submitted or AI-generated code, while the bot protection offering reflects lessons from practical concerns involving captchas and rate limiting.

Questions & Answers

Q: How does Vercel's active CPU billing help AI apps?

Vercel's active CPU billing helps AI applications by focusing charges more closely on the periods when an endpoint is actively using the processor. An AI proxy request may remain open for around 20 seconds while an external model generates tokens, yet the endpoint spends much of that time waiting and forwarding a stream. Charging for active computation better reflects this workload than charging uniformly for total elapsed duration.

Q: Why can duration-based serverless billing be expensive for AI responses?

Duration-based serverless billing can become expensive when many requests remain open for streamed AI responses. A conventional database-backed page might complete in roughly 100 to 200 milliseconds, while a model response can take around 20 seconds. Even though the proxy endpoint uses extraordinarily little CPU while waiting for the external API, its allocated instance remains alive, so accumulated duration grows quickly across active users.

Q: What is the difference between traditional servers and serverless computing?

A traditional server has a fixed cost and provisioned capacity regardless of whether it receives no users or heavy traffic. If demand exceeds its concurrency limit, more servers must be started and later stopped. Serverless computing instead gives requests separate short-lived instances and scales the number of provisioned instances with current demand, reducing the need to maintain enough idle capacity for peak traffic.

Q: What does gigabyte-hour billing measure in serverless platforms?

Gigabyte-hour billing measures the amount of memory allocated to a workload over the time that workload remains active. Under the model described, each request receives a separate instance that lives until the response completes. The approach works well for short requests because many small durations can still cost less than keeping a large server provisioned continuously, especially during periods with little or no traffic.

Q: Why are CPU intensity and request duration separate concerns?

CPU intensity and request duration describe different resource demands. A request can finish quickly while performing meaningful computation, or remain open for a long time while doing almost none. Server-side rendering is described as medium usage with low duration, image encoding as high usage with medium duration, and an AI streaming proxy as extraordinarily low usage with high duration. Billing and capacity decisions should account for both dimensions.

Q: How does an AI streaming endpoint use server resources?

An AI streaming endpoint receives the user's request, attaches metadata, memory, or other context, and sends the resulting prompt to an external service such as the OpenAI API using the application's API key. It then receives the generated stream and forwards it to the user. The endpoint is not performing model inference itself, so it remains open for a long period while using relatively little CPU.

Q: What new infrastructure features did Vercel announce?

Vercel announced active CPU billing for Fluid Compute, queues in limited beta, sandboxed code execution, BotID for bot protection, and an AI gateway. The sandbox is intended for user-submitted or AI-generated code, while the gateway is presented as an alternative to a service such as OpenRouter. Together, the features target cost management, background work, security, bot handling, and access to AI providers.

Q: When is serverless computing a good fit for a web application?

Serverless computing is a good fit when an application handles short, intermittent requests and needs capacity to follow current traffic automatically. Examples include retrieving database records, generating HTML, and sending responses within roughly 100 to 200 milliseconds. Because instances disappear after their requests finish, the application avoids paying continuously for idle peak capacity and avoids manually starting additional servers whenever concurrency rises.

Summary & Key Takeaways

• Vercel released several infrastructure features aimed at workloads that have become common in AI applications. The announcements include active CPU billing for Fluid Compute, queues in limited beta, sandboxed code execution, BotID for bot protection, and an AI gateway positioned as an alternative to services such as OpenRouter.

• Traditional servers have fixed costs regardless of traffic, but they require operators to provision enough capacity for peak concurrency. Serverless platforms instead create separate instances for requests and commonly charge according to memory and duration, which works well when web requests finish within roughly 100 to 200 milliseconds.

• AI response endpoints expose a weakness in duration-based serverless billing. They may stay open for around 20 seconds while forwarding a model's streamed output, despite performing little computation during most of that interval. Active CPU billing is compelling because it separates waiting time from actual processor usage.