Why Does GitHub Actions Waste CI Compute?

TL;DR

A faulty GitHub Actions runner loop could consume an entire CPU while waiting and could run forever if Bash’s SECONDS variable skipped the exact target value. A proposed comparison change prevented the endless loop, but the fix was automatically closed before being merged roughly a year and a half after the problem became an issue, despite reported financial and operational damage.

Transcript

Now, I do not like making fun of people's code, okay? We've all been there. We've all written code under duress, shall we say, and it's not impressive. You know, caught absolute cinema. I've been caught on many a PR trying to sneak in some just truly heinous lines of code. So, therefore, I'm not going to make fun of some random person's code, excep... Read More

Key Insights

• The central GitHub Actions bug was an equality-dependent busy-wait loop that continued until Bash’s SECONDS variable exactly matched the requested duration. If system load caused the counter to skip that precise value, the loop could continue forever instead of completing its intended wait.
• The safe-sleep function originally tried the system sleep utility first, used ping as a fallback, and otherwise echoed to dev null 5,000 times. The ping approach sent five packets but waited only about four seconds because the first packet was sent at second zero.
• The later safe-sleep implementation consumed an entire CPU while waiting because its loop repeatedly checked elapsed time without yielding. On a runner described as having only two CPUs on average, this waiting behavior could occupy half of the available compute and worsen scheduling delays.
• The endless-loop risk arose because the condition tested whether the elapsed time was not equal to the target. A delayed loop iteration could move from below the target to above it, meaning the exact terminating value was never observed and the condition remained true indefinitely.
• The proposed fix changed the loop’s stopping logic from exact inequality to a bounded comparison. This allowed the function to terminate once the elapsed time reached or exceeded its requested duration, preventing a skipped counter value from creating a permanently running process.
• A documented GitHub Actions runner remained active for 5,135 hours because of the faulty waiting loop. Using the transcript’s cheapest stated rate of 8 cents per CPU minute, the presenter estimated that this single process cost about $2,400 while burning CPU without useful work.
• The proposed correction was automatically closed about a month after submission, despite addressing the hanging behavior with a small comparison change. According to the transcript, GitHub merged the fix roughly a year and a half later, after the problem became a broader issue.
• GitHub Actions reliability problems affected real project workflows, according to the examples discussed. Zig moved to Codeberg after describing neglected Actions bugs, seemingly random job scheduling, backlogged continuous integration, and an inability to ensure that even master branch commits received checks.

Questions & Answers

Q: How could the GitHub Actions sleep loop run forever?

The loop waited while Bash’s SECONDS variable was not equal to the requested duration. Because SECONDS increases as time passes, a heavily loaded machine could delay an iteration long enough for the value to jump past the exact target. Once that happened, equality could never occur, so the condition stayed true and the process continued running indefinitely.

Q: Why did the GitHub Actions wait function waste CPU?

The function implemented waiting as a busy loop that repeatedly checked whether enough time had passed. It did not simply yield execution through the normal sleep utility. Consequently, the loop could consume an entire CPU while performing no useful work. With an average runner described as having two CPUs, waiting alone could occupy half of its available compute.

Q: What was the proposed fix for the endless runner loop?

The proposed fix replaced a condition that required elapsed time to equal one exact value with a bounded comparison that stopped after the duration had been reached or exceeded. This small change handled delayed iterations safely. Even if Bash’s SECONDS variable moved past the target, the loop could recognize completion and exit instead of running forever.

Q: How much did one reported GitHub Actions loop cost?

The example cited a single GitHub Actions runner that remained active for 5,135 hours. The transcript states that the cheapest machine cost 8 cents per CPU minute and estimates that the dead sleeping process cost about $2,400. The expense came from a loop burning CPU continuously while waiting rather than performing productive continuous integration work.

Q: What did the original safe-sleep function do?

The original function attempted several waiting methods. It used sleep for five seconds when that command was available. If sleep was unavailable but ping existed, it sent five packets, producing roughly a four-second wait because the first packet left at second zero. Its final fallback echoed to dev null 5,000 times, consuming CPU as a crude delay.

Q: Why was the ping fallback shorter than five seconds?

The function sent five ping packets at intervals of about one second. The first packet was sent immediately at second zero, the second at second one, and the sequence continued from there. Because only the gaps between transmissions created elapsed waiting time, five packets produced approximately four seconds of delay rather than the intended five seconds.

Q: How quickly did GitHub merge the proposed loop fix?

GitHub did not merge the change promptly, according to the transcript. The pull request proposing the comparison fix was automatically closed about a month after submission. After the problem later became an actual issue, the change was merged roughly a year and a half later. The delay is presented as evidence that GitHub Actions maintenance was being neglected.

Q: Why did Zig move its continuous integration away from GitHub?

Zig moved from GitHub to Codeberg after describing GitHub Actions as having inexcusable and neglected bugs. Its cited complaint said jobs appeared to be scheduled seemingly at random, other bugs compounded the problem, and maintainers could not manually intervene. The resulting continuous integration backlog became severe enough that even master branch commits could not reliably be checked.

Summary & Key Takeaways

• GitHub Actions runners contained a safe-sleep function whose fallbacks evolved from calling sleep, using ping, or repeatedly echoing to dev null into a Bash busy-wait loop. The resulting implementation consumed CPU while waiting and depended on reaching one exact value, creating both wasted compute and the possibility of an endless process.

• The loop compared Bash’s incrementing SECONDS variable with a requested duration using inequality. On machines handling long, difficult jobs, more than one second could pass between loop evaluations. If the counter skipped the target, the condition never became false. Even without that failure, the loop could occupy one full CPU merely to wait.

• A contributor proposed changing the comparison so the loop ended after reaching or passing its target. The pull request showed a runner that had continued for 5,135 hours, with an estimated cost of about $2,400 at the stated cheapest rate. The proposal was automatically closed, then merged roughly a year and a half later.