How SolarWinds Responded to the SUNBURST Attack

TL;DR

SolarWinds began its response after FireEye reported that tainted Orion code had been shipped and provided enough evidence for rapid validation. The company engaged outside counsel, CrowdStrike, and KPMG, then combined legal coordination, forensic investigation, malware reverse engineering, remediation, and outreach to law enforcement and the NSA while working through exceptionally long days.

Transcript

Hello and welcome. You are in for a real treat. My name is Diana Kelly, and I am so excited to be hosting this panel. It's on SolarWinds: the detailed account of the incident response. So a lot of times you've probably heard a lot of, of reports and, and news and headlines, and you've heard people talking about what happened in the incident respons... Read More

Key Insights

• FireEye's initial report contained enough information for SolarWinds to validate quickly that tainted code had been shipped, reducing the need for extensive preliminary research before the investigation began.
• Three software builds were identified as affected during SolarWinds' early analysis, giving the response team an initial boundary for investigating the compromised software and the internal activity behind it.
• Outside counsel was engaged before the technical response firms, with DLA Piper directing the investigation and working to establish attorney-client privilege through formal agreements with the participating teams.
• CrowdStrike's early role included obtaining and reverse engineering the SUNBURST backdoor, determining how it operated, what actions it supported, and what its presence could mean for potential victims.
• KPMG led its portion of the forensic investigation after joining close to a week into the response, while CrowdStrike handled reverse engineering, malware analysis, remediation activity, and relevant threat-intelligence guidance.
• SUNBURST used complex command-and-control channels, including DNS to narrow down victims and multiple protocols intended to resemble legitimate SolarWinds communication patterns.
• The implant contained loops and analytical tricks that made its behavior difficult to understand, and the complete reverse-engineering effort required several days rather than producing immediate answers.
• Large incident responses require simultaneous legal, forensic, intelligence, remediation, and government-outreach work, with participants reporting 17-hour, 18-hour, and 20-hour days during the first two weeks.

Questions & Answers

Q: How did SolarWinds first discover the tainted Orion code?

FireEye called SolarWinds on a Saturday and reported that the company had shipped tainted code. FireEye supplied enough supporting information for SolarWinds to confirm the report quickly without extensive side research. SolarWinds then analyzed when the compromise occurred, determined that three builds were affected, assembled internal personnel, and began bringing external specialists into the investigation.

Q: Who participated in the SolarWinds incident response?

SolarWinds security leader Tim Brown was involved from the beginning and throughout the investigation. DLA Piper served as outside counsel and directed the investigation. KPMG led its forensic investigation work, while CrowdStrike provided incident-response support, reverse engineering, malware analysis, threat-intelligence guidance, and remediation activity. The teams coordinated closely with SolarWinds leadership during the response.

Q: Why was outside counsel engaged during the SolarWinds response?

DLA Piper was engaged to direct the investigation and organize the work under attorney-client privilege. Ronald Plesco and his team prepared agreements with the external responders, including CrowdStrike, and later helped bring KPMG into the effort. Counsel also supported coordination around multiple early priorities, including outreach to law enforcement and the NSA.

Q: What did CrowdStrike investigate about the SUNBURST backdoor?

CrowdStrike rapidly obtained the SUNBURST backdoor and began reverse engineering it. The team sought to determine how the implant worked, what it could do, and what it meant for potential victims. After examining its behavior and communications, investigators also turned to the question of how the malicious code had been dropped or planted in the software.

Q: How did the SUNBURST command-and-control system work?

SUNBURST used complex command-and-control channels. According to the panel, it used DNS to narrow down or select victims and supported several protocols designed to mimic SolarWinds communication protocols. The adversary also included loops and other analytical obstacles, making the implant difficult to examine and delaying a complete understanding of its functions and behavior.

Q: How long did reverse engineering the SUNBURST implant take?

Fully reverse engineering the complete SUNBURST implant took several days. Investigators could begin analyzing the backdoor rapidly once they obtained it, but its complex command-and-control channels, multiple communication protocols, loops, and deliberate analytical tricks prevented an immediate understanding. The team worked through the code while participating in a continuous series of response calls.

Q: How intense was the early SolarWinds incident response?

The early response required exceptionally long working hours. SolarWinds personnel gathered virtually on Saturday and were in the office by Sunday, checking details until around 2:00 in the morning on multiple nights. Participants described working 17-hour, 18-hour, and 20-hour days during the first two weeks because many legal, technical, investigative, and government-outreach tasks needed attention.

Q: What incident-response lessons emerged from the SolarWinds investigation?

The account shows that prior incident-response testing helped but did not fully prepare the organization for an investigation of this size and scope. Effective response required rapid validation, identification of affected builds, early engagement of counsel, specialized technical teams, clearly divided responsibilities, and sustained coordination. Malware analysis, forensic investigation, remediation, legal work, and external outreach all proceeded together.

Summary & Key Takeaways

• FireEye contacted SolarWinds on a Saturday and reported that the company had shipped tainted Orion code. The supplied information enabled rapid validation, and SolarWinds determined that three builds were affected. Internal leaders then assembled an investigation and began coordinating the extensive technical, legal, and communications work required by the incident.

• Outside counsel DLA Piper directed the investigation and worked to place the response under attorney-client privilege. CrowdStrike was engaged quickly for incident response, intelligence, reverse engineering, malware analysis, and remediation. KPMG joined close to a week later and led its forensic investigation work, with the participating teams operating under counsel's direction.

• CrowdStrike obtained the SUNBURST backdoor and began analyzing its behavior, potential effects, and origin. The implant used complex command-and-control mechanisms, including DNS for selecting victims and protocols designed to resemble SolarWinds communications. Its loops and analytical obstacles meant that fully understanding the implant required several days of reverse engineering.