You’re an airline pilot, responsible for 290 souls who are seated a few feet behind your cockpit door, enjoying a meal. You’re flying a state of the art Boeing 787 on a long-haul flight from San Francisco to Sydney. You’re cruising at 41,000 feet, flying over water, thousands of miles from the nearest land. You see storms looming out of the cockpit windows, the Intertropical Convergence Zone. Suddenly, most of your cockpit’s instrumentation lights go dark. You no longer have an indication of your altitude or airspeed. You don’t know the condition of your engines or other mechanical equipment. You don’t have communications with the outside world. You’re flying blind, at night, over water, far from shore, with a gigantic responsibility on your shoulders: 290 souls, your crew, a $300 million dollar aircraft, the reputation of your company, your colleagues, and maybe even the reputation of your country. Airline pilots are trained to deal with casualties midflight. They’re trained not to panic. They’re trained to follow their procedures in a structured and methodical fashion. Still, they’re human, and somewhere in the back of their minds, they understand the gravity of the situations they face and of their grave responsibilities.
The feeling that an airline pilot likely felt in the example above is akin to the feeling that any major corporate CIO/CTO would have if he or she were running the technology infrastructure for a multi-million dollar company who was forced to turn off his monitoring and alerting systems on short notice, as CISA recommended during the peak of the Solarwinds breach in December of 2020. These are the systems that are monitoring sometimes nationally critical services to customers, generating the revenue for the company that is needed for the payroll of thousands, and to satisfy investor obligations. Without alerting and monitoring, you’re blind to much of the key information that you require to keep your enterprise operating in a safe condition. Without alerting and monitoring, you lose the ability to see other threats.
Two items glaringly stood out from last year’s Solarwinds breach: 1) the vector of the attack, targeting an alerting and monitoring platform, and 2) the fact that Solarwinds is reported to have used a weak password on their update server, “Solarwinds123”.
Attack Vector
At the risk of mixing metaphors, one of the first actions that the US military took during the invasion of Iraq in 1991 was to eliminate Saddam Hussein’s radar installations. Coalition military forces did not want Iraq to be able to see or target coalition aircraft. Similarly, a serious cyberthreat actor working to attack critical infrastructure would first intelligently seek to blind his opponents by denying them the use of endpoint management technologies and monitoring and alerting technologies.
The Solarwinds attack might have been a template for how future cyberwarfare, cyberespionage, and cybertheft attacks might begin. Information security professionals should be alert to the reality that threat actors might look to compromise the very tools that we use to protect our enterprises as a first step and as a precursor to a broader attack. The mechanism to protect from such attacks exist. Diversify your security portfolio; use network segmentation and leverage multiple tools and technologies so that if you have a breach in one, you still retain coverage in another. This of course comes at the added cost of complexity in the environment, the lack of centralization, as well as additional costs associated with having staff that must be trained in multiple products. So, the answer is easy but it doesn’t come without costs.
Compliance
The second glaring issue that arose from the Solarwinds attack deals with compliance. Virtually every information security and compliance program standard like ISO27001, NIST, TPN, CMMC, SSAE18, TSC, etc. have a core component that deals with Access Control. Virtually every auditing and certification group will evaluate a company’s Access Control policy and its execution.
Solarwinds’ website states that their datacenters pass a SOC2 Type 2 audit:
Access to areas where systems, or system components, are installed or stored are segregated from general office and public areas. The cameras and alarms for each of these areas are centrally monitored 24×7 for suspicious activity, and the facilities are routinely patrolled by security guards. Servers have redundant internal and external power supplies. Data centers have backup power supplies, and can draw power from diesel generators and backup batteries. These data centers have completed a Service Organization Controls (SOC) 2 Type II audit and are SSAE16 accredited.
As a CISO responsible for information security and compliance for a global organization, I deliver SOC2 Type 2 reports so I understand the seriousness with which these audits are conducted, which are designed to provide a degree of assurance to the organization and give confidence to customers that our systems are being operated in the manner in which they were designed. Was the Solarwinds SOC2 current? Did the Solarwinds SOC2 cover the correct controls like access control?
Solarwinds also has a section on its website that addresses Access Control:
Access Controls – Role Based Access
Role based access controls are implemented for access to information systems. Processes and procedures are in place to address employees who are voluntarily or involuntarily terminated. Access controls to sensitive data in our databases, systems, and environments are set on a need-to-know / least privilege necessary basis. Access control lists define the behavior of any user within our information systems, and security policies limit them to authorized behaviors.
Authentication and Authorization
We require that authorized users be provisioned with unique account IDs. Our password policy covers all applicable information systems, applications, and databases. Our password best practices enforce the use of complex passwords that include both alpha and numeric characters, which are deployed to protect against unauthorized use of passwords. Passwords are individually salted and hashed.
SolarWinds employees are granted a limited set of default permissions to access company resources, such as their email, and the corporate intranet. Employees are granted access to certain additional resources based on their specific job function. Requests for additional access follow a formal process that involves a request and an approval from a data or system owner, manager, or other executives, as defined by our security guidelines. Approvals are managed by workflow tools that maintain audit records of changes.
The statement does not explicitly call out that passwords are changed on a regular basis but the statement does cover strong passwords. It seems like Solarwinds has a password policy and that their procedures are audited. So, what happened? Do they have an effective program that is auditing all of the areas needed or are the words on the Solarwinds website simply boilerplate?
Reports are that cybersecurity professionals had warned Solarwinds since 2019 of a known vulnerability of using simple passwords for the Solarwinds update server. As a CIO friend recently stated to me, “Was the convenience of access for an $85,000 a year engineer worth putting the entire company and the ecosystem of important customers that the company served at risk?” Did the engineers who use the simple password, themselves professionals, not recognize the significant risk to their actions? Were the engineers trained in best practices? How did this simple password control break so badly? Why was this shortfall never uncovered by their audits? Why did warnings of this vulnerability go unaddressed?
Third-Party Vendor Management
This brings us to our final topic: Third-Party Vendor Management. Solarwinds claims to have a SOC2 Type II report. How many of the customers who use Solarwinds actually obtain the SOC2 report and read it? Was the report even available to customers? How many customers asked probing questions for any gaps within the report?
Third-Party Vendor Management programs generally fall into Governance, Risk, and Compliance, but sadly most GRC departments are understaffed and overworked. Some departments leverage technology tools but many third-party vendor management programs quickly devolve into mindless questionnaire exercises that create busywork to gather answers to questions from our vendors but which do little to help us understand the true risk associated with a product or vendor. You buy a tool and you claim victory. Some of these programs give us a sense of comfort that we’re doing something, but the reality is that in many cases the back doors could be wide open and we would not even know it — that should not be acceptable to any CISO. In this case, the back door was wide open.
Conclusion: A Wakeup Call
Solarwinds is a wake-up call and should represent a watershed moment for cybersecurity, compliance, and third-party vendor management. It merits more than a superficial reaction by the industry. First, the Solarwinds breach shows us the strategies and tactics that threat actors will likely employ in the future: target monitoring and infosec toolsets first, before launching other attacks. We must architect our security and monitoring solutions so that they are more resilient to these kinds of attacks: network segmentation, different toolsets from different vendors, backup plans for when a critical toolset is compromised, etc.
Second, the Solarwinds breach shows us the need to revisit our information security and compliance programs to gauge their effectiveness. It’s not good enough to pass an audit. Just because a program is audited does not mean that the correct controls are being evaluated or that adequate compensating controls are present. As cybersecurity and compliance professionals, it is not good enough for us to have programs “in name only”. It’s not good enough for us to put the boilerplate on our websites. We must continuously strive to make our environments better. No system is perfect, but we should demand excellence and continuous improvement every single day because there are very bad people out there who will take advantage if given even the smallest opportunity. We must have continuous improvement and excellence in our programs that elevate the bar on information security and compliance. As the threats evolve, so too must audit and compliance evolve.
Finally, Solarwinds should have us all revisit our third-party vendor management programs. If your organization uses Solarwinds and you or someone in your company never obtained the SOC2 Type II report and critically pored over it, looking for gaps, then you should ask yourself, “Why not?” The answer for most organizations is that their GRC departments are understaffed and overworked. At the end of the day, we can all blame Solarwinds for a breach, but the reality is that we all share some of the blame. When it comes to information security and compliance, the buck stops with the CISO, and it’s up to the CISO to make the case for the appropriate support to protect their enterprise.
About the Author:
Joe Marroquin is a CISO and cybersecurity professional. A former U.S. naval officer and combat veteran, Joe spent the last 25 years consulting for Fortune 500 and Global 50 corporations. A c-suite executive and board advisor, Joe is a published author and public speaker who writes and speaks on security and on the intersection of business and disruptive technologies.