Reflections on the recent SolarWinds breach
So… 2020 is turning out to be the gift that keeps on giving. So much has happened within the last year both in InfoSec, and more importantly, in non-InfoSec, that we are pretty sure we will all be glad when 2021 comes along. With unexpected events coming our way in almost every single month of 2020, December has not failed to deliver.
December brought along the most newsworthy, InfoSec related story of 2020. On December 13th, 2020 the world became aware that SolarWinds’ Orion Platform’s updates were compromised by a threat actor to include malware between March 2020 and June 2020. This was categorised as a supply chain attack as the inserted malware is utilised by the threat actor to compromise systems running SolarWinds’ Orion Platform, these systems acting as a conduit to access within potentially more interesting targets. One week previous, FireEye, a prominent global information security firm, announced a breach of its systems by an advanced threat actor. After the SolarWinds announcement, FireEye released a Threat Research brief about a global threat actor that leverages SolarWinds as a means to compromise additional victims via a supply chain attack. Although FireEye did not attribute the cause of its breach to their use of the Orion platform, it is generally believed that the SolarWinds compromise presented the initial attack vector in their case as well. Other prominent targets of the attack are reportedly several US government branches such as the Treasury and Commerce departments. Commentary suggests in the region of 18,000 organisations may be impacted by the incident to various levels.
As of today, a method to neutralise the malware has been devised by Microsoft, along with a few industry partners. However, this should not be considered the end of the breach by affected entities; based on the skills and resources exhibited, it is very likely that the threat actor would have deployed and switched to alternative means of maintaining access to the compromised systems. It may be the end of this particular piece of malware, but compromised and at-risk organisations should consider a deep inspection and incident response exercise.
What are supply chain attacks?
We have mentioned above that this is categorised as a supply chain attack. It is probably useful to define that term. We have blogged about this type of attack in the past, but essentially, a supply chain attack is an attack that targets organisations (or sometimes individuals as well) indirectly, through a vulnerable element in their supply chain. This element would be used by the targeted entities in day-to-day operations and/or be generally relied upon. An attacker compromising that element may in some cases, by extension, result in compromising the targeted entities’ operations and data as well.
A supply chain attack can be highly targeted, (meaning that there is a clearly defined target at the end of it and enough reconnaissance has taken place to identify a link between who is being compromised as an intermediary and the end target), or blind (meaning that an entity is compromised in order to gain access to as many organisations that rely upon it as possible and then make as much use of those subsequent compromises as possible). There is of course the middle ground of this scenario where a specific entity is the primary target. In such an instance, the supply chain attack produces notable collateral victims while not specifically highlighting the principal target.
The case at hand
Timeline of events
Now that we have the definitions out of the way, let’s have a look at what has been published to date regarding this case.
- December 8, 2020 - FireEye announced a breach of its systems by an advanced threat actor. While FireEye did not directly name the attack vector, it did acknowledge the information believed to be sought (information about its government clients) and what was stolen (the red teaming toolkit used by its analysts on engagements). FireEye also published Indicators of Compromise (IOCs) for the stolen toolkit in its GitHub account.
- December 13, 2020 - it became known that a threat actor had compromised the updates of SolarWinds’ Orion Platform and introduced malware. The time window during which the threat actor was active and tainting updates has been reported to be between March 2020 and June 2020. At this moment, the specifics of the attack vector used by the threat actor are not known.
- December 13, 2020 - FireEye provided details of a global campaign by an advanced threat actor related to the tainted updates of the SolarWinds Orion Platform. While not directly acknowledging any link or association with their own incident within that specific announcement, it is generally believed that the SolarWinds compromise presented the initial attack vector into the FireEye environment.
- December 13, 2020 and December 14, 2020 - several other prominent organisations announced breaches related to the tainted updates of the SolarWinds Orion Platform, namely U.S. Treasury and Commerce departments and the U.S. Department of Homeland Security (DHS). This led the U.S. DHS Cybersecurity and Infrastructure Security Agency to direct all agencies to “disconnect/power down all affected SolarWinds Orion Platforms within their infrastructure”.
- December 14, 2020 - SolarWinds submitted an SEC filing stating that the compromised update was affecting approximately 18,000 customers who had applied a tainted update, out of 33,000 customers with an installation of SolarWinds Orion.
- December 17, 2020 - Microsoft, along with a few industry partners, have seized control of the domain used by the malware to reach its Command and Control server, as well as identified a kill switch to terminate the malware. This is done in effort to identify and notify all affected victims as well as terminate any old and new malware installations. Care should be taken though since this only terminates the initial callback. If the threat vector has moved on to a different callback protocol and infrastructure then the above kill switch will do little to terminate this access.
The malware introduced to the SolarWinds product line is contained in the SolarWinds.Orion.Core.BusinessLayer.dll. This is a SolarWinds digitally signed Dynamic Link Library (DLL) which is essentially a module and a valid component of the Orion Platform that, once activated, does the following:
- It enters an idle state for a period of time. Our guess is that it does this in case a mature organisation is monitoring outbound access and the callback is identified. In triaging, a 10 to 15 days old update will probably not be prioritised as the root cause of an alert and more recent changes will take precedence.
- When the idle period passes, the component reaches out to third-party servers in the following manner:
- Initially, the malware will attempt to resolve a host within the avsvmcloud[.]com domain.
- The DNS response will return a CNAME record, this will be the hostname and, through the subsequent resolution, IP address of the Command and Control (C2) server.
- The malware will then utilise the HTTP protocol to receive commands from that server in true C2 fashion. The callback’s look and feel is engineered to masquerade itself as in internal SolarWinds protocol (Orion Improvement Program protocol), again in an effort to subvert analysis and hide itself as long as it can.
What can you do?
So now that we have the theory, the timeline and the facts out of the way, it should be obvious that this is a very tricky situation to protect against. So what can you do about this?
Let’s take the easy one off the table, you can’t prevent this situation from happening to you, anyone who is capitalising on this and contacting you right now with the latest and greatest solution that prevents this type of attack is trying to sell snake oil. It’s just that simple.
Account for this in your threat model
It is our belief that this threat vector is one that should, if not already present, be included in your threat models. We are firm believers that anything you add on your network basically increases your attack surface. We have previously written about a case like this in our blog.
When you acquire a new product or infrastructure, it is to cover a need you have and not because you have some money lying around! When you do introduce something new, make sure your threat model is modified accordingly, by revisiting it and including the new product you are acquiring.
As such, it all begins with modelling:
- Identify and list your critical functions - ideally, in a mature company this will already be in place.
- Include the new acquisition within the threat models - the vendor should be able to provide detailed information and documentation to support this exercise.
- Identify and list its interactions with the environment it is being incorporated into:
- What are its incoming interfaces ?
- What are the outbound or egress requirements?
- What are its communications with the outside world?
- The rest of the infrastructure ?
- The Internet?
- How finely can these requirements be tuned or limited?
- What accounts does it use ?
- Does it really need administrator level access on XYZ ?
- How about on ABC ?
- Really question hard on any required administrative level access required as in most cases it is simply a convenience and not a necessity ?
- Are these communication requirements constant? (could they be enabled on demand?)
- Can these communications be “inspected” by technology, through which anomaly detections can be defined?
It goes without saying that this is not an easy feat and it should not be applied across the board on all new acquisitions. Prioritisation is also something that is industry and company specific so one cannot be prescriptive from an outsider perspective. As a general rule of thumb, prioritise systems supporting critical functions and systems requiring administrator level access. Also if you have implemented data classification within your environment, systems within, or that interact with, highly classified (in your own context) or highly sensitive systems and information should also be considered a priority.
Plan for compromise
Create and implement a response plan in the event this acquisition is compromised. This should include actions for containment, eradication and recovery and should also cover any dependencies or communicating functions and systems. Seldom can an organisation simply “pull the plug” on infrastructure, and assuming the asset was incorporated to address a particular need, is it possible to address the risk while maintaining the necessary service level?
Once the threat modelling phase is complete, a tabletop exercise should be run with all involved parties and the threat model should be validated for completeness and accuracy. A high level intrusion scenario should be run against the threat model to identify blind spots, inaccuracies and misconceptions. This tabletop exercise should also include walking through the plans in place in case of a compromise.
Scenario based testing
Finally, put theory into practice and validate the threat model with an actual red team exercise designed to mimic, as closely as possible, a potential compromise of the new acquisition. This will again help in identifying blind spots, inaccuracies and misconceptions and validate the perceived versus the actual operating environment.
We talked previously about not being able to do anything to prevent this type of breach. What you can do, and what is entirely in your control, is the operating environment the new acquisition will be placed into. Care should be taken so the environment it is placed into is as hardened as possible, both at the network level with properly configured firewalls and at the system level with controls such as application whitelisting. Apply the concept of Least Privilege across the system, its dependencies and communications.
Having a hardened environment means having detective controls and a capable security operations team that will be in a position to identify a misbehaving system, either as a malfunction or, in the case at hand, as a result of a malicious actor. An example relevant to the specific breach is network communication to an external third party. The following (oversimplification for illustration purposes) could be the relevant firewall rules:
|Orion servers||updateurl.solarwinds.com||Allow||Yes (ideally)|
Having an equivalent ruleset, and a team to act on suspicious / anomalous events, would have identified the malicious C2 channel and acted upon it.
Responding goes hand in hand with detecting as one can’t exist without the other. There is no point detecting malicious activities if you are not responding to them and you can’t respond to a malicious activity you are not detecting. We touched upon this briefly in the “Plan for compromise” section above. You will need to ensure a specific plan is included in the overall Response plan and that it is being followed.
The threat actor’s perspective
At this point we would also like to take a look at the attacker’s perspective. In its SEC filing, SolarWinds indicated that the tainted updates have been applied to circa 18,000 installations. For an attacker this means 18,000 shells coming back to the command and control server.
This is no easy feat both from an infrastructure perspective but, and perhaps more importantly, from a campaign management perspective. It either implies manual inspection and categorisation or an automated way of identifying and categorising interesting / non-interesting targets. Both of these methods imply multiple infrastructures for Command and Control, having one where the more interesting ones will be directed and the operators will spend most of their time analysing, and multiple others where the less important ones will be passed on.
The latter implies that the attackers were casting a wide net but, in reality, they had a pretty good idea who to target and how they would be connecting back. At that stage of the callback channel establishment, primarily network based indicators are in place to make this judgement call, e.g. DNS servers performing resolution attempts, Internet egress points, etc. This implies a significant reconnaissance effort that preceded the SolarWinds Orion updates compromise. The latest findings in this case point towards the automated cherry-picking method but there is no concrete evidence to definitively prove it.
Looking back, we are reminded of the CCleaner breach in 2017. This was another supply chain attack that utilised the update mechanism of a popular piece of software used by millions of people around the world. This was a staged attack and part blind, part targeted. While the first stage of the malicious code would download and execute on all victims that updated their software, the second stage would only be downloaded and executed on victims that were part of large organisations, based on a filter in the malware’s Command-and-Control server. One can’t help but draw on the similarities of cherry-picking in both of these attacks.
Finally, quoting from the initial FireEye announcement “the attacker primarily sought information related to certain government customers”. The path in this case appears to be “SolarWinds” -> “FireEye” -> “Government agencies” (and these are the steps we know of at this point in time) which makes this the supply chain attack of all supply chain attacks!
In this rather long post, we tried to put the recent SolarWinds breach in context. We began by defining what a supply chain attack is and moved on to presenting facts about the case at hand. It should be rather obvious at this point that this is not an attack that is easily guarded against so we devoted a fair part of the post to presenting a “strategy” for defending and at the very least minimising the effects and detecting such an attack as early as possible. It should be mentioned that this strategy is geared towards more mature companies that already have a level of threat modelling and incident response plans within their security operations. Finally, we speculated on the threat actors’ perspective and their overall campaign management and objectives. One should keep in mind that this is still a developing situation with a lot of unknowns regarding the threat actors origins, drives and intentions.
Thanks a lot for staying with us!