CASE: One Wrong Click Can Create a 92-day Recovery
Imagine walking into work on a warm summer day, checking your inbox and seeing an email from your colleague about the great weekend he had, asking you to check out the pictures. Would you click on it? One employee did that using the building’s OT application host, which controls the building control system (BCS) and HVAC. That action set a cascading series of events that took the building’s HVAC system offline, leading to a recovery process that took 92 days to bring the HVAC system back online and operating within specifications.
The following case is based on true events of how the actions over two days led to a 92-day recovery.
Thank you to Fred Gordy, Director, OT Risk Assessments, Michael Baker International, for his insights, inspiration, and support in creating this case.
Day 1: 7:00 a.m.
It is Monday morning in July at a large office building in downtown Washington, DC, USA office building. The sun is shining, and the building is just starting to come to life. Workers are arriving, and the facility management (FM) team is unlocking the doors.
An FM worker opens the building’s OT application host that controls the building control systems and HVAC. While on the host, he opens a browser and decides to check his Gmail. He sees and opens a message from his colleague about a fantastic party he attended this past weekend. He clicks on a link to see pictures from the party, but the link doesn't work.
7:30 a.m.
9:00 a.m.
The ransomware has encrypted the system's front end, and the team has been locked out of the host, the BCS and the HVAC. Hackers are demanding Bitcoin payment in return for access. The team decides not to pay and contacts their HVAC vendor. The HVAC system is operating in the last command status.
12:00 p.m.
5:00 p.m.
After the FM installed their backup on a new machine brought by the HVAC vendor, they brought the system back online. At 5:00 PM, the FM team leaves for the day, thinking they have avoided catastrophe.
Day 2: 6:00 a.m.
6:30 a.m.
8:00 a.m.
customer complaints keep flooding in.
“We need to activate our business continuity plan and send workers home for the day until we can get bring our HVAC system back online and find alternative solutions.”
4:00 p.m.
the fm team has been working tirelessly all day, finally reaching the chiller tear-down. The FM team starts to understand the full extent of the damage caused by the ransomware attack. It will take the team 92 days to bring the HVAC and the building back into operating spec.