In this business case study, Lightening Electric, a power supplier in Northeast Ohio, learns that a threat actor has gained remote access to a programmable logic controller (PLC). Upon investigation, the security team finds additional security threats lurking in their network. Learn how Lightening Electric handled the incident, then use the discussion points to explore how your business may address access security.

It was 11:30 pm on a Friday night and Carlos Alvarez had just returned from watching a movie with his friends. He was thankful he wasn’t on call this weekend. On-call weekends were really hard on his sleep schedule, but as the lead network security engineer at Lightening Electric, he knew how important his job was. The facility supplied power to more than 2 million residential, business, and public sector customers, and those customers counted on Lightening to keep their lights on, their heat running, their appliances functional, and so much more.

Nonetheless, Carlos, who considered himself a typical security guy, couldn’t help checking in on things before he crawled into bed for, hopefully, some sound sleep. He knew the on-call admin would ring him if something was awry, so he logged on for a quick check, then off to sleep he’d go. But, as kismet would have it, as Carlos perused his logs, he saw some unusual activity-nothing glaringly obvious, but a few sensors and motors looked to be working overtime. The region wasn’t undergoing any major events, like the huge snowstorms common in this area of the country, which would precipitate extra energy generation, so Carlos decided-without too much thought-to look into it.

More than an hour later it was clear that someone with remote access had logged onto the system and was tinkering with levels-just a little bit, but enough to be suspiciously high as compared to baselines for a normal day. This is probably why the on-call admin hadn’t escalated it, he thought to himself. Fortunately, no customers had reported outages, so they weren’t dealing with an incident . Still, this warranted a closer look at whom the remote user was and why they were messing with sensors and motors. Maybe there was a legitimate reason.

After a bit of investigation, he found a connection to a remote maintenance worker’s account . The login was time stamped at 9:38 pm. Why was a maintenance worker logging in then? Carlos checked his logs and saw that this worker’s account had been dormant for several days. But all of a sudden, too late for normal working hours and without an outage or identified problem reported, this worker appeared to have logged on and accessed several areas of the OT net work.

There wasn’t a reason to panic, Carlos knew rationally, but something in his gut said otherwise.

Too anxious to go to bed, Carlos rang the on-call admin, who confirmed that something was not quite right but the signals weren’t enough to escalate on a Friday night . Together, they monitored sensor and motor data for a few more hours and saw that the activity was erratic, speeding up a little bit here, then suddenly slowing down too much there. By early morning, Carlos knew he had to call his boss, the chief information security officer.

The call to the CISO

“Dennis, hi, it’s Carlos. Sorry to wake you so early on a Saturday morning, but I think we have a problem.” “Hi, Carlos:• answered Dennis Matthews, Lightening Electric’s CISO. “It’s fine. What’s up?”

“I have some slightly suspicious activity on the network from a maintenance worker’s account, starting just after 9:30 pm last night. Is there any reason someone would be doing maintenance late on a weekend? Or was there an event we don’t know about?”

Still groggy from sleep, Dennis walked out of his bedroom and into his home offi ce. As he did, he asked Carlos to explain what he had seen. Dennis had been with the company 11 years and he’d never experienced maintenance work needing to be done at that time of night unless there was an outage, accident, or disaster.

However, power supply is a 24x7x365 business, Dennis knew, so he didn’t want to jump to any conclusions. He listened to Carlos carefully.

After a lengthy conversation and his own examination of the network data, Dennis instructed Carlos to gather a small team and start looking through logs for further evidence of an incident. Immediately after, he called Jean Haskell, Lightening’s head of identity and access management, a team with a dotted line to him and the CIO.

The call to the 1AM team

“Jean;• he started, “Sorry to bother you on a weekend but I need you to shut off all network access for one of our employees, Greg Hill . Find anything and everything he has access to and reset all of his passwords. This can’t wait.”

Oh, boy, thought Jean. It sounds simple, but in reality, Lightening Electric hadn’t gotten its 1AM program totally under control. Access governance was something they’d been working on for a few years but hadn’t implemented tools or processes to help identify users’ myriad and disparate access controls across tech

stacks . There were dozens of places she or someone on her team would have to look to find all access for Greg. She’d start with Active Directory, but she knew she would have to do a lot of manual searching to be certain that all access was disabled.

”I’ll get right on it;’ Jean said, knowing that whatever was going on was the result of systemic oversight on access controls, and the fix wouldn’t be easy or fast.

Jean instructed her team to start at the problem point: access to various components. To get to the motion and sensor controls, Greg would have had to access the distributed control system-or DCS for short-and he probably logged in with his field device. Before field devices were sent to maintenance employees, they were locked down pretty well. But they were also just wireless devices that were subject to all kinds of vulnerabilities . If Greg was communicating over an insecure wireless connection, for instance, it was possible someone had exploited the connection. There had also been some pushback from the field on multi-factor authentication so it had been rolled out spottily, meaning, Greg could have turned it off on his end.

Further, Lightening used the distributed network protocol 3 (DNP3) and MODBUS, which was worrisome. Neither MODBUS nor DNP3 support authentication or encryption, making communication from field devices to control system components vulnerable to spoofing and eavesdropping. In addition, when MODBUS is implemented in TCP/IP, which it was, attackers can spoof packets in the transport layer. And everyone dealing with ICS knows that the lack of broadcast suppression in MODBUS is a problem. Although, since Dennis didn’t mention anything about a denial of service, Jean didn’t think that was the vulnerability they were dealing with .

The IAM-security post assessment

It took Jean and her team the rest of the weekend and most of the week to dig into their toolkits and find every authorization for Greg. From email access to controls on the equipment, it was an arduous task. And still Jean was not 100% certain they’d found everything .

Nonetheless, they’d been able to reset Greg’s accounts and reinstate multi-factor authentication for the most sensitive, privileged account s. They were still waiting to speak with Greg, himself.

As it turned out, Greg was on vacation when the incident happened. Dennis had discovered this after many calls to business colleagues. He’d also spoken with the CIO and various architects and engineers about backup controls; they had to make sure that if an attacker managed to get into Lightening’s systems again, the attacker would not be able to use privileged access or escalate privileges on an insecure account to tamper with any OT components. Thankfully, the company had some network segmentation in place to reduce the risk of an attacker using a set of credentials to move laterally or hide in approved traffic and access everything from one user account. Still, valid credentials were valid credentials. And employees need access to systems and data to do their jobs.

Access security and ease of use were often at odds, Dennis knew.

On Friday morning following the incident, Dennis stood in front of the security and IT teams, including the identity team, to discuss what had happened and explore how to move forward with better access management. He was determined to move ahead with his identity and access management plans and had even garnered support from the CEO.

Fortunately, whomever illicitly accessed Lightening’s systems only affected minor damage. But that was enough to convince the CEO that changes were necessary. He’d read about Stuxnet and BlackEnergy and knew that critical infrastructure was a coveted target of cyber attack. While Lightening Electric wasn’t the biggest utility in the country or even the region, they had to take extra precautions on the cyber security front . Of course that meant looking at budgets and technology changes, but the CEO also knew he couldn’t delay any longer.

The CEO gave Dennis strict orders to keep costs down and reduce as many interruptions as possible, but he’d agreed to changes in their security approach.

The conversation about tools and requirements

“Thanks, everyone, for your hard work on this incident. We managed to avoid a disaster:• started Dennis, “but we might not be so lucky next time. And I also know that what you did was exception; you didn’t have all the tools in place that should be in place at a modern organization. We’re here today to talk about that and discuss what capabilities we need to procure and new processes to ensure they’re used correctly by our teams . This is a joint effort between security, IT, and IAM.”

He laid out the rough plan he’d presented to the CEO several times previously and recently after the intrusion.

Identity governance

“We have to start with identity governance:• Dennis said. ‘The very first thing we need is uniform visibility across our environments and technologies. We need support for Active Directory, all cloud environments, our OT environment, LDAP, our email client, any business applications in use-I have a list of about 100 right now, including our CRM tool, Box for collaboration, and more.”

“I couldn’t agree more, Dennis;• Jean chimed in and addressed the whole group; “We have a full list here of applications and entry points, then we have to layer endpoint management. This is a big job, so we need tools, technologies, and processes that allow us to do this work automatically instead of manually. I think our work this weekend was great on scoping the problem, but I am certain that once we choose a few identity-focused platforms, we will be surprised at the results.”

Dennis and Jean continued with the group, explaining the categories of controls they would like to see chosen and deployed.

not enough, they explained, just to have an inventory and visibility into identities. What was incredibly important, especially given their place in critical infrastructure, was to take identity governance and access management to the next level. That is, they said, elevating their capabilities to privileged access management, including endpoint privileged management, and 100% policy enforcement for MFA.

For the next hour or so they talked about the requirements for the tools they’d choose.

Access management requirements

“In addition to cross-environment, cross-application capabilities;• said one of the team members, ”I’d like to see comprehensive access security features. To me, that means a platform that rolls in least privilege capabilities, privilege escalation and delegation management, session management, password management, and secure remote access .”

“For the secure remote access part;’ added another team member, “endpoint privilege management is a must.” “What’s the difference?” asked another person.

“That’s a great question;• responded Jean. “Privileged access management within an on-prem, cloud, or IT environment has a different set of requirements than those that we’d use for the field or other remote workers. The remote access risks linked to these access points require centralized management and visibility-part of what we’re lacking now-and traceability to the source. In addition, securing the endpoints themselves, whether that’s field devices for our maintenance teams or laptops and tablets for our home-based workers, have to be able to defend against malware and ransomware, viruses, and other compromise tactics targeting individual devices.”

From the corner of the room, Peter, the help desk manager, spoke up, “Can I also request we look at a few tools that incorporate zero trust access principles? I’ve been reading a lot about this and it would save us a lot of time, effort, and unnecessary calls if we could replace the VPN and provide direct access to resources but make sure the access is appropriate. We get calls all the time about people wanting to access this or that, but they don’t have the right permissions. At times, the opposite is true; people who aren’t working here anymore haven’t been deprovisioned and those permissions haven’t been revoked. A platform that incorporates zero trust-based controls combined with single sign-on and MFA will reduce unnecessary calls to the help desk.”

“Great point;’ said Dennis . “centralizing and streamlining user access to corporate applications, not just for privileged users, but for our average users, too, brings us in line with Electric’s employee mission of providing easy, efficient access to the tools people use everyday.

“Plus, zero trust principles should be the default for all our systems . Especially because we’re in critical infrastructure, we have to consider every connection, every user potentially compromised. A tool that supports just-in-time access, least privilege access, adaptive identity, identity-based verification for every request supports digital transformation that the CEO keeps talking about .”

They talked for a while more about how remote access and cloud-based applications require a zero trust-based approach to privileged access management and endpoint privilege management and then turned to next steps in selecting tools.

Selecting the right tool for the job

“We’ve talked a bit about what we think we need;’ said Dennis, “but let’s recap a bit and keep a running list of what we want. Sound good?”

Everyone nodded and Jean broke out the trusty whiteboard.

“First, we need a PAM platform that can support all our environments: multi-cloud, on-prem, OT, and virtual. It should include comprehensive access security features such as secure remote access, and optimally incorporate a zero trust model. Other important functionality includes multi-factor authentication, session management, centralized password management, and privilege management.

“We also need endpoint privilege management to be able to find and deactivate accounts with excessive permissions, apply least privilege access across our user base, grant process- and application-level privileges based on context, and be able to centrally manage user-level privileges across environments and application. Additional endpoint controls for preventing ransomware are important, too.

“At a higher level, all of this needs to support business initiatives like cloud migration and digitalization of our OT network, remote work, mobility-in other words, all that digital transformation talk-and helps keep the company in compliance with privacy and security regulations and audit requirements, but delivers the highest level of security.

“Does this sound right to everyone?”

They were al in agreement. Now the task was, find the best tool or tools for their identity-based requirements.