Skip to main content
AmigoFish

CCT 378: Two Feeds, Full Backup, And The Generators Never Started (Domain 3)

From CISSP Cyber Training Podcast - CISSP Training Program · · 43 min

This plays from the publisher's own server. We do not host a copy.

Open at the source

Not rated yet

Description

Send us Fan Mail (https://www.buzzsprout.com/2167626/fan_mail/new)

CISSP Domain 3.9 — design site and facility security controls, built on a real data center power failure where every control in the building worked and the building still went dark.

Start free: https://www.cisspcybertraining.com/free-cissp-self-study (360 practice questions, a Domain 1 diagnostic, weekly questions by email, and the slides from this episode)

In October, a cloud provider's London data center lost its utility power. The batteries picked up the load exactly the way they were supposed to, ran for about five minutes, and then ran out. The generators should have taken over in that window. They never got the signal.

Nobody attacked anybody here. There are no bad guys in this one. No ransomware, no intrusion, nothing for incident response to chase. It was a cable fault outside the building and a logic error inside it.

HOW THE BUILDING WAS BUILT

Start with the design, because the design was good. Two completely independent utility feeds coming in from the power company — they call them A and B. Google's own words for that are a 2N utility power architecture.

Now listen to where that label stops. Behind both of those feeds sits one central bank of high-voltage generators, run by a dual-redundant controller, a PLC. Two of the thing coming in. One of the thing that was supposed to save them. And between the utility going away and the generators coming up, a battery system with about five minutes of runtime. That isn't a lot and it isn't supposed to be. It's a bridge.

So when somebody tells you a site is 2N, the only honest follow-up is: 2N at which layer?

Five days before any of this, they ran a quarterly load test on the generators. A real test, under load. And that test found a problem. So they did the right thing — they scheduled the repair. But to do the repair they had to physically open the breaker on feed B and lock it out, which they did properly: keyed interlock, locked out, tagged out.

And that is the moment the building stopped being 2N on the utility side. For several days — Google doesn't publish exactly how many — that site ran on one utility feed. Google's own words for it are a temporary single-feed, 1N, bridged maintenance posture.

Then the cable broke. An underground cable out on the utility network, outside the building entirely, developed a fault. And it was on feed A. The only feed that was live.

WHY THE GENERATORS NEVER STARTED

The PLC decides when to start the generators, and it decides it by sensing voltage. If the voltage goes away, start the generators. Simple enough.

But those voltage sensors were wired upstream of the feed breakers. On the line side. And feed B — the one that was open for repairs — was still energized upstream, all the way up to that open breaker. The power company was still feeding it. It just wasn't connected to the building anymore.

So the PLC looks at its sensor, sees voltage on feed B, concludes that a healthy utility source is available, and never issues a start command.

Google's own root cause is this: a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. It checked whether there was voltage. It never checked whether that voltage was actually connected to anything.

And here's the part worth a line in your notes. That PLC was dual-redundant. There were two of them. Both of them were wrong in exactly the same way, because they shared a configuration. Duplicating the box does not duplicate the judgment. That's common-mode failure, and it is the reason a redundancy count is not a resilience measurement.

Five minutes later the batteries were done and the data halls went dark.

TWO NUMBERS, KEPT APART

About ninety minutes with the data halls unpowered. Six hours and eight minutes before the services on top of them were whole again. Both figures are in Google's report and they measure different things.…

Back to CISSP Cyber Training Podcast - CISSP Training Program · Report a problem