Bug That Took Down 8% of Ethereum's Validators Sparks Worries About Even Bigger Outage

Ethereum just had one of those moments where the big story is what didn't go wrong.

A bug in Ethereum's Nethermind client software – used by validators of the blockchain to interact with the network – knocked out a chunk of the chain's key operators on Sunday.

It was a manageable incident, but the episode revived a long-simmering debate in the Ethereum ecosystem around the need for "client diversity." Some experts took the opportunity to point out how bad things could have been if another client software, Geth, the chain's most popular execution client, had gone out; the question is whether Ethereum could have kept going since Geth stands out as a possible single point of failure for the network.

Nethermind powers around 8% of the validators that operate Ethereum, and this weekend's bug was critical enough to pull those validators offline. Ethereum stayed up and running despite the issue, and Nethermind's developers released a patch fixing things within hours. The main consequence of the bug was that modest financial penalties fell onto some Nethermind-based validators, but the Nethermind incident followed a similar outage earlier in January that impacted Besu, the client software behind around 5% of Ethereum's validators.

The back-to-back outages have reignited a spirited discussion on X, the platform formerly known as Twitter, around Ethereum's persistent problem with client diversity. The idea is that the network becomes more resilient if it's not dependent on any single client software.

Around 85% of Ethereum's validators are currently powered by Geth, and the recent outages to smaller execution clients have renewed concerns that Geth's dominant market position could pose grave consequences if there were ever issues with its programming.

Geth, which stands for "Go Ethereum," is primarily developed and maintained by the Ethereum Foundation, the main nonprofit that supports Ethereum development. Geth hasn't been totally immune from bugs (no software is), but it has never suffered from a critical outage like the ones that hit Nethermind and Besu. If it did, the consequences would be far more serious for Ethereum.

Depending on the nature of the bug, a Geth glitch could halt the entire network, rendering it impossible for validators to add new blocks to the blockchain. Ethereum is also programmed to penalize validators that fall offline or break the network's rules, meaning thousands of Geth-based validators could be held financially liable in the event of a bug, and the penalties could get even bigger if the bug proves difficult to patch.

Notably, some of the leading services that stake Ethereum on behalf of users – effectively turning people into validators with less of a headache – rely on Geth to power their operations.

Cygaar, a crypto educator, noted in an X post that "Ethereum has terrible client diversity," adding that, "A critical issue in Geth can lead to potentially millions of ETH being destroyed from validators running Geth."

Cygaar cited data from the website execution-diversity.info noting that popular crypto exchanges like Coinbase, Binance and Kraken all rely on Geth to run their staking services. "Users who are staked in protocols that run Geth would lose their ETH" in the event of a critical issue," Cygaar wrote.

DCinvestor, a pseudonymous crypto investor with a large social media following, claimed in an X post that they were pulling their staked funds from Coinbase until the company switches its validator operations to a system that relies less on the Geth client. "[I] can't ignore the risks of what appears to be a single client staking setup (reliant on Geth) at this time," DCinvestor wrote, adding that "[I] could stand to lose a large percentage of my deposit" if things go wrong.

For Daniel Hwang, a validator expert who helms the Kintsugi Tech incubator, the attention on Ethereum's client diversity stems in part from the fact that the network is held to a higher standard than competing chains.

"Almost all other chains don't have the type of client diversity that Ethereum has," Hwang told CoinDesk in an interview. "Most are just running on one client."

"I think perhaps the bar is just held higher for Ethereum because it is the dominant smart contract chain," he said.

While Geth has a strong record of reliability, Hwang says many of Ethereum's validators just default to using it (rather than alternatives like Nethermind) out of laziness. In his experience, validators are "not doing their own research" into the strengths and weaknesses of competing client software.

The Ethereum Foundation urges validators to help improve client diversity, and Dankrad Feist, one of its researchers, was widely-cited this week for a 2022 article imploring validators not to use majority clients. Nethermind's development was also funded, in part, by a 2018 grant from the Ethereum Foundation.

Hwang likened Geth's dominance despite all this to an old business adage: "Nobody gets fired for buying IBM." In other words, if everyone else is using Geth, then it would be hard to blame an upstart validator for using it, too – even if things eventually go awry.

Counterintuitively, Hwang sees a silver lining in the recent Nethermind and Besu bugs.

"I don't want to say it's great that a client suffered a bug, but I think it's great if it begins to get people thinking about responsibilities," he said. "Validators should have been checking this s–t for themselves instead of just picking it off a supermarket shelf, and then raising their hands when things go wrong."