Solana Network Stumbles, On-Chain Trading Slows After ‘Forking’ Incident

Some validators are downgrading their software in an attempt to restore activity.

AccessTimeIconFeb 25, 2023 at 7:15 a.m. UTC
Updated Mar 2, 2023 at 12:52 p.m. UTC
10 Years of Decentralizing the Future
May 29-31, 2024 - Austin, TexasThe biggest and most established global hub for everything crypto, blockchain and Web3.Register Now

Infrastructure operators on the Solana network rushed to right the struggling blockchain early Saturday after a tech issue throttled users’ ability to trade crypto, transfer assets or do anything on-chain.

The blockchain started “forking” (creating conflicting versions of its transaction history) at around 12:53 a.m. ET, according to Solana’s Discord server. Shortly after that, validators’ RAM began increasing while the chain’s transaction throughput fell off a cliff.

Those tech troubles combined to effectively freeze nearly all on-chain activity on the Solana network. By 2 a.m., the network was processing about 93 transactions per second (TPS), well below the rate near 5000 TPS some 15 minutes prior, according to the data site Solana Explorer.

It was a crisis reminiscent of the chain’s rocky tech incidents in 2022, when a series of halts and slowdowns prompted reforms to how Solana manages inbound traffic. Saturday’s incident had no immediate culprit, though validator operators and network engineers suspected a bug in the new version of Solana code that had come online hours before.

Without a definite bug to squash, some validators began downgrading to the previous version in hopes of reviving Solana’s throughput, said the pseudonymous SolBlaze, who runs a liquid staking pool and is active in developer circles. Later, Solana employees began advocating for the downgrade, too.

Within hours a supermajority of validators had switched back to the old software in their attempt to restore Solana’s operations. But it did little to solve the still unknown problem that was weighing down performance. The effort then turned to a more drastic solution: restarting the chain to the point immediately prior to the forking.

“Coordinating a restart attempt means that the chain will be offline completely, which is always the last resort,” SolBlaze said.

UPDATE (Feb. 25, 2023 04:09 UTC): Added details.


Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

CoinDesk is an award-winning media outlet that covers the cryptocurrency industry. Its journalists abide by a strict set of editorial policies. In November 2023, CoinDesk was acquired by the Bullish group, owner of Bullish, a regulated, digital assets exchange. The Bullish group is majority-owned by Block.one; both companies have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary with an editorial committee to protect journalistic independence. CoinDesk employees, including journalists, may receive options in the Bullish group as part of their compensation.

Danny Nelson

Danny is CoinDesk's Managing Editor for Data & Tokens. He owns BTC, ETH and SOL.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.


Read more about