Infrastructure operators on the Solana network rushed to right the struggling blockchain early Saturday after a tech issue throttled users’ ability to trade crypto, transfer assets or do anything on-chain.
The blockchain started “forking” (creating conflicting versions of its transaction history) at around 12:53 a.m. ET, according to Solana’s Discord server. Shortly after that, validators’ RAM began increasing while the chain’s transaction throughput fell off a cliff.
Those tech troubles combined to effectively freeze nearly all on-chain activity on the Solana network. By 2 a.m., the network was processing about 93 transactions per second (TPS), well below the rate near 5000 TPS some 15 minutes prior, according to the data site Solana Explorer.
It was a crisis reminiscent of the chain’s rocky tech incidents in 2022, when a series of halts and slowdowns prompted reforms to how Solana manages inbound traffic. Saturday’s incident had no immediate culprit, though validator operators and network engineers suspected a bug in the new version of Solana code that had come online hours before.
Without a definite bug to squash, some validators began downgrading to the previous version in hopes of reviving Solana’s throughput, said the pseudonymous SolBlaze, who runs a liquid staking pool and is active in developer circles. Later, Solana employees began advocating for the downgrade, too.
Within hours a supermajority of validators had switched back to the old software in their attempt to restore Solana’s operations. But it did little to solve the still unknown problem that was weighing down performance. The effort then turned to a more drastic solution: restarting the chain to the point immediately prior to the forking.
“Coordinating a restart attempt means that the chain will be offline completely, which is always the last resort,” SolBlaze said.
UPDATE (Feb. 25, 2023 04:09 UTC): Added details.
The leader in news and information on cryptocurrency, digital assets and the future of money, CoinDesk is a media outlet that strives for the highest journalistic standards and abides by a strict set of editorial policies. CoinDesk is an independent operating subsidiary of Digital Currency Group, which invests in cryptocurrencies and blockchain startups. As part of their compensation, certain CoinDesk employees, including editorial employees, may receive exposure to DCG equity in the form of stock appreciation rights, which vest over a multi-year period. CoinDesk journalists are not allowed to purchase stock outright in DCG.