Solana stakeholders rushed to right the network Saturday night after what one insider called “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding block production to a halt.
Bots had swarmed the popular non-fungible token (NFT) minting tool known as Candy Machine earlier Saturday with an unprecedented tsunami of inbound traffic: four million transaction requests and 100 gigabits of data every second – a record for the network, one person at the Solana Foundation said.
For reasons not yet clear, this swarm pushed validators out of consensus. Block production became impossible and the network went dark at 4:32 p.m. ET. By 11 p.m. ET, validators (coordinating through Solana’s Discord channels and a Google doc created by one of the validators) restarted the cluster at slot 131973970.
Co-founder Anatoly Yakovenko, who said he was traveling during much of the fracas, credited the validator community for spearheading mainnet recovery. He'd caught flak on Twitter Saturday for allegedly being "MIA" during a network crisis.
Unlike last September’s 17-hour outage, Saturday’s hard fork restart did not resolve with new-and-improved code populating across the validators. It simply picked up where the network flopped seven hours prior.
In preparing for the restart validators mulled whether to implement code that would temporarily block Candy Machine transactions. Some debated in the Discord whether such a move constituted censorship. Regardless, it would only be effective if two-thirds of validators opted in. Few appeared to do so on Saturday night.
Other parts of the ecosystem quickly moved to bolster their defenses. At 11:36 p.m. ET, Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted it would soon deploy a 0.01 SOL “botting penalty” to help NFT projects stanch excessive traffic.
Solana ecosystem services like Phantom wallet and decentralized exchange Mango Markets struggled to stand up in the aftermath as RPC node providers plodded back online.
The outage contributed to a bloody, albeit brief, drawdown in SOL markets. Solana’s native token crashed to a 24-hour low of $83.13 about three hours into the outage before climbing back toward $89, according to CoinGecko.
Solana core developers have yet to diagnose what went wrong Saturday or how the apparent botting attack overcame existing safeguards to stymie consensus, a member of the foundation told CoinDesk.
The leader in news and information on cryptocurrency, digital assets and the future of money, CoinDesk is an award-winning media outlet that strives for the highest journalistic standards and abides by a strict set of editorial policies. In November 2023, CoinDesk was acquired by Bullish group, owner of Bullish, a regulated, institutional digital assets exchange. Bullish group is majority owned by Block.one; both groups have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary, and an editorial committee, chaired by a former editor-in-chief of The Wall Street Journal, is being formed to support journalistic integrity.