Solana Goes Dark for 7 Hours as Bots Swarm ‘Candy Machine’ NFT Minting Tool

It wasn’t immediately clear how the bot traffic overcame network safeguards.

AccessTimeIconMay 1, 2022 at 5:29 a.m. UTC
Updated May 11, 2023 at 6:37 p.m. UTC
10 Years of Decentralizing the Future
May 29-31, 2024 - Austin, TexasThe biggest and most established global hub for everything crypto, blockchain and Web3.Register Now

Solana stakeholders rushed to right the network Saturday night after what one insider called “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding block production to a halt.

Bots had swarmed the popular non-fungible token (NFT) minting tool known as Candy Machine earlier Saturday with an unprecedented tsunami of inbound traffic: four million transaction requests and 100 gigabits of data every second – a record for the network, one person at the Solana Foundation said.

For reasons not yet clear, this swarm pushed validators out of consensus. Block production became impossible and the network went dark at 4:32 p.m. ET. By 11 p.m. ET, validators (coordinating through Solana’s Discord channels and a Google doc created by one of the validators) restarted the cluster at slot 131973970.

Co-founder Anatoly Yakovenko, who said he was traveling during much of the fracas, credited the validator community for spearheading mainnet recovery. He'd caught flak on Twitter Saturday for allegedly being "MIA" during a network crisis.

Unlike last September’s 17-hour outage, Saturday’s hard fork restart did not resolve with new-and-improved code populating across the validators. It simply picked up where the network flopped seven hours prior.

In preparing for the restart validators mulled whether to implement code that would temporarily block Candy Machine transactions. Some debated in the Discord whether such a move constituted censorship. Regardless, it would only be effective if two-thirds of validators opted in. Few appeared to do so on Saturday night.

Other parts of the ecosystem quickly moved to bolster their defenses. At 11:36 p.m. ET, Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted it would soon deploy a 0.01 SOL “botting penalty” to help NFT projects stanch excessive traffic.

Solana ecosystem services like Phantom wallet and decentralized exchange Mango Markets struggled to stand up in the aftermath as RPC node providers plodded back online.

The outage contributed to a bloody, albeit brief, drawdown in SOL markets. Solana’s native token crashed to a 24-hour low of $83.13 about three hours into the outage before climbing back toward $89, according to CoinGecko.

Solana core developers have yet to diagnose what went wrong Saturday or how the apparent botting attack overcame existing safeguards to stymie consensus, a member of the foundation told CoinDesk.

Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

CoinDesk is an award-winning media outlet that covers the cryptocurrency industry. Its journalists abide by a strict set of editorial policies. In November 2023, CoinDesk was acquired by the Bullish group, owner of Bullish, a regulated, digital assets exchange. The Bullish group is majority-owned by Block.one; both companies have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary with an editorial committee to protect journalistic independence. CoinDesk employees, including journalists, may receive options in the Bullish group as part of their compensation.

Danny Nelson

Danny is CoinDesk's Managing Editor for Data & Tokens. He owns BTC, ETH and SOL.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.