Solana Goes Dark for 7 Hours as Bots Swarm ‘Candy Machine’ NFT Minting Tool

It wasn’t immediately clear how the bot traffic overcame network safeguards.

AccessTimeIconMay 1, 2022 at 5:29 a.m. UTC
Updated May 2, 2022 at 6:46 p.m. UTC

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

Solana stakeholders rushed to right the network Saturday night after what one insider called “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding block production to a halt.

Bots had swarmed the popular non-fungible token (NFT) minting tool known as Candy Machine earlier Saturday with an unprecedented tsunami of inbound traffic: four million transaction requests and 100 gigabits of data every second – a record for the network, one person at the Solana Foundation said.

For reasons not yet clear, this swarm pushed validators out of consensus. Block production became impossible and the network went dark at 4:32 p.m. ET. By 11 p.m. ET, validators (coordinating through Solana’s Discord channels and a Google doc created by one of the validators) restarted the cluster at slot 131973970.

Co-founder Anatoly Yakovenko, who said he was traveling during much of the fracas, credited the validator community for spearheading mainnet recovery. He'd caught flak on Twitter Saturday for allegedly being "MIA" during a network crisis.

Unlike last September’s 17-hour outage, Saturday’s hard fork restart did not resolve with new-and-improved code populating across the validators. It simply picked up where the network flopped seven hours prior.

In preparing for the restart validators mulled whether to implement code that would temporarily block Candy Machine transactions. Some debated in the Discord whether such a move constituted censorship. Regardless, it would only be effective if two-thirds of validators opted in. Few appeared to do so on Saturday night.

Other parts of the ecosystem quickly moved to bolster their defenses. At 11:36 p.m. ET, Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted it would soon deploy a 0.01 SOL “botting penalty” to help NFT projects stanch excessive traffic.

Solana ecosystem services like Phantom wallet and decentralized exchange Mango Markets struggled to stand up in the aftermath as RPC node providers plodded back online.

The outage contributed to a bloody, albeit brief, drawdown in SOL markets. Solana’s native token crashed to a 24-hour low of $83.13 about three hours into the outage before climbing back toward $89, according to CoinGecko.

Solana core developers have yet to diagnose what went wrong Saturday or how the apparent botting attack overcame existing safeguards to stymie consensus, a member of the foundation told CoinDesk.

DISCLOSURE

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

The leader in news and information on cryptocurrency, digital assets and the future of money, CoinDesk is a media outlet that strives for the highest journalistic standards and abides by a strict set of editorial policies. CoinDesk is an independent operating subsidiary of Digital Currency Group, which invests in cryptocurrencies and blockchain startups. As part of their compensation, certain CoinDesk employees, including editorial employees, may receive exposure to DCG equity in the form of stock appreciation rights, which vest over a multi-year period. CoinDesk journalists are not allowed to purchase stock outright in DCG.

CoinDesk - Unknown

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

CoinDesk - Unknown

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

Trending

1
CoinDesk - Unknown
First Mover Asia: Chip Maker Nvidia Isn’t an Ether Proxy, Bitcoin Holds Near $21K

Crypto mining aided Nvidia's bottom line, but it hasn't been a major cause of the stock's steep decline. Ether falls but hovers around $1,200.

CoinDesk - Unknown
2
CoinDesk - Unknown
CoinFLEX to Launch a $47M Recovery Token to Solve Withdrawal Issues

CoinFLEX said last week it was halting withdrawals amid market declines and counterparty uncertainty.

CoinDesk - Unknown
3
CoinDesk - Unknown
Solana’s Macalinao Brothers Double Down on Crypto Venture Fund

Crypto VCs told CoinDesk that building projects and investing in them is a tricky mix.

CoinDesk - Unknown
4
CoinDesk - Unknown
Ms. Crypto Goes to Washington

The Lummis-Gillibrand bill represents a mainstreaming of crypto, even if it's unlikely to pass.

CoinDesk - Unknown