Solana Goes Dark for 7 Hours as Bots Swarm ‘Candy Machine’ NFT Minting Tool

It wasn’t immediately clear how the bot traffic overcame network safeguards.

May 1, 2022 at 5:29 a.m. UTC
Updated May 2, 2022 at 6:46 p.m. UTC

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

Solana stakeholders rushed to right the network Saturday night after what one insider called “insane amount of data” flooded the proof-of-stake chain, knocking validators out of consensus and grinding block production to a halt.

Bots had swarmed the popular non-fungible token (NFT) minting tool known as Candy Machine earlier Saturday with an unprecedented tsunami of inbound traffic: four million transaction requests and 100 gigabits of data every second – a record for the network, one person at the Solana Foundation said.

For reasons not yet clear, this swarm pushed validators out of consensus. Block production became impossible and the network went dark at 4:32 p.m. ET. By 11 p.m. ET, validators (coordinating through Solana’s Discord channels and a Google doc created by one of the validators) restarted the cluster at slot 131973970.

Co-founder Anatoly Yakovenko, who said he was traveling during much of the fracas, credited the validator community for spearheading mainnet recovery. He'd caught flak on Twitter Saturday for allegedly being "MIA" during a network crisis.

Unlike last September’s 17-hour outage, Saturday’s hard fork restart did not resolve with new-and-improved code populating across the validators. It simply picked up where the network flopped seven hours prior.

In preparing for the restart validators mulled whether to implement code that would temporarily block Candy Machine transactions. Some debated in the Discord whether such a move constituted censorship. Regardless, it would only be effective if two-thirds of validators opted in. Few appeared to do so on Saturday night.

Other parts of the ecosystem quickly moved to bolster their defenses. At 11:36 p.m. ET, Metaplex, the key steward of Solana NFT infrastructure and one closely intertwined with Candy Machine, tweeted it would soon deploy a 0.01 SOL “botting penalty” to help NFT projects stanch excessive traffic.

Solana ecosystem services like Phantom wallet and decentralized exchange Mango Markets struggled to stand up in the aftermath as RPC node providers plodded back online.

The outage contributed to a bloody, albeit brief, drawdown in SOL markets. Solana’s native token crashed to a 24-hour low of $83.13 about three hours into the outage before climbing back toward $89, according to CoinGecko.

Solana core developers have yet to diagnose what went wrong Saturday or how the apparent botting attack overcame existing safeguards to stymie consensus, a member of the foundation told CoinDesk.

The Festival for the Decentralized World
Thursday - Sunday, June 9-12, 2022
Austin, Texas
Save a Seat Now

DISCLOSURE

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

The leader in news and information on cryptocurrency, digital assets and the future of money, CoinDesk is a media outlet that strives for the highest journalistic standards and abides by a strict set of editorial policies. CoinDesk is an independent operating subsidiary of Digital Currency Group, which invests in cryptocurrencies and blockchain startups. As part of their compensation, certain CoinDesk employees, including editorial employees, may receive exposure to DCG equity in the form of stock appreciation rights, which vest over a multi-year period. CoinDesk journalists are not allowed to purchase stock outright in DCG.

CoinDesk - Unknown

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

CoinDesk - Unknown

Danny is CoinDesk's deputy business editor. He owns BTC, ETH and SOL.

Trending

1
CoinDesk - Unknown
How the US Can Establish Itself as a Crypto Leader

Regulators have an opportunity to map out thoughtful, strategic policy on stablecoins and beyond.

Regulators have an opportunity to map out thoughtful, strategic policy on stablecoins and beyond.

CoinDesk - Unknown
2
CoinDesk - Unknown
No, the UK Is Not Going to Make USDC and USDT Legal Tender

For “legalize” read “regulate.”

For “legalize” read “regulate.”

CoinDesk - Unknown
3
CoinDesk - Unknown
Thoughts From Davos

The crypto industry showed up in force at the World Economic Forum’s annual meeting.

The crypto industry showed up in force at the World Economic Forum’s annual meeting.

CoinDesk - Unknown
4
CoinDesk - Unknown
Bitso, primer unicornio cripto de América Latina, despide a 80 empleados

El exchange, que tenía más de 700 empleados antes de los recortes, cuenta con cuatro millones de usuarios en la región.

El exchange, que tenía más de 700 empleados antes de los recortes, cuenta con cuatro millones de usuarios en la región.

CoinDesk - Unknown