Blockchain Restores Services After Outage, Promises 'Post-Mortem' Review

Blockchain experienced a long outage this week, but launched a commendable press push to stem the damage.

AccessTimeIconMar 19, 2014 at 8:10 p.m. UTC
Updated Sep 11, 2021 at 10:33 a.m. UTC
10 Years of Decentralizing the Future
May 29-31, 2024 - Austin, TexasThe biggest and most established global event for everything crypto, blockchain and Web3.Register Now

More than a day after bitcoin wallet provider Blockchain.info first indicated it was experiencing a severe outage, its services have been fully restored, a company blog post has revealed.

Issues began with Blockchain noting that it had suspended SharedCoin transactions in what was initially suspected to be a short delay. However, that announcement, made on 17th March, proceeded a long period of downtime that extended into the late evening the next day.

Blockchain updated users that all Blockchain services were operational again as of 10pm PST, 18th March.

— Blockchain.info (@blockchain) March 19, 2014

The company had originally indicated that the problems stemmed from an 'unusual database issue', though the problem was later revealed to be the result of an error that affected several tables meant to track information contained in the bitcoin blockchain ledger.

Blockchain stated that though the problem was easy to identify, fixing the issue proved challenging due to the size of the affected data sets.

Explained Blockchain's security and distributed systems expert Andreas Antonopoulos:

"We decided to pursue multiple methods to try to fix the problem, including re-compiling a custom version of MySQL to get around the error, while simultaneously restoring databases from our backup systems onto new servers. Because of the large dataset, each step in testing, restoring or copying data can take a very long time, making the entire process slow."

Blockchain has promised a forthcoming 'post-mortem' report on the issue that will elaborate further on the facts within the next few days.

Continued issues

In his written comments, Antonopoulos expressed his thanks to the company's consumer and business users for their patience while it worked to correct the issue.

"We understand how frustrating it can be to be unable to access funds and we would like to take every precaution necessary to ensure this does not recur. Our team has worked non-stop to ensure services are returned to you in a timely manner."

At press time, however, Blockchain had issued another blog post detailing additional problems being experienced by some users, though they were unrelated to the larger outage.

Blockchain indicated that the company is currently working to address an issue being experienced by existing iPhone users, and that SharedCoin would continue to remain offline.

It reassured iPhone users that all funds are currently accounted for despite the glitch.

Blockchain's response

In contrast to measures taken by other high-profile companies during outages, the Blockchain team was unusually communicative about issues, releasing five blog posts between the initial message and the final declaration that services would be restored.

In between, the company stressed to users that it was dedicating its efforts to restoring services, and reiterated to users that no funds were affected by the error.

The blockchain team was active updating users on reddit - posting multiple, live edits to messages, and in responding to frustrated users on Twitter in what amounted to an agressive PR campaign to educate and assure users.

— Blockchain.info (@blockchain) March 19, 2014

Community reaction

Blockchain's response to the ordeal was greeted positively by the digital currency community, with many lauding the willingness of the team to remain open about problems and progress.

— CryptoBucket (@CryptoBucket) March 19, 2014

Mt. Gox was roundly criticized for its silence in the wake of massive theft and its eventual insolvency. Further, what statements it did issue did little to stem frustration and caused huge sell offs among investors.

Though Blockchain's efforts seem to have avoided such mistakes, some remained unsatisfied with the efforts.

— Cuke King (@cukeking) March 19, 2014

Next steps

Speaking to CoinDesk, CEO Nicolas Cary indicated that Blockchain would embark on a large-scale review of the service issues.

Cary indicated that the company would continue its policy of remaining open with users and any and all concerns.

"We've just started our post-mortem research and it will be a few days before we have comprehensive documentation of the incident. [...] We will be publishing the entire account on our blog so the whole Bitcoin community can learn from this."

Antonopoulos also expressed his frustration at the outage, vowing that more work would be done to restore user confidence in the coming days and weeks.

"We take pride in our work and care deeply about our customers, so this outage was humbling and painful for us too. We will do everything we can to regain your trust and make sure this doesn’t happen again."

Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

CoinDesk is an award-winning media outlet that covers the cryptocurrency industry. Its journalists abide by a strict set of editorial policies. In November 2023, CoinDesk was acquired by the Bullish group, owner of Bullish, a regulated, digital assets exchange. The Bullish group is majority-owned by Block.one; both companies have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary with an editorial committee to protect journalistic independence. CoinDesk employees, including journalists, may receive options in the Bullish group as part of their compensation.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.