Blockchain Restores Services After Outage, Promises 'Post-Mortem' Review

More than a day after bitcoin wallet provider Blockchain.info first indicated it was experiencing a severe outage, its services have been fully restored, a company blog post has revealed.

Issues began with Blockchain noting that it had suspended SharedCoin transactions in what was initially suspected to be a short delay. However, that announcement, made on 17th March, proceeded a long period of downtime that extended into the late evening the next day.

Blockchain updated users that all Blockchain services were operational again as of 10pm PST, 18th March.

As of 10pm PT, 5am UTC Blockchain services are RESTORED http://t.co/IssrKmpKRI Thank you for your patience and support

— Blockchain.info (@blockchain) March 19, 2014

The company had originally indicated that the problems stemmed from an 'unusual database issue', though the problem was later revealed to be the result of an error that affected several tables meant to track information contained in the bitcoin blockchain ledger.

Blockchain stated that though the problem was easy to identify, fixing the issue proved challenging due to the size of the affected data sets.

Explained Blockchain's security and distributed systems expert Andreas Antonopoulos:

Blockchain has promised a forthcoming 'post-mortem' report on the issue that will elaborate further on the facts within the next few days.

In his written comments, Antonopoulos expressed his thanks to the company's consumer and business users for their patience while it worked to correct the issue.

At press time, however, Blockchain had issued another blog post detailing additional problems being experienced by some users, though they were unrelated to the larger outage.

Blockchain indicated that the company is currently working to address an issue being experienced by existing iPhone users, and that SharedCoin would continue to remain offline.

It reassured iPhone users that all funds are currently accounted for despite the glitch.

In contrast to measures taken by other high-profile companies during outages, the Blockchain team was unusually communicative about issues, releasing five blog posts between the initial message and the final declaration that services would be restored.

In between, the company stressed to users that it was dedicating its efforts to restoring services, and reiterated to users that no funds were affected by the error.

The blockchain team was active updating users on reddit - posting multiple, live edits to messages, and in responding to frustrated users on Twitter in what amounted to an agressive PR campaign to educate and assure users.

@nawazkiani That is categorically false. No funds or accounts have been compromised. We don’t have access to your keys.

— Blockchain.info (@blockchain) March 19, 2014

Blockchain's response to the ordeal was greeted positively by the digital currency community, with many lauding the willingness of the team to remain open about problems and progress.

@aantonop@blockchain Keep up the great work! A+ for giving users a way to access their funds while going through the outage!

— CryptoBucket (@CryptoBucket) March 19, 2014

Mt. Gox was roundly criticized for its silence in the wake of massive theft and its eventual insolvency. Further, what statements it did issue did little to stem frustration and caused huge sell offs among investors.

Though Blockchain's efforts seem to have avoided such mistakes, some remained unsatisfied with the efforts.

@aantonop @blockchain 72 hours of downtime should result in a policy review and someone fired instead of tweets honestly. But, bitcoin.

— Cuke King (@cukeking) March 19, 2014

Speaking to CoinDesk, CEO Nicolas Cary indicated that Blockchain would embark on a large-scale review of the service issues.

Cary indicated that the company would continue its policy of remaining open with users and any and all concerns.

"We've just started our post-mortem research and it will be a few days before we have comprehensive documentation of the incident. [...] We will be publishing the entire account on our blog so the whole Bitcoin community can learn from this."

Antonopoulos also expressed his frustration at the outage, vowing that more work would be done to restore user confidence in the coming days and weeks.

"We take pride in our work and care deeply about our customers, so this outage was humbling and painful for us too. We will do everything we can to regain your trust and make sure this doesn’t happen again."