Blockchain Restores Services After Outage, Promises ‘Post-Mortem’ Review

Pete Rizzo
Mar 19, 2014 at 20:10 UTC
Updated Jul 16, 2014 at 12:09 UTC

More than a day after bitcoin wallet provider Blockchain.info first indicated it was experiencing a severe outage, its services have been fully restored, a company blog post has revealed.

Issues began with Blockchain noting that it had suspended SharedCoin transactions in what was initially suspected to be a short delay. However, that announcement, made on 17th March, proceeded a long period of downtime that extended into the late evening the next day.

Blockchain updated users that all Blockchain services were operational again as of 10pm PST, 18th March.

 

The company had originally indicated that the problems stemmed from an ‘unusual database issue’, though the problem was later revealed to be the result of an error that affected several tables meant to track information contained in the bitcoin blockchain ledger.

Blockchain stated that though the problem was easy to identify, fixing the issue proved challenging due to the size of the affected data sets.

Explained Blockchain’s security and distributed systems expert Andreas Antonopoulos:

“We decided to pursue multiple methods to try to fix the problem, including re-compiling a custom version of MySQL to get around the error, while simultaneously restoring databases from our backup systems onto new servers. Because of the large dataset, each step in testing, restoring or copying data can take a very long time, making the entire process slow.”

Blockchain has promised a forthcoming ‘post-mortem’ report on the issue that will elaborate further on the facts within the next few days.

Continued issues

In his written comments, Antonopoulos expressed his thanks to the company’s consumer and business users for their patience while it worked to correct the issue.

“We understand how frustrating it can be to be unable to access funds and we would like to take every precaution necessary to ensure this does not recur. Our team has worked non-stop to ensure services are returned to you in a timely manner.”

At press time, however, Blockchain had issued another blog post detailing additional problems being experienced by some users, though they were unrelated to the larger outage.

Blockchain indicated that the company is currently working to address an issue being experienced by existing iPhone users, and that SharedCoin would continue to remain offline.

It reassured iPhone users that all funds are currently accounted for despite the glitch.

Blockchain’s response

In contrast to measures taken by other high-profile companies during outages, the Blockchain team was unusually communicative about issues, releasing five blog posts between the initial message and the final declaration that services would be restored.

In between, the company stressed to users that it was dedicating its efforts to restoring services, and reiterated to users that no funds were affected by the error.

The blockchain team was active updating users on reddit – posting multiple, live edits to messages, and in responding to frustrated users on Twitter in what amounted to an agressive PR campaign to educate and assure users.

Community reaction

Blockchain’s response to the ordeal was greeted positively by the digital currency community, with many lauding the willingness of the team to remain open about problems and progress.

 

Mt. Gox was roundly criticized for its silence in the wake of massive theft and its eventual insolvency. Further, what statements it did issue did little to stem frustration and caused huge sell offs among investors.

Though Blockchain’s efforts seem to have avoided such mistakes, some remained unsatisfied with the efforts.

 

Next steps

Speaking to CoinDesk, CEO Nicolas Cary indicated that Blockchain would embark on a large-scale review of the service issues.

Cary indicated that the company would continue its policy of remaining open with users and any and all concerns.

“We’ve just started our post-mortem research and it will be a few days before we have comprehensive documentation of the incident. […] We will be publishing the entire account on our blog so the whole Bitcoin community can learn from this.”

Antonopoulos also expressed his frustration at the outage, vowing that more work would be done to restore user confidence in the coming days and weeks.

“We take pride in our work and care deeply about our customers, so this outage was humbling and painful for us too. We will do everything we can to regain your trust and make sure this doesn’t happen again.”