Chainalysis Testimony Raises the Question: Do We Know How Well Any Such Software Works?

Chainalysis' head of investigations doesn't seem to have a great understanding of the accuracy of her company's flagship software. She’s not alone.

AccessTimeIconJul 24, 2023 at 10:19 p.m. UTC
Updated Aug 31, 2023 at 3:42 p.m. UTC
AccessTimeIconJul 24, 2023 at 10:19 p.m. UTCUpdated Aug 31, 2023 at 3:42 p.m. UTC
AccessTimeIconJul 24, 2023 at 10:19 p.m. UTCUpdated Aug 31, 2023 at 3:42 p.m. UTC

Editor-in-chief’s note: An earlier form of this pseudonymously written opinion piece was retracted due to its containing a perceived personal attack on a Chainalysis employee. We did this because it violated our policy, which is based on the principle that people have the right to know the identity of their critics, of not publishing personalized attacks under pseudonymous bylines. It was not our intent to besmirch the reputation of the writer, who has for some time used the same pseudonym and built a reputation around it. To resolve this, in collaboration with the writer, we are republishing a further edited version of the piece, removing language that unreasonably targeted the individual in question and making clear that its sole intent is to question the accuracy of blockchain demystification software, and not attack any particular individual.

Elizabeth Bisbee, head of investigations at Chainalysis Government Solutions, testified she was “unaware” of scientific evidence for the accuracy of Chainalysis’ Reactor software on June 23, 2023, an unreleased transcript of the hearing shared with CoinDesk shows.

The fact that Chainalysis’ blockchain demystification tools have become so widespread is a serious threat to the crypto ecosystem. Although industry insiders have raged against Chainalysis since it was founded, often accusing it of violating people’s financial privacy, there may be a better argument to make against the company and analysis firms like it: it’s within the realm of possibility that these “probabilistic” machines don’t work as well as advertised. This could lead to unjustified account restrictions when blockchain surveillance tools are used for compliance, and in the worst case land unsuspecting individuals on the radar of law enforcement agencies without probable cause.

That’s precisely the argument that renowned lawyer Tor Ekeland is making in his latest defense of an accused early bitcoin adopter, and why he was quizzing a Chainalysis executive on the stand.

Bisbee was testifying in a case between the U.S. government and Roman Sterlingov, the alleged creator of the once popular Bitcoin Fog cryptocurrency mixer used to anonymize bitcoin transactions. Chainalysis’ Reactor software was used to track cryptocurrency payments in Sterlingov’s criminal investigation, and is now being challenged by Sterlingov’s defense.

Sterlingov is represented by Ekeland, who has made a career out of defending hackers and technology providers. Ekeland said Chainalysis’ Reactor is “a black box algorithm” that “relies on junk science.”

In a hearing aimed to establish the admissibility of expert testimony, Bisbee was pressed for details on the accuracy of the Reactor software Chainalysis sells to governments for law enforcement purposes, including what evidence the company has that suggests it works.

Bisbee said she was unable to provide the court with statistical error rates for Chainalysis’ Reactor software. She said she unaware of any scientific peer-reviewed papers or “anything published anywhere” attesting to the accuracy of Chainalysis Reactor.

Instead, Chainalysis reportedly judges its software’s accuracy using customer feedback, she said.

None of this is to cast aspersions on Bisbee's personal understanding of the software. For example, Bisbee’s statements are in line with a blog post published by Chainalysis competitor Coinbase, which describes blockchain analytics as “more of an art than a science.” Coinbase offers blockchain analytics services to law enforcement via its Tracer software.

A statement issued by Chainalysis on July 18th confirms that Bisbee is not only unaware of margins of error rates for Chainalysis’ Reactor software, but that Chainalysis has failed to gather and record false positive and negative rates for its software overall.

Unfortunately for Chainalysis, we still live in a democracy in which criminal convictions prerequisite the existence of scientific evidence.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.


Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

The leader in news and information on cryptocurrency, digital assets and the future of money, CoinDesk is an award-winning media outlet that strives for the highest journalistic standards and abides by a strict set of editorial policies. In November 2023, CoinDesk was acquired by Bullish group, owner of Bullish, a regulated, institutional digital assets exchange. Bullish group is majority owned by Block.one; both groups have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary, and an editorial committee, chaired by a former editor-in-chief of The Wall Street Journal, is being formed to support journalistic integrity.

L0la L33tz

L0la L33tz does not actually exist. You can reach her at the end of surveillance capitalism, PO Box 77, C72B 398B 7C04 8F04.