Crypto White Hat Platform Immunefi Banned 15 ChatGPT-Generated Bug Reports. Here's Why

Bug reports are “really an art,” an Immunefi developer told CoinDesk.

AccessTimeIconJan 17, 2023 at 8:32 a.m. UTC
Updated Jan 17, 2023 at 4:35 p.m. UTC
10 Years of Decentralizing the Future
May 29-31, 2024 - Austin, TexasThe biggest and most established global hub for everything crypto, blockchain and Web3.Register Now

OpenAI’s ChatGPT has stirred the hornet’s nest since its launch, with the Twitterverse readily commenting on how the software could eventually create code, write stories, steal jobs and even fight wildfires.

But the crypto platform Immunefi is having none of that. The bounty-paying white hat – a term for individuals who hack products with the goal of patching security instead of theft – banned 15 users last week who submitted ChatGPT-regenerated reports on the platform, albeit to mixed sentiment.

Immunefi developers told CoinDesk that the platform’s reasoning on the subject is crystal clear for now.

“There’s a difference between something like GitHub Copilot and ChatGPT. With the former, you are driving the process and the role of Copilot is offering useful suggestions in context, which you accept or reject as you write your program,” Immunefi told CoinDesk in a Twitter message. “With the latter, you are using a single prompt to generate something that looks like a well-written bug report, but is nonsense when analyzed further.”

“This wastes the white hat’s time, Immunefi’s time and the projects’ time. That’s why we ban ChatGPT reports,” they added.

“It’s really an art,” developers further said, referring to the act of carefully writing a well-crafted bug report that clearly lays out issues and solutions for any exploit or bug.

Immunefi went a step ahead and asked ChatGPT itself why the software shouldn’t be used for generating bug reports, to a “satisfying response.”

The bug bounty platform has over $135 million in rewards available for white hats who find out vulnerabilities on decentralized finance (DeFi) platforms. It claims to have paid out more than $60 million in bounties and says the service has saved an estimated $25 billion in user funds.

As such, Immunefi said it would continue to monitor ChatGPT-generated reports even as they eventually get more sophisticated. “There are a lot of very obvious tells. If you play around with ChatGPT, you’ll start to see a lot of patterns in the output,” developers said.

“For example, no regular white hat would go out of their way to waste their time submitting a long, nicely written and well-structured bug report that has no relation to a project’s smart contracts at all,” they added.

But that’s not to say they would ever not consider unbanning the use of AI-generated bug reports in the future.

“We will keep an eye out for the development of AI tools, and we will consider unbanning their use if they are ever capable of generating real bug reports.” For now, however, it’s ban time.

Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

CoinDesk is an award-winning media outlet that covers the cryptocurrency industry. Its journalists abide by a strict set of editorial policies. In November 2023, CoinDesk was acquired by the Bullish group, owner of Bullish, a regulated, digital assets exchange. The Bullish group is majority-owned by Block.one; both companies have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary with an editorial committee to protect journalistic independence. CoinDesk offers all employees above a certain salary threshold, including journalists, stock options in the Bullish group as part of their compensation.

Shaurya Malwa

Shaurya is the Deputy Managing Editor for the Data & Tokens team, focusing on decentralized finance, markets, on-chain data, and governance across all major and minor blockchains.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.