BlockSec, a prominent blockchain security firm, has challenged recent optimistic claims about AI's ability to audit smart contracts, citing significant limitations in current benchmarking methods. The critique centers on EVMBench, a framework developed by researchers from OpenAI and Paradigm that evaluates AI performance in smart contract security audits.
Fundamental Gaps in AI Auditing Benchmarks
BlockSec co-founder Yajin Zhou argues that EVMBench's methodology doesn't accurately reflect real-world auditing complexity. The research team found that the benchmark's test cases are too simplified and fail to capture the nuanced analysis required in production security audits. According to Zhou, the framework tests AI models on isolated, well-defined vulnerabilities rather than the ambiguous, context-dependent issues auditors encounter in practice.
The firm's analysis reveals that current AI models struggle with several critical aspects of security work:
- Understanding complex business logic and economic incentives
- Identifying vulnerabilities that emerge from interactions between multiple contracts
- Recognizing context-dependent security issues that require domain expertise
- Adapting to novel attack vectors not present in training data
Zhou emphasized that "the real question is not 'Can AI replace humans?' but 'How should humans and AI work together?'" This perspective suggests a collaborative rather than replacement model for blockchain security professionals.
Implications for Security Professionals
For web3 security specialists and smart contract auditors, BlockSec's findings offer reassurance about career longevity while highlighting the need for AI literacy. Rather than fearing displacement, security professionals should focus on developing skills that complement AI tools—particularly expertise in novel protocol designs, economic security, and cross-chain vulnerabilities.
The debate underscores growing demand for professionals who can effectively integrate AI assistance into security workflows without over-relying on automated analysis. Companies hiring security talent should prioritize candidates who demonstrate both traditional auditing expertise and the ability to critically evaluate AI-generated findings.
As the industry matures, blockchain security roles will likely evolve to emphasize human judgment in complex scenarios while leveraging AI for routine vulnerability detection and code analysis.
