Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
Published in ICSE, 2025
Recommended citation: Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, and Wei Le. 2025. Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE. In 2025 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’25), April 27–May 3, 2025, Ottawa, Canada. https://arxiv.org/abs/2412.14306
We present DeepVulGuard, an AI-powered vulnerability detection & repair tool built into the VSCode IDE. We ran a user study of our tool with 17 professional software developers where study participants scanned a total of 24 projects, 6.9k files, and over 1.7 million lines of source code, and generated 170 alerts and 50 fix suggestions.
- We explored how AI features, including confidence scores, explanations, and chat interaction, can be useful for vulnerability detection and fixing.
- We offer practical recommendations for evaluating and deploying AI detection and fix models.
- We analyzed user feedback to reveal several actionable pain points, ranging from incomplete context to lack of customization for the user’s codebase.
- Although state-of-the-art AI-powered detection and fix tools show promise, they are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes.