Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE

We present DeepVulGuard, an AI-powered vulnerability detection & repair tool built into the VSCode IDE. We ran a user study of our tool with 17 professional software developers where study participants scanned a total of 24 projects, 6.9k files, and over 1.7 million lines of source code, and generated 170 alerts and 50 fix suggestions.

• We explored how AI features, including confidence scores, explanations, and chat interaction, can be useful for vulnerability detection and fixing.
• We offer practical recommendations for evaluating and deploying AI detection and fix models.
• We analyzed user feedback to reveal several actionable pain points, ranging from incomplete context to lack of customization for the user’s codebase.
• Although state-of-the-art AI-powered detection and fix tools show promise, they are not yet practical for real-world use due to a high rate of false positives and non-applicable fixes.