Benjamin Steenhoek, PhD

benjaminjsteenhoekgmail.com Scholar github.com/bstee615 linkedin.com/in/ben-steenhoek

AI/ML researcher building code-editing and agentic systems for software engineering. Current focus areas are LLM post-training and agent harness engineering. Research interests include agent evaluation, LLM fine-tuning, software security, and program analysis.
Based in Des Moines, Iowa.

Experience

Senior Researcher

Fall 2024 - Present

Microsoft Code|AIDes Moines, IA

Drove research engineering for GitHub Copilot's primary agentic evaluation platform, building its scalable harness environment & tooling and onboarding key benchmarks. During my tenure, we grew it from a greenfield prototype to 1M+ monthly executions, and it now powers evaluations for flagship coding agents, including VS Code Agent, Copilot Cloud Agent, and Copilot CLI.
Led post-training, evaluation, and rollout of a production model for GitHub Copilot Inline Suggestions that increased suggestion acceptance by 10% and enabled a configurable eagerness feature that addressed 10+ long-standing community issues.

Research Intern

May 2023 - Aug 2024

Microsoft Code|AIDes Moines, IA

Conducted a user study on in-IDE AI-powered security vulnerability detection & repair with 17 professional developers across 24 projects, 6.9K files, and 1.7M+ lines of code; Closing the Gap, ICSE 2025 (21% acceptance rate).
Improved unit-test readability and maintainability by up to 21% through reinforcement learning; the fine-tuned Codex model was less than half the cost of GPT-4 and outperformed it on 4 of 7 quality metrics; Reinforcement Learning from Automatic Feedback, DeepTest @ ICSE 2025.

Research Assistant

2020 - 2024

Program Analysis & AI LabAmes, IA

Achieved state-of-the-art Big-Vul vulnerability detection performance (96.46 F1) by combining a dataflow-inspired graph model with an LLM. Our model trained in 9 minutes—75x faster than the strongest baseline—and detected 8.7 of 17 real-world vulnerabilities where baselines found none; DeepDFA, ICSE 2024 (code; 22% acceptance rate).
Built a C/C++ execution tracing engine from scratch for TRACED, ICSE 2024 (code; 22% acceptance rate), enabling execution-aware pre-training that improved complete path prediction by 12.4% and runtime value prediction by 25.2%. Built a Java counterpart later used for CodeSense, ICLR 2026.
Reproduced and evaluated 9 state-of-the-art vulnerability detection models across 2 datasets, uncovering substantial run-to-run variance and low agreement among model outputs; An Empirical Study of Deep Learning Models for Vulnerability Detection, ICSE 2023 (code; 26% acceptance rate).
Collaborated with Columbia University's ARiSE Lab and Carnegie Mellon University's CERT division; built open-source static and dynamic analysis tools including tree-climber and pal-tools.

More professional experience

Software Developer Intern

Ag Leader Technology, AgFiniti teamAmes, IA

Democratized public datasets by adding GIS capability for geolocation and remote sensing.
Widened customer reach by integrating AgFiniti with John Deere data platform.
Improved UX by modernizing satellite mapping interface with Javascript/Vue.
Enabled agronomic analysis by maintaining a domain-specific language using Antlr.

Teaching Assistant

Jan 2020 - May 2020

Iowa State University, COM S 227: Object-oriented Programming (Java)Ames, IA

Instructed 30 students in weekly labs and office hours.
Volunteered to create a GUI visualization for Conway's Game of Life (gol-gui) to increase student engagement.

Freelance Software Developer

Aug 2019 - Aug 2021

Roney Innovations (ecommerce retailer)Des Moines, IA

Collaborated with 2 other developers to create Amazon product listing web app using C#, ASP.NET Core, SQL Server, and Azure cloud services.

Selected Publications

See all on Google Scholar

CodeSense: a Real-World Benchmark and Dataset for Code Semantic ReasoningPaper Code
Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2026. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning. In The Fourteenth International Conference on Learning Representations (ICLR 2026), April 24–28, 2026, Singapore.
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent EvaluationPaper Code
Spandan Garg, Benjamin Steenhoek, and Yufan Huang. 2026. Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation. In IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN 2026).
Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDEPaper
Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, and Wei Le. 2025. Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE. In 2025 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’25), April 27–May 3, 2025, Ottawa, Canada.
Reinforcement Learning from Automatic Feedback for High-Quality Unit Test GenerationPaper
Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. 2025. Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation. In 2025 Sixth International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest ’25), April 27–May 3, 2025, Ottawa, Canada.
To Err is Machine: Vulnerability Detection Challenges LLM ReasoningPaper
Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T. Barr, and Wei Le. 2024. To Err is Machine: Vulnerability Detection Challenges LLM Reasoning. ArXiv.
Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability DetectionPaper Code
Benjamin Steenhoek, Hongyang Gao, and Wei Le. 2024. Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24), April 14–20, 2024, Lisbon, Portugal.
TRACED: Execution-aware Pre-training for Source CodePaper Code
Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2024. TRACED: Execution-aware Pre-training for Source Code. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24), April 14–20, 2024, Lisbon, Portugal. ACM, New York, NY, USA, 12 pages.
Do Language Models Learn Semantics of Code? A Case Study in Vulnerability DetectionPaper
Benjamin Steenhoek, Md Mahbubur Rahman, Shaila Sharmin, & Wei Le. (2023). Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection. ArXiv.
An Empirical Study of Deep Learning Models for Vulnerability DetectionPaper Code
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An Empirical Study of Deep Learning Models for Vulnerability Detection. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023).
Validating static warnings via testing code fragmentsPaper
Ashwin Kallingal Joshy, Xueyuan Chen, Benjamin Steenhoek, and Wei Le. 2021. Validating static warnings via testing code fragments. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021). Association for Computing Machinery, New York, NY, USA, 540–552.

More publications

Understanding and improving deep learning models for vulnerability detectionPaper
Benjamin Steenhoek. 2024. Understanding and improving deep learning models for vulnerability detection (Publication No. 31562057). Available from Dissertations & Theses @ Iowa State University; ProQuest Dissertations & Theses Global.
A Study of Static Warning Cascading Tools (Experience Paper)Paper Code
Guo, X., Joshy, A. K., Steenhoek, B., Le, W., & Flynn, L. (2023). A Study of Static Warning Cascading Tools (Experience Paper). ArXiv.
Refactoring programs to improve the performance of deep learning for vulnerability detectionPaper
Steenhoek, Benjamin. (2022). Refactoring programs to improve the performance of deep learning for vulnerability detection (Poster). Presented at: Iowa State University 6th Annual Research Day.
Refactoring programs to improve the performance of deep learning for vulnerability detectionPaper
Steenhoek, Benjamin. (2021). Refactoring programs to improve the performance of deep learning for vulnerability detection (Publication No. 28648161). Available from Dissertations & Theses @ Iowa State University; ProQuest Dissertations & Theses Global.