Curriculum Vitae

Senior Researcher at Microsoft, working on developer tools for secure software engineering and next-generation agent systems. Research interests are the intersection of machine learning and software engineering.

Education

• Thesis: Understanding and improving deep learning models for vulnerability detection.

• Thesis: Refactoring programs to improve the performance of deep learning for vulnerability detection.
• Released code as open-source library cfactor.
• GPA 3.91/4.00.

• Magna Cum Laude honors (GPA 3.84/4.00).

Professional Experience

• Building developer tools for secure software engineering and next-generation agent systems for software engineering.
• First author, "Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE" (accepted ICSE 2025).
• Co-author, "Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation" (accepted CAIN 2026).

• Summer 2023: Conducted research on improving large language models such as Codex using reinforcement learning, resulting in "Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation", DeepTest @ ICSE 2025.
• Summer 2024: Conducted research on AI-powered vulnerability detection and repair in the IDE, contributing to "Closing the Gap", ICSE 2025.

• First author, "An Empirical Study of Deep Learning Models for Vulnerability Detection" (accepted ICSE 2023; 26% acceptance rate).
• First author, "Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection." (accepted ICSE 2024).
• Contributing author, "TRACED: Execution-aware Pre-training for Source Code" (accepted ICSE 2024).
• Contributing author, "Validating static warnings via testing code fragments" (accepted ISSTA 2021; 27% acceptance rate).
• Collaborated with ARiSE lab at Columbia University and CERT lab @ Carnegie-Mellon University.
• Improved experiment iteration time by building tools for static analysis (tree-climber) and dynamic analysis and code generation (pal-tools).
• Enabled collaboration on experiments by acquiring and maintaining bug benchmarks for use in experiments.

• Fall 2021: Democratized public datasets by adding GIS capability for geolocation and remote sensing.
• Summer 2020: Widened customer reach by integrating AgFiniti with John Deere data platform.
• Summer 2019: Improved UX by modernizing satellite mapping interface with Javascript/Vue.
• Summer 2018: Enabled agronomic analysis by maintaining a domain-specific language using Antlr.

• Instructed 30 students in weekly labs and office hours.
• Volunteered to create a GUI visualization for Conway's Game of Life (gol-gui) to increase student engagement.

• Collaborated with 2 other developers to create Amazon product listing web app using C#, ASP.NET Core, SQL Server, and Azure cloud services.

Publications

CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning

Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2026. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning. In The Fourteenth International Conference on Learning Representations (ICLR 2026), April 24–28, 2026, Singapore.

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation

Spandan Garg, Benjamin Steenhoek, and Yufan Huang. 2026. Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation. In IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN 2026).

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. 2025. Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation. In 2025 Sixth International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest ’25), April 27–May 3, 2025, Ottawa, Canada.

Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE

Benjamin Steenhoek, Siva Sivaraman, Renata Saldivar, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, and Wei Le. 2025. Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE. In 2025 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’25), April 27–May 3, 2025, Ottawa, Canada.

Understanding and improving deep learning models for vulnerability detection

Benjamin Steenhoek. 2024. Understanding and improving deep learning models for vulnerability detection (Publication No. 31562057). Available from Dissertations & Theses @ Iowa State University; ProQuest Dissertations & Theses Global.

To Err is Machine: Vulnerability Detection Challenges LLM Reasoning

Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T. Barr, and Wei Le. 2024. To Err is Machine: Vulnerability Detection Challenges LLM Reasoning. ArXiv.

Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection

Benjamin Steenhoek, Hongyang Gao, and Wei Le. 2024. Dataflow Analysis-Inspired Deep Learning for Efficient Vulnerability Detection. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24), April 14–20, 2024, Lisbon, Portugal.

TRACED: Execution-aware Pre-training for Source Code

Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2024. TRACED: Execution-aware Pre-training for Source Code. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24), April 14–20, 2024, Lisbon, Portugal. ACM, New York, NY, USA, 12 pages.

Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection

Benjamin Steenhoek, Md Mahbubur Rahman, Shaila Sharmin, & Wei Le. (2023). Do Language Models Learn Semantics of Code? A Case Study in Vulnerability Detection. ArXiv.

An Empirical Study of Deep Learning Models for Vulnerability Detection

Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An Empirical Study of Deep Learning Models for Vulnerability Detection. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023).

A Study of Static Warning Cascading Tools (Experience Paper)

Guo, X., Joshy, A. K., Steenhoek, B., Le, W., & Flynn, L. (2023). A Study of Static Warning Cascading Tools (Experience Paper). ArXiv.

Refactoring programs to improve the performance of deep learning for vulnerability detection

Steenhoek, Benjamin. (2022). Refactoring programs to improve the performance of deep learning for vulnerability detection (Poster). Presented at: Iowa State University 6th Annual Research Day.

Refactoring programs to improve the performance of deep learning for vulnerability detection

Steenhoek, Benjamin. (2021). Refactoring programs to improve the performance of deep learning for vulnerability detection (Publication No. 28648161). Available from Dissertations & Theses @ Iowa State University; ProQuest Dissertations & Theses Global.

Validating static warnings via testing code fragments

Ashwin Kallingal Joshy, Xueyuan Chen, Benjamin Steenhoek, and Wei Le. 2021. Validating static warnings via testing code fragments. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2021). Association for Computing Machinery, New York, NY, USA, 540–552.

My Projects

Primary developer

• cfactor: Scalable, policy-driven refactoring for C programs (Python/srcML).
• tree-climber: Scalable program analysis tools for C built on tree-sitter (Python).
• pal-tools: Dynamic analysis and code generation, using Intel Pin (C++) and LLVM (Python).
• rarl: Reproduction of Robust Adversarial Reinforcement Learning (Pinto et al. 2017) (PyTorch/stable-baselines).

Team project

• animal-cognitive: Deep reinforcement learning models with embodied animal cognition (PyTorch/rllib).
• precise-interrupts: Reproducing a historical interrupt handling paper in ARM architecture (C++/gem5).

Technical Skills

• Programming Languages: Proficient in Python and C#. Knowledge of C++, Java, JavaScript, SQL.
• Machine Learning & data scraping: PyTorch, rllib, pandas, numpy, Selenium, beautifulsoup.
• Web Development: Vue, ASP.NET Core, .NET Framework, SQL Server, Azure Functions, ACI, VMs, ML Studio.
• Computer architecture and program analysis: Antlr, LLVM, Intel Pin, gem5, abstract interpretation, fuzzing.
• DevOps: Git, Azure DevOps, and CI/CD, Slurm batch processing, Linux server administration.

Leadership

• Science education outreach at Greenville County Juvenile Detention, Fall 2018/Spring 2019
• Vice president of Phi Beta Chi society, Spring 2018