Research & Projects
Selected work in agent evaluation, cybersecurity, and simulation.
Selected research and coursework in autonomous-agent evaluation, cybersecurity, and simulation. Each entry identifies my role, collaborators, methods, and source availability.
Flagship research
ModelScope
Evaluation platform for autonomous AI agents
- Role
- Research fellow and contributor
- Collaborators
- Dr. Jaime C. Acosta; Army Research Laboratory (ARL)
- Methods
- Agent benchmarking, objective validation, telemetry, simulated environments
- Technologies
- Python, modular plugin architecture, Linux
Problem
ModelScope evaluates whether autonomous agents complete objectives, how efficiently they operate, what resources they consume, and whether they comply with policy.
My contribution
As an Army HBCU-MI SPARK research fellow, I am helping build ModelScope and designed a modular plugin architecture for configurable agents and objective ground-truth criteria. The platform captures task success, operational speed, resource usage, and policy-compliance metrics for auditable validation.
Evidence and status
The work is an active 2026 research deliverable connected to a collaboratively developed cyber Capture-the-Flag environment. Dr. Jaime C. Acosta owns and maintains the public repository; this page describes my contribution and does not imply sole authorship.
Research prototype
SpecGuard
AI-assisted vulnerability discovery system
- Role
- Designer and developer
- Methods
- Static analysis, LLM reasoning, RAG, pairwise-correctness evaluation
- Technologies
- Python, LangChain, ChromaDB, Ollama
- Evidence
- 1,999 real-world CVEs used as indexed ground truth
Problem
Vulnerability discovery tools often trade interpretability and coverage against one another. SpecGuard explores whether deterministic analysis and language-model reasoning can complement each other.
Approach and results
I built a hybrid framework that retrieves from a ChromaDB knowledge base indexing the hzhu721/vulnerability-specifications dataset. Detection logic covers memory corruption, use-after-free, and integer overflow, while an evaluation pipeline scores output against CVE ground truth at scale.
Source status: The repository is not yet public, pending documentation, licensing, reproducibility, and security review.
Advanced Algorithms coursework
Evacuation Optimization Simulator
Congestion-aware multi-agent routing
- Role
- Developer
- Methods
- Graph modeling, simulation, comparative algorithm analysis
- Technologies
- Dijkstra and Floyd-Warshall routing
- Outcomes measured
- Evacuation rate, time-to-safety, population throughput
Approach
I modeled edge capacity, congestion, and variable movement speed, then compared routing behavior across realistic constraints. The project connected algorithmic correctness with system-level outcomes rather than treating shortest paths in isolation.
Source status: No public repository is available for this course project.
CyberRIG project
Blue Cheese
Lightweight, modular cyber defense
- Role
- Developer and workflow designer
- Technologies
- Python, Suricata, Linux, Docker
Problem and approach
Blue Cheese explores a lightweight alternative to a full security monitoring stack, with structured alerts and modular analysis agents.
Implemented
Suricata JSON alert ingestion and normalization into structured records for downstream analysis.
In development
A live ingestion loop and a multi-agent workflow spanning orchestration, triage, correlation, investigation, and reporting. The workflow is designed; end-to-end autonomous defense and performance improvements have not yet been demonstrated.
Machine Learning research project
LLM Uncertainty Signals for Cybersecurity Analysis
Predicting when an LLM is likely to be wrong
- Signals
- Stated confidence, token entropy, top-token probability
- Models
- Logistic regression and shallow decision trees
- Evaluation plan
- AUROC, Brier score, calibration curves, feature ablation
Research question
Can confidence and uncertainty signals help predict the correctness of LLM cybersecurity judgments, including overconfident errors?
Approach and status
I am investigating signals from LLM outputs on cybersecurity scenarios with ground-truth vulnerable and non-vulnerable labels. The evaluation design compares signals individually and together using simple classifiers, discrimination metrics, and calibration. Results and performance claims are not yet available.