Hybrid AI Agent Skill Auditor

The dual-path architecture — Rust-based fast static filtering followed by Docker-sandboxed dynamic analysis — is a sensible design for the threat model, and the canary token + literal-IP routing combination for exfiltration detection is a genuinely clever engineering choice. The use of SurrealDB as an in-memory store for the static phase shows attention to latency.

The main weakness is that the paper's result section is very thin relative to the claims. You mention a 100-skill evaluation dataset (50 benign, 50 malicious) but report only two individual case studies (032 and 065). There are no aggregate precision, recall, false positive rate, or AUC metrics across the full 100-skill corpus. The reader cannot assess whether the framework works beyond these two examples. Please add an aggregate results table showing true positives, false positives, false negatives, and execution time distribution across the full 100 skills. Additionally: (1) The 40-45 second per-skill runtime for the dynamic phase is notable — for a skill marketplace with thousands of uploads this could be a bottleneck; what is the per-hour throughput? (2) The paper would benefit from discussing at least one adaptive evasion scenario: how would a malicious actor evade canary token detection specifically?

The engineering direction is solid and worth pursuing — the results just need to catch up with the architecture section.

I would conditionally approve this project — it demonstrates genuine engineering merit and addresses a real, validated threat, but has gaps that need addressing before it can be considered a strong contribution.

While the system demonstrates genuine engineering merit, several critical gaps prevent full approval in its current form. The most significant omission is the absence of aggregate evaluation metrics — the paper presents only two case studies but never reports precision, recall, F1, or overall accuracy across the full 100-skill dataset, leaving the reader unable to determine how many of the 50 malicious skills were actually caught or how many false positives occurred on the 50 benign ones. The related work section is extremely sparse, citing only two references, which suggests the authors are either unaware of or have not engaged with several closely related 2026 papers including SkillScan (which analyzed 42,447 skills), Agent Audit, and the Hierarchical Triage Framework — this considerably weakens the novelty argument. The dataset itself, at 100 skills with a 50/50 split, functions as a proof-of-concept rather than a convincing evaluation, especially when competing work operates at scales of 3,984 to 42,447 skills. Furthermore, without any baseline comparison against existing tools such as SkillScan or even a simple regex-based scanner, there is no way to assess whether the dual-path approach actually improves detection over simpler alternatives. Finally, scalability remains unaddressed — at 40 to 45 seconds per skill with Docker overhead, the paper does not discuss how this framework would function at marketplace scale where thousands of skills require auditing.

The authors address a very current and relevant issue here. Specifically, they talk about “malicious or poisoned” AI/ML “skill” functionality, as well as “supply chain” type attacks using “tool tampering,” and “indirect prompt injection” which ultimately allow attackers to remove data. Their “dual-path” approach makes sense and their specific “security engineering” is by far the best aspect of this work. In particular, I like how they use literal IP-routing (of both curl and pbpaste), combined with “shimmed system binaries” (for example, the curl and pbpaste system binaries are modified to catch exfiltrated data before the skill gets alerted). I also appreciate how they paired a simple (but fast) Rust static filter with a heavier dynamic sandbox for managing costs.

There are several possible avenues for improving this work.

(1) Report total statistics regarding performance on your entire corpus. Although the paper mentions a balanced collection of 100 skills (50 good, 50 bad) there are only two examples given. Your central assertion of "effective protection" requires a complete confusion matrix along with some metrics of accuracy, precision, and recall. You will need to include the false positive rate on the 50 good skills if you want to know if this technology is deployable.

(2) Disable one of the paths. Provide the detection rate for static-only vs. dynamic-only vs. combined. This will show the reader what the dynamic sandbox adds over the static filter, and what the LLM judge adds over the deterministic baseline decision that you have already made.

(3) Characterize the LLM judge. Because the Gemini 2.5 has the last word in this scenario, provide information about how many times it agrees with your deterministic baseline. Also describe the sensitivity of the verdicts that the Gemini 2.5 returns based on the phrasing of the prompts you send it and the choice of models you make. Then readers can evaluate the extent of their reliance on the automated judgment.

(4) Discuss evasion strategies. The assumed threat model posits that each skill uses a defined tool call schema. Therefore, you should describe what happens when a malicious skill avoids your shimmed binaries, stalling beyond your established time limit, or detecting your sandbox. Consider testing a few adversarial skills that were designed specifically to avoid being detected.

(5) Describe the origin of your dataset. A single paragraph describing where your 100 skills originated from (from marketplace skills, crafted by hand, etc.) would be helpful to other researchers evaluating how representative your study is.

Overall: solid security engineering that was carefully thought out and applies to a previously under researched area. However, the missing piece in the research is actually providing the necessary benchmarks because at present your described 100 skill evaluation is represented by only two anecdotal examples.