RAGE

There is real engineering here and I want to credit it first. RAGE is an actual working system, not a sketch: four layers, a deterministic SQL gateway, 232 regression tests, a product path, and a holdout set kept outside the knowledge base. The central idea is right — Crescendo attacks are a trajectory, not a single message, so stateless filters miss them, and anchoring cumulative drift Δ to the first turn T0 is a sensible way to catch the slow walk. The honesty notes are good too: you state plainly that the benchmark measures labeled-text detection, not attack success against a real LLM, that the victim in the demo is simulated, and that the EWMA ratchet is off in benchmark mode. That candor is worth a lot, and it is rarer than it should be.

A few things to work on, in order of importance:

1. The biggest issue is that the headline 80.6% recall comes from the two layers that are basically keyword and pattern matching (L1 regex + L2 TF-IDF), and your own ablation shows L2 plus multi-turn context adds only about 3 points over regex alone (77.8% to 80.6%). So Layer 3 — the cumulative-drift idea that is the actual research contribution — contributes almost nothing to the measured number, because the LLM judge is off in benchmark mode. The thing you are selling and the thing you are measuring are not the same thing. You need an evaluation where L3 is active and carrying weight, otherwise the central claim is untested.

2. The drift metric does not behave the way the method assumes, and your own star case shows it. In Table 4, δ jumps to 0.83 at T1 on a totally benign "product breakdown," and Δ actually falls from T2 to T4 (0.70 → 0.66) even as the attack escalates. So drift is not rising monotonically with malice — benign topic shifts spike it and real escalation sometimes lowers it. The Crescendo case looks blocked mostly because the deterministic gateway caught UNION and a non-allowlisted table, not because the drift signal tracked the attack. That means the gateway is doing the work, and L3 is riding along. Worth testing whether drift adds anything the gateway does not already catch.

3. The evaluation is calibrated to its own target, not an external standard. The CI requires recall to land in [75%, 85%], and the holdout is tuned to about 80%. That is circular — you are measuring against a band you chose. A frozen external benchmark, or at least Crescendo runs against a real LLM (which you list as future work), is what would make the number mean something. Right now 80.6% recall and 100% precision describe text classification on a self-built set, not defense against a live attacker.

4. Smaller points: 36 attack cases in the holdout is thin, so per-category recall is rough. The dual-use note is honest but the threshold values (τ ≈ 0.72, the fusion weights, the bands) are printed in the paper, which slightly cuts against your own "do not publish production thresholds" advice. And TF-IDF / HashingVectorizer embeddings are a weak base for semantic drift — a transformer embedding would test the idea on its real footing.

On presentation: the paper is clear and well-organized, the layer tables and the turn-by-turn Crescendo walk are easy to follow, and the two-layer evaluation split is explained well. It is a little dense with system detail in places, and a few claims in the results lean on tests the reader cannot see ("H1 validated in tests"), but overall the work is easy to understand.

Overall this is solid, honest engineering with a sound core intuition, held back by an evaluation that does not actually exercise the part that is novel. The next step is to turn the judge on, measure with L3 carrying real weight, and test against a live Crescendo attack rather than a labeled holdout calibrated to your own band.

Good engineering on a problem people usually underrate: the slow multi-turn attacks where every single message looks fine but the session drifts somewhere bad. Stacking the layers and putting a dumb deterministic SQL gate at the very end is the right instinct. What I liked most was how you evaluated it, testing on prompts that are not in your own threat list, and scoring with metrics tied to what actually happened rather than your detector's own opinion. You are honest that the benchmark checks text detection, not a real agent getting compromised, and that is the main thing to fix next, even one real Text-to-SQL agent end to end would make the case much stronger. TF-IDF is a weak choice for "semantic" drift, and one Crescendo scenario is thin, so more of both would help. Well built.

In this project the authors tackle the problem of agents directly interacting with sensible information as SQL databases. This has been a documented issue properly explain by the authors, to tackle it they propose a pipeline that evaluates a prompt in 4 stages by comparing the content to harmful instructions in fixed patterns, embeddings and semantic drifts to produce a score. The strategy tackles an important problem but testing on its scalability are lacking as DB’s can include thousand of transactions from different agents at very similar times. Likewise the paper needs to properly explain each step of the strategy in detail and properly structuring ideas in full paragraphs while connecting to displayed figures.

On impact, I would say that the general problem the paper is trying to address is quite important, given the security risks associated with the wide adoption of AI. In particular, future models may be both better exploiters of SQL access vulnerabilities and better conversationalists, so that they can either attack these systems directly or do social engineering so that a prompter with access to SQL databases retrieves harmful data and contaminates databases. This can create huge downstream problems on the scale of millions or even billions of dollars lost, or even lives lost (depending on the system). For this reason, I believe that trying to tackle security is a valuable endeavor. Moreover, the proposal itself shows an understanding of how the defense system would be applied in the real world, considering defense costs, reliability, and so on. The central contribution — both proposing the entire system and the L3 layer to avoid Crescendo-like attacks — is quite interesting and applicable.

On execution, I see that the layer is implemented, but crucially, there is no evaluation of L1+L2+MT vs L1+L2, only of L1+L2+MT vs L1, which is the entire proposal of the paper: to create a further layer that is robust against Crescendo. What there is, is a trajectory qualitatively shown in Table 4, whose blocks are produced by the deterministic SQL gateway and the ratchet, neither of which is the drift mechanism that is one of the central contributions. In fact, comparing Table 5 and Table 3, it seems that L1+L2+MT shows the same recall as L1+L2 — but it is not even clear what the difference between these two configurations is, so I am mostly inferring that. I understand that this is a 2-day hackathon, and such evaluations, as well as the tuning of the L3 layer, would need focused work, and I strongly encourage the continuation of this work. But as it stands, what is shown is the failure of the L1+L2 layer only, not that the L3 layer is ready to address this problem — which is still a good contribution, incentivizing more work on the L3 layer.

On presentation, the paper is quite confusing and poorly structured, and I had to use several turns of LLM conversation to get at what the paper is even trying to say in its results. There is no clear description of the attack data being used, there is the above-mentioned failure to show how the L3 layer is performing, and the text in general is badly formatted and confusing. Again, I truly understand the time constraints, and an inspection shows that the repository contains more information, but the paper's presentation should be evaluated on its own terms, and in this, the paper is lacking. This is not to discourage what I think is a great work direction, just to ask the authors to pay attention to how they are presenting their work, lest good work be rejected as not worth reading to begin with.