Sentinel Trace: Open-Source AI Monitoring Dashboard With Pre-Training Data Tracing And In-Flight DPO Dataset Creation

We present Sentinel Trace, a monitoring and safety system for open-weight language models that combines real-time guardrails with interpretable failure analysis. Our architecture pairs a frontier model (OLMo-2 13B Instruct) with a lightweight guard model (Qwen3Guard-Gen 0.6B) that detects unsafe outputs and triggers automatic re-prompting. When adversarial prompts cause problematic responses, we make use of OLMoTrace to trace outputs back to the real pre-training data of the model, providing actionable insights to model providers for pre-training corpus filtering. All interactions are monitored through a Grafana dashboard tracking jailbreak attempts, detection rates, and re-prompting success. Using these insights, we implement a Direct Preference Optimization (DPO) feedback loop that refines training data based on guard classifications and traced origins, enabling the model to learn from failures instead of relying solely on post-hoc filtering. This approach demonstrates that combining runtime safety guardrails with training data traceability creates continuous improvement: interpretable monitoring informs data refinement, which improves model robustness against future attacks.