LLM Fingerprinting Through Semantic Variability

Thinking about potential transparency challenge in router-based ecosystems is valuable and the research questions are well-motivated. The technical execution seems good and the visualizations are really well done.

Here are some key areas to strengthen this work:

- Get even clearer on the impact / motivation: Wouldn't routing providers surface info about the model selection as a feature to sell? Are there specific scenarios where opacity matters? If so zoom in on those

- Make it clearer how we could reliably assign a single response to a specific model. The fingerprint distributions seem to heavily overlap such that attribution does not seem easy at all

- The robustness testing part is not integrated well enough with the fingerprinting. If the connection is not too strong the project becomes better by splitting it off and keeping each contribution focused on one core topic

- Add simple baselines or methods from the literature. How does your fingerprinting approach compare to those?

Thank you for your submission. This is an interesting paper and the results seem good.

As I understand what is proposed, I’m not clear on the benefits of the approach. I assume that the EO implementers will be transparent about which model(s) they are routing to. Certainly this is Martian’s intent / implementation. Can you foresee a use case where this is not so? If so, then it would be good to document this in the paper.

Being able to decode which model is actually behind the router is an interesting and good choice of problem. While this approach of semantic variability analysis is interesting, I worry that it's brittle in it's current state to (for example) the hyperparameters of the decoding algorithms being applied from "behind the router" (such as temperature, decoding strategy, etc).

It would be very interesting to see if a classifier could correctly map the signature t one of the models being specified.

Investigation 2 on how distraction affects tool use is interesting, but perhaps slightly off topic for this hackathon.

I like the approach of combining embedding metrics and prompt engineering. It would be good to explore further what dimensions can be added, and how well judges handle multiple attributes.

great idea to avoid semantic drift

This is phenomenal! My team and I will be using it moving forward.

This is a well-executed research project that tackles critical AI transparency challenges with impressive methods. The cognitive load experiments reveal fascinating insights about AI attention management, particularly that technical jargon causes 96% tool reduction despite 100% acknowledgment, mirroring human selective attention failures. The most important finding, in my opinion, is that meta-analysis of the task constitutes cognitive load that isn’t recognized. Identifying this is crucial for AI safety improvements and defending against potential prompt injections.

Furthermore, this project addresses the model-routing opacity problem for AI governance and compliance with an impressive and novel method: the "hyperstrings" concept and approach to probe model consistency patterns and successfully distinguishing between 11 different models with statistical significance demonstrates real technical achievement. The experimental design shows scientific rigor with systematic similarity measurements and clear statistical testing, while the preferential tool-dropping behavior (abandoning enhancements while preserving core functionality) provides actionable guidance for building robust AI systems.

The research successfully bridges theoretical AI interpretability with practical system transparency. While sample sizes could be larger for broader validation, the work creates exactly the kind of transparency tools needed as AI systems become more complex - understanding both "which model is running" and "how reliably it performs under real-world conditions." This is a strong contribution with immediate applicability to production environments.

Very cool

Interesting problem and interesting approach; I'm curious how the compression method could be used for other applications too, e.g. AI detectors?

The paper focusses on a single task domain (restaurant reservations) limits generality; diversifying tool-calling tasks or adding a qualitative error analysis might result in richer and different failure modes.

The project is reproducible and shows a systematic methodology for evaluating. The metrics are clear shown, the code is available in a github repository. 10/10

Very innovative solution!