Do AI models leave fingerprints in the security of the sites they build?
I gave different AI coding models the same specification for the same web app, then tested the finished sites for the same security weaknesses. This is what happened, including the parts that did not work.
The 25 sites
Each one is a separate build of the same app, written by the model named above it and never edited afterward. Hover over a print to flip it, then click to look through the site.Tap a print to look through the site.
These are screenshots, because the sites are left unhardened and stay on my lab machines. Every site shows the same demo project, so the differences come from the builder. I took them on 3 and 4 Nov 2026, so dates on the pages read 4 Nov UTC. The prints are drawn for illustration and do not encode data.
Frontends often look AI-made. What about the code you cannot see?
People have started to recognize AI-built sites by how they look. I wanted to know whether something similar exists underneath, in security decisions. If a model keeps making the same mistakes, those mistakes could work like a fingerprint.
The main question: when different models independently build the same full-stack app under the same conditions, do they show repeatable, model-specific patterns in their security weaknesses? A side question grew from it: is there one universal flag that marks a weak AI-made site from the outside?
Same app, same tests, different builder
Every model builds ProjectHub, a project-management app with accounts, roles, tasks and file uploads. Each gets the same prompt, the same files and the same tools, and only generic functional feedback. No model is told anything about security.
A site counts as finished only when it passes the functional checks. Its source is then frozen and never repaired. Next, a fresh evaluator that does not know which model built the site goes through a fixed 35-item weakness checklist. I re-test the important findings live on a fresh copy.
The sites are intentionally left unhardened, so they only ever run on my own lab machines and are never put on the public internet.
25 sites took 150 tries
Getting models to finish the same app turned out to be most of the project. Each square below is one attempt, in date order by batch. Sonnet 5.5 and Haiku 5.5 usually finished. Haiku 4.5, GPT-6 Luna and the local models almost never did.
Within the local-model batches, squares are grouped by outcome instead of run order. “Lab or harness blocked it” is the label my harness recorded, and for Haiku 4.5 it hides a problem described below.
How the 150 attempts ended
What broke, and what I changed
What the sites got wrong
Darker cells mean a larger share of that model's sites. Nemotron has only 3 sites, so its shares move in big steps. This table covers the 23 fully evaluated sites; the 2 Haiku 4.5 sites had a shorter live check, described below.
Across the 23 evaluated sites I confirmed no injection flaws, though one Nemotron site served uploaded files in a way that allowed stored XSS. Most weaknesses cluster in things the functional tests never check, such as session secrets, rate limits and security headers. Models build what gets tested.
Some items do differ by model. All 10 Haiku 5.5 sites set a SameSite cookie, while 7 of 10 Sonnet sites and all 3 Nemotron sites left it out. This is an early hint, not a result: the samples are small and the Sonnet and Haiku 5.5 evaluators had different tools.
The 2 Haiku 4.5 sites: the full 35-item check could not run on these (the blinded evaluator was blocked part way), so I re-tested them live by hand on the items you can check over HTTP. Both had the same core weaknesses as the other models: a forgeable default secret that let me mint an admin session, no limit on login attempts, and no security headers. Both also left out SameSite, which is the opposite of every Haiku 5.5 site and looks more like the Sonnet sites. Only 2 sites, and only the visible items, so this is a hint and not a count.
The search for one sign that gives a weak AI site away
The idea was simple and useful: find something visible from outside a site that shows up on AI-built sites and also goes along with real weaknesses. A real flag has to pass three tests. It recurs across sites, it comes from the AI and not from my setup, and anyone can see it from outside.
I checked public signals on the 3 Nemotron sites: pages, headers, cookies and the 404 page. The evaluation labels were saved first, so this search could not influence them. Items from the checklist cover the 23 evaluated sites. None of the 9 candidates passed all three tests.
Everything that recurred traces back to how my lab runs the sites, or to the spec itself. The page markup, where a model's own habits could show, varied from build to build.
This negative result is useful, but it is not proof that no flag exists. It tells me what a real test needs: more models, the same public check on every set, a different hosting setup, and human-written sites to compare against.
How it was built
I set the research question, made the methods decisions and approved every run. AI agents did much of the implementation and analysis under my direction, including Claude Code and Codex. I kept them separate from the generators, so no model building a site could see the research plan.
Everything is logged: decisions, source hashes, failed attempts and the fixes that followed. Failures stay in the record because they are part of what the project found.
This is exploratory work and leaves out final statistics. It has 4 models, small samples and one evaluator per site. The checklist is a pilot, and the evaluator setup changed between the Sonnet and Haiku 5.5 sets. The count of 25 leaves out earlier calibration runs and one GPT-5.6 Sol site that was validated separately. I cannot yet claim that fingerprints exist.
Next: run the full 35-item check on the 2 Haiku 4.5 sites, get a Codex set working, run the same public check on every set, try another hosting setup, and add human-written sites to compare against.