How the SimplyScan Engine Delivers Consistent Results
Run the same scan twice and you should get the same answer. Here is how the SimplyScan Engine makes that true · deterministic detection and scoring, with AI reserved for the reasoning rules cannot do.
By Daniel A · Kraftwire Software
· 6 min readRun the same scan twice and you expect the same answer. For a long time, security tools that lean entirely on a large language model could not promise that. Ask an LLM to grade a page today and it might say 82; ask it again an hour later, with nothing changed, and it might say 74. That is fine for brainstorming. It is not fine for a report you paid for, a trust badge you display publicly, or a monitor that is supposed to tell you when something actually got worse.
This is the single most common complaint about LLM-only security tooling. As one analysis of AI code review versus rule-based scanners puts it, non-determinism is a serious problem in security, "where consistency and reproducibility is paramount" (Fraim). We agreed. So we rebuilt how SimplyScan produces a score around a principle we call the SimplyScan Engine.
The one-line version
The deterministic engine detects issues and sets your score. The AI explains, triages, and catches the things rules cannot. The AI never touches the number.
That ordering is deliberate, and it is also where the industry has landed. The current best practice is not "AI instead of rules" or "rules instead of AI" · it is a hybrid where a deterministic engine does the detection and the model does the reasoning on top. Checkmarx describes their modern approach as keeping a deterministic core and using the LLM for what comes after (The New Stack). Reviews of 2026 AI security tools reach the same conclusion · the accountable pattern pairs pattern-matching detection with AI triage and remediation (Aikido). Academic work on hybrid static-analysis-plus-LLM pipelines reports large accuracy gains and steep false-positive reductions from exactly this split (SAST-Genius, arXiv).
Why an LLM alone drifts
A language model generates text by sampling. Even at the most conservative setting, the output is only mostly repeatable · Google's own guidance is blunt that a temperature of zero "does not necessarily" produce full determinism (Google Cloud). Fixing a random seed helps, but providers document cases where identical requests still diverge. So we do pin those settings · it is cheap and it reduces the wobble. We just refuse to depend on it. Anything that decides your score has to be something we control completely.
What the engine measures deterministically
Most of a SimplyScan report already came from deterministic analyzers · SEO, AI visibility, accessibility, GDPR compliance, domain health, and email authentication are all computed by pure code that returns the same result every time. The rebuild extended that same discipline to the two dimensions that used to be the model's opinion · security and speed.
For security, the engine looks for the things that are genuinely rule-detectable:
- Leaked credentials · provider API keys, tokens, private keys, and database connection strings, using the same regex-plus-entropy approach proven by open-source secret scanners like Gitleaks. Every match is masked before it ever leaves the scanner · we confirm the leak without echoing the secret back to you.
- Live database exposure · the engine makes real unauthenticated calls to detect tables anyone can read, so the finding is a demonstrated fact rather than a guess.
- Response headers · Content-Security-Policy, HSTS, clickjacking protection, and the rest, with severities calibrated to real-world impact.
- Exposed sensitive files, mixed content, and risky inline code patterns.
- Outdated dependencies · known-vulnerable versions of common front-end libraries, matched against a curated CVE database in the style of open-source scanners like retire.js.
The engine also grades threat exposure separately · whether a WAF or CDN sits in front of the app, whether sensitive files are reachable, whether server banners leak version numbers, and whether a login form has any abuse protection. These are the controls that stop automated scanning, scraping, and brute-force attempts before they land, and they get their own score.
For speed, the engine uses signals that are stable by design · real-user field data from the Chrome UX Report, which is a 28-day rolling average, plus static checks on render-blocking resources, image loading, fonts, third-party scripts, and bundle weight. We deliberately do not put a lab performance score on the critical path · those vary by roughly five points run to run even on an unchanged page (DebugBear), which is the exact flakiness we are trying to remove.
Your headline security score is then computed from those findings with fixed weights. Same site, same score · today, tomorrow, and on your teammate's machine.
Where the AI still earns its place
Rules are precise but literal. They will never notice that an authentication check is backwards, that an endpoint lets one user read another user's records, or that a block of AI-generated code quietly swallows an error and grants access instead of denying it. That kind of reasoning is where a model genuinely shines, and it is the reason SimplyScan understands AI-built apps better than a generic scanner.
So the AI stays · it reads your code, writes the plain-English explanation, generates the ready-to-paste fix prompt, and surfaces the semantic issues that no regex can see. Those insights are clearly marked as analysis. If the wording shifts slightly between runs, nothing breaks · because that wording was never allowed to move your score.
What this means for you
- Rescans are trustworthy. Fix an issue, scan again, and the number moves by exactly what you changed · not by model noise.
- Your trust badge is stable. It reflects the state of your site, not the mood of a model.
- Pro Monitoring alerts mean something. A drop is a real regression worth investigating.
- Free and Pro agree. The same site gets the same security score on both · Pro simply unlocks the fixes, the deep AI analysis, and repo scanning.
We keep the exact rule set, scoring weights, and model configuration private · that is the SimplyScan Engine, and it is ours. But the philosophy is not a secret · deterministic where it counts, intelligent where it helps, and honest about which is which.
Ready to see it? Run a free scan and check the score twice.