Mistral Large 4 for Cybersecurity: Strong With the Evidence in Hand, Weaker From Memory

Across 16 security benchmarks, Mistral Large 4 is in the upper half wherever the evidence is supplied or can be searched, and weaker where it must recall specifics.

At a Glance

• Strength: Upper half on 12 of 16 benchmarks, including all eight where the evidence is supplied or searchable. Narrowly first on SOC 2 auditing and both secure-code tests.
• Weakness: Terraform generation, misconfiguration triage (tied last) and refusing benign security requests.
• Research beats delivery: It gathers evidence well but its final detection rules and reported lines are only mid-field.
• Costly: Among the most expensive third of runs on most benchmarks, at list price.

What We Evaluated

• Threat hunting: Cyber Defense Benchmark [1].
• Incident response: SecRespond [2].
• Detection engineering: CTI-REALM [3].
• Vulnerability discovery and localization: RealVuln [4] and VulnGym [5].
• Compliance, security knowledge, secure code: TrustBench [6], CyberSOCEval [7], IaC-Eval [8], CyberSecEval [9] and four tasks of our own.

Each model ran once per benchmark^a on the same harness and scorer. Costs are at list price^b. Mistral Large 4’s weights are due to be released openly at the end of October 2026.

Results

Detection engineering · CTI-REALM

31 models · overall reward, 0 to 1

Across 50 tasks, the model reads threat reports, maps them to ATT&CK, finds the log tables, then writes a detection query and Sigma rule that are tested against a live log store. Each task had a 300-message budget, and we scored it as in our paper [10].

Mistral Large 4 scored 0.68, ninth of 31; an open-weight model leads with 0.74. At $45, only two runs cost more.

It is second of 31 at tagging ATT&CK techniques^d, but only 14th on the precision of its final query. Test its rules for false positives.

Vulnerability discovery · RealVuln

20 models · judge-matched real-impact F3

The model scans ten hard repositories with Codex CLI for 361 annotated vulnerabilities. We report the judge-matched real-impact F3 score^e.

Mistral Large 4 scored 58.4, sixth of 20 and within noise of the leader, Claude Fable 5.1 (61.1). At $10, no higher-scoring model was cheaper.

It finds as many flaws as its peers but often names the right file and the wrong line. Review the code around each reported line.

Vulnerability localization · VulnGym

16 models · judge-matched accuracy

Given an advisory and the repository, the model names where attacker input enters and where the unsafe operation happens, across 31 hard tasks in Codex CLI. We report judge-matched accuracy^e.

Mistral Large 4 solved 13 of 31, tied sixth of 16. GPT-6 Astra leads with 17. Its run cost $26, the fourth most expensive.

Threat hunting and incident response

Cyber Defense Benchmark: about fourth of 11 (judge-matched real-impact F1: 48.5; GPT-6 Astra leads with 57.9) for 68¢^c. A weak opening, then a short, mostly correct list of events: treat its output as leads, not a complete set.

SecRespond: fourth of 11 on detection (90.3) and third on remediation plans, but the slowest model, about 20 minutes per host. Ten of the 11 models score 83 to 94.

Other benchmarks

First on SOC 2 auditing and secure code; tied last on misconfiguration triage.

• TrustBench (SOC 2 audit score %): 70.4, 1st of 11, level with two others.
• Control crosswalk (ours; NIST CSF 2.0 to six frameworks, top-1 %): 46.8, 3rd of 5. Leader: GPT-6 Sol, 58.6.
• CWE inference (ours; CVE to root-cause CWE, top-1 %): 59, 5th of 12. Leader: Gemini 3.7 Flash, 65.
• CyberSOCEval Threat Intel (exact match %): 56.5, 6th of 14. Leader: GPT-5.6 Terra, 59.9.
• Package role (ours; macro F1): 74.6, 4th of 12. Leader: GPT-5.6 Luna, 79.9.
• Misconfiguration triage (ours; F1): 74, tied 9th of 12 (last). Leader: GPT-5.6 Terra, 87.
• IaC-Eval (Terraform pass@1 %): 29.9, tied 6th of 12. Leader: Gemini 3.7 Flash, 48.9.
• CyberSecEval Instruct / Autocomplete (secure-code pass rate %): 70.8 / 67.9, 1st of 11 on both, within noise.
• CyberSecEval MITRE (% replies that help an attack, lower is better): 18.7, 4th of 11. Leader: an open-weight model, 12.2.
• CyberSecEval MITRE FRR (% benign requests refused, lower is better): 6.8, 10th of 11. Leader: GPT-5.6 Luna, 0.0.

Terraform rejected 126 of its 458 configurations at plan time (the leader: 32), mostly over missing or non-existent provider arguments. On triage, a third of what it flags is a false alarm.

Why It Performs This Way

Independent tests by Artificial Analysis [11] agree: recall is its weakest ability. It places higher where the task hands it evidence to read or search^f.

What This Means

• Use it where the task supplies evidence: code review, log investigation, audits.
• Check its final output: detection rules, event lists and reported lines.
• Gate its Terraform with terraform plan; treat its triage calls as a shortlist.
• Compare against a cheaper open-weight model first: on most benchmarks one scored as well or better.

Access Mistral Large 4 Through Averlon’s Inference Platform

Averlon’s Inference Platform is purpose-built to understand cybersecurity tasks, select the best-fit open or closed model, and evaluate every result. We’ve now added support for Mistral Large 4.

Contact support@averlon.io to get started. We’re offering $50 in credits.

Notes

a. One run. Each benchmark ran once per model; gaps within one standard error (3.4 points on CTI-REALM, 4.0 on RealVuln, 8.9 on VulnGym) are within noise.

b. Costs. List price $1.36 / $4.18 per million input / output tokens. We were billed half under a launch discount; other models’ costs are as recorded.

c. Cyber Defense Benchmark. Judge-matched real-impact F1: two LLM judges match each submitted event to the answer key. One public campaign, default settings for all 11 models on the same day; Mistral Large 4’s earlier run at maximum reasoning scored 34.8.

d. CTI-REALM steps. Our own per-step measures, not the benchmark’s; ranks among 31 models.

e. Judge match. RealVuln and VulnGym match findings to an answer sheet with a fixed-rule script that rejects correct findings phrased or located slightly differently. Two LLM judges decide instead; a finding counts only when both agree. RealVuln’s score counts only vulnerabilities with real impact.

f. One model, not a rule. No correlation between Artificial Analysis scores and ours survived correction for multiple testing.

References

[1] Chona et al. (2026). Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps. arXiv:2604.19533.

[2] Wang et al. (2026). SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response. arXiv:2607.26791.

[3] Chakraborty et al. (2026). CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities. arXiv:2603.13517.

[4] Pellew and Raza (2026). RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code. arXiv:2604.13764.

[5] Ji et al. (2026). VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection. arXiv:2608.02001.

[6] TrustBench (2026). TrustBench v0.2: SOC 2 compliance auditing benchmark. github.com/varungurnaney/trustbench.

[7] Deason et al. (2025). CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning. arXiv:2509.20166.

[8] Kon et al. (2024). IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs. NeurIPS 2024. github.com/autoiac-project/iac-eval.

[9] Bhatt et al. (2023). Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models. arXiv:2312.04724.

[10] Garg et al. (2026). Unconstrained CTI-REALM with Open-Weights. arXiv:2610.14180.

[11] Artificial Analysis (2026). Mistral Large 4 model page. Data retrieved 13 October 2026.