Why frontier LLMs failed a new cyber defence benchmark
Frontier LLMs failed Simbian’s new cyber defence benchmark, exposing the gap between cybersecurity knowledge and real-world SOC threat hunting.
Read More
Cybersecurity practitioners aren’t immune to worries about AI taking their jobs. A new cyber threat and defence report from CyberEdge shows how intensely that anxiety has entered the profession. In a survey of 1,200 IT security professionals across 17 countries, 80% said AI will significantly reduce the number of people needed to perform their current role at some point – and 46% expect that pressure within two years.
That makes it a near-term career concern rather than a vague unease about one of many possible futures.
But the same report also found that 96.8% of organisations now consider at least one AI skill important when hiring for IT security roles. The most sought-after skill was experience using AI to improve and automate incident response and remediation, selected by 43.8% of respondents.
So instead of just asking if they need fewer people now they have AI, they’re asking who knows how to work with this thing.
The wider jobs data tells a similar story. The World Economic Forum’s 2025 Future of Jobs report projects major disruption by 2030:
It also says 41% of employers expect to reduce workforces where AI automates tasks, while 77% plan to upskill workers.
If that sounds dramatic, it is. But it’s not a simple replacement curve. WEF lists AI and big data, networks and cybersecurity, and technology literacy among the fastest-growing skills through 2030. Meanwhile, ISC2 estimates a global cybersecurity workforce gap of 4.76 million people, up 19.1% year-on-year.
A profession with a multi-million-person shortfall is highly unlikely to become overstaffed.
Earlier this week we wrote about Simbian’s new cyber defence benchmark, and it’s a useful reality check here. The company tested 11 frontier LLMs across 105 attack procedures and 884 runs. None passed. The best model, Claude Opus 4.6, averaged 46% MITRE tactic performance and found just 4.5% of malicious flags.
The models were asked to hunt through real Windows telemetry, including Sysmon and Security logs, and submit exact malicious timestamps.
And Simbian’s conclusion was straightforward: “an LLM alone is not a defender.” What makes it valuable in cyber defence is the harness around it – organisational context, retrieval, tool access, investigation loops and checks that stop the agent giving up too early.
So yes, AI will reduce some cyber tasks. Tier-one triage, repetitive reporting, basic enrichment and first-pass remediation are obvious candidates.
But cybersecurity is not only task execution. It involves judgement under pressure. It requires the ability to decide whether a strange login is an intern, an admin script, or the beginning of lateral movement. You have to know when the tool is confidently wrong.
CyberEdge’s own advice fits neatly here: security professionals should stay ahead in areas where experience and human judgement still trump AI, while building AI security expertise.
That’s the career lesson. Don’t bet against AI, but don’t mistake automation for autonomy.
The future cyber professional is not replaced by the model. They’re the person who knows when to trust it, when to test it, and when to take control away from it.
Join the newsletter to receive the latest updates in your inbox.
Frontier LLMs failed Simbian’s new cyber defence benchmark, exposing the gap between cybersecurity knowledge and real-world SOC threat hunting.
Read More
New research shows AI models can now autonomously find vulnerabilities and generate exploits. Explore what this means for cybersecurity risk and defence strategies.
Read More
Two new reports reveal gaps in incident response readiness, from poor coordination to visibility blind spots. Learn how connected ecosystems improve cyber resilience.
Read More