Human, Script or AI? Spotting AI-Driven Attacks in the SOC

1Executive summary

• Where AI is in an intrusion, orchestration has largely displaced assistance. Anthropic reports that a majority of the operations described in its September 2026 threat assessment were enabled by AI through direct execution or orchestration rather than as analytical support, with multi-agent frameworks running for hours or days under limited human oversight.¹³ That is a statement about documented AI-misuse cases, not a measure of how common AI involvement is across all intrusions. The operational consequence is still real. Where a SOC judges AI to be in play, the working assumption should now be orchestration rather than a human operator with a good assistant.¹³

• A security operations centre (SOC) remains unlikely to prove from logs alone that an attack is driven by artificial intelligence (AI). Current evidence supports confidence-based assessment, not definitive attribution.^{1, 2, 3, 6} That limitation matters less than it did. The more useful question is no longer whether AI is present but what changes if it is.

• AI-driven cyber activity usually reuses familiar tactics, techniques, and procedures (TTPs). The observable difference is more often tempo, adaptation, sequencing, and breadth across telemetry.^{1, 2, 3, 7, 12}

• The stronger observable indicators are compressed kill-chain timing, rapid troubleshooting, environment-specific payloads, high command variation, and cross-domain progression.^{2, 4, 6} Two publicly disclosed incidents now give several of these indicators observed anchors rather than research-setting estimates.^{14, 15} Emerging research methodologies, including defensive prompt injection, command lexical fingerprinting, canary tripping rates, and semantic diversity measurement, may add further signal as they mature. Section 5 summarises these.^{8, 9, 10, 11}

• Large language model (LLM) driven activity is not always noisier. Noise is useful context when agents fail, retry, or explore too broadly. It is not a reliable attribution method.^{3, 4, 6, 12}

• The practical ceiling today is a confidence score. A SOC can combine the metric indicators in this document with subjective binary factors, for example whether a payload mutated faster than a human plausibly could, into a weighted likelihood rating for suspected AI involvement. No single indicator supports a definitive determination.^{1, 2, 6}

• Incident response fundamentals hold. Two distinct pressures now sit on them. The first is decision speed inside an incident, where stage compression puts weight on pre-approved containment, clear authority, and cross-domain correlation.⁵ The second is detection durability across incidents. Anthropic describes adversaries able to iterate past a newly deployed detection faster than defenders can build and ship the next one. That is a different problem from containment arriving late, and it is not solved by the same controls.¹³ AI and classical machine learning detection techniques are emerging that look promising, and vendor tooling built on these techniques is anticipated within 6 to 12 months.^{8, 10, 11} Teams can review the techniques in Section 5 now, and start planning for in-house AI expertise or vendor adoption as these options mature.

2What changes and what does not

3SOC observable indicators

The indicators below are largely non-distinct and should be treated as lower-confidence signals. Each overlaps with legitimate automation, testing activity, and skilled human operators. None is sufficient on its own. They are presented as scoring inputs that may raise or lower suspicion when several appear together in one coherent sequence.^{1, 3, 6} The combination with speed matters most. High entropy or adaptive behaviour arriving at machine speed weakens both the human-operator explanation and the static-script explanation at once.

4Are AI-driven attacks noisier?

Public evidence does not support a universal rule that AI-driven attacks are always noisier than humans or standard automation.^{3, 6} They can be noisy when agents fail, retry, produce malformed payloads, explore too broadly, or loop through troubleshooting.^{4, 6, 12} They may become quieter as tooling improves, when operators constrain the agent, or when the activity is mostly orchestration of existing tools.^{1, 2, 3} Treat noise as a contextual signal. It can increase suspicion when paired with speed, adaptation, and cross-domain sequencing. It is weaker as a standalone AI label.^{1, 2, 6}

Volume is worth separating from noise. The Hugging Face intrusion was not subtle, and the sheer count of logged actions was what made reconstruction possible at all. The same case shows that a high action count can coexist with an operation that ran for a weekend before it was stopped.¹⁴ Volume alone did not shorten the detection window.

5Emerging methodologies for distinguishing AI-driven attacks

Best practice methodologies for distinguishing LLM-driven attacks are not yet firmly established. Even so, numerous promising potential indicators of AI-driven attack are emerging from academic and industry researchers. The more established methodologies in the research literature are summarised below.

These findings are indicative-only. They are not a brief for production deployment. At best, these methodologies provide an ensemble of techniques which could be used to score the likelihood an attack is LLM-driven, rather than provide a definitive determination.

I suggest these methodologies inform long-term plans for adopting AI and AI-skilled professionals in the SOC.

I anticipate major security vendors will release tooling within 6 to 12 months that applies these same principles.

Defensive prompt injection

Prompt injections embedded within system response payloads or other attacker-visible telemetry are emerging as a technique both to disrupt and fingerprint AI attackers.

One example is the Trace framework. It uses a two-step methodology that includes terminal command fingerprinting (see below) and prompt injection to attempt to extract the attacking agent's system prompt.⁸

Prompt injections are optimised for specific models and model families detected utilising LLMmap, an open-source research tool for prompt-based LLM fingerprinting.⁹ Success on a prompt injection can therefore contribute not only to detecting the presence of AI but also to fingerprinting the precise model and instructions the attacker is utilising. The Trace team were able to achieve an average 81.9% system prompt extraction rate using fingerprint-guided prompt injection payloads.⁸

Command lexical fingerprinting

A large volume of research papers and tooling have been produced to detect AI-generated code and identify the associated model family. The Trace team were able to predict with approximately 98% accuracy the attacking model in their simulation by applying classical machine-learning models to attacker terminal commands.⁸

"For example, DeepSeek chains find with -exec grep. Claude Opus constrains searches with -maxdepth. GPT-5.4 reads files via sed. Such idioms emerge from pretraining corpora and we observed that they persist across different scaffolds."⁸

Fingerprinting work has since extended past the terminal. Research published in 2026 applies the same approach to browser agents using interface interaction traces, a surface this document does not otherwise address and one that a SOC relying only on command telemetry would miss.¹⁷ A 2026 survey of terminal agents gives a broader view of the environment these command idioms emerge from.¹⁸

Canary tripping

Recent research indicates that LLMs are more than twice as likely as human attackers to fall for honey tokens and other canary resources.¹⁰ More capable models also appear to trip more canaries.¹¹ Where historical baseline data is available, the tripping of an unusually high number of canaries may therefore serve as an additional measurable factor in determining the probability an attack is LLM-driven.

This signal may degrade where agents share memory. Later work argues that a honeytoken cannot simultaneously be harmless to trusted agents and remain unrecognisable to an attacker able to pool information across instances, because a coalition probing repeatedly can learn to separate real objects from decoys.¹⁹ That is a theoretical result rather than a measured one, and it does not contradict the higher tripping rates above. It does suggest the signal is most reliable early and against single agents, and least reliable against the kind of coordinated swarm seen in disclosed incidents.^{14, 19}

Semantic diversity of high speed and high volume successive commands

LLM agents are unique in their ability to produce contextually-coherent retries with high-semantic-diversity in rapid succession. Each command is a meaningfully different reformulation, adapted to the previous command's output.

This behaviour is distinct from human activity (high semantic entropy, but slow cadence) and from fuzzy script-driven commands (high volume, low semantic entropy or combinatorial mutations).

Exact thresholds for the timings and volume indicative of AI-attack do not exist. Some research attack simulations have centred on an average of 1.7 seconds between local model commands, however this timing is likely longer where frontier APIs are used.⁶ A disclosed production incident provides a coarser but real anchor, with a documented 31 seconds between a failed login and a working corrective payload, diagnosis included.¹⁵

I therefore suggest that the overall volume and lowest latency of high-entropy request clusters (both over short intervals and multi-hour sessions) may serve as an additional useful, but not definitive, metric in rating the likelihood of a sequence of actions as being AI-driven.

6Detection principles

Behaviour in this document means the combination of measurable execution features described above, such as timing between stages, retry cadence, payload shape and diversity, and sequence coherence across telemetry, rather than any single indicator. These features can serve as weighted inputs to a scoring model for the likelihood of an LLM-driven attacker, alongside subjective binary factors. Most organisations are more likely to receive this capability through vendor tooling than to build it in house.^{6, 8, 10, 11}

• Detect behaviour, not AI attribution. Label activity as suspected AI-assisted only with stated confidence.^{1, 2, 3, 6}
• Measure time compression between stages, especially recon to exploit and foothold to lateral movement.⁴
• Correlate telemetry across endpoint, identity, network, cloud, WAF, SaaS, application, and SIEM (security information and event management) sources.^{1, 2, 5}
• Baseline commands, payload shape, user behaviour, service behaviour, and admin automation windows.^{5, 6}
• Alert on rapid adaptive retries, payload mutation, and environment-specific payload generation.^{2, 6}
• Use deception controls, honeypots, and canary assets as signal sources, not as proof that an agent is present, and treat their value as strongest against single agents rather than coordinated ones.^{6, 19}
• Record the assessment basis, false positives considered, and confidence level in incident notes.^{1, 3, 5}

7Response implications

Incident response fundamentals do not change because suspected AI is involved.⁵ Two things about the operating environment do.

The first is tempo inside an incident.^{4, 5} SOC and cyber defence teams commonly maintain pre-approved containment paths, documented kill-chains, dependency maps, clear authority for rapid containment, rehearsed playbooks, vendor escalation paths, and SIEM plus XDR (extended detection and response) correlation. These are worth reviewing against compressed timelines.⁵ Identity, host, cloud, and application containment are usually considered before broad network containment. Network containment remains available when speed and blast radius justify it. It is not typically treated as the default response.⁵

The second is detection durability across incidents. Faster containment does not address it. Anthropic's assessment is that where defenders could previously slow an attacker by deploying a new detection, a capable adversary may now close that loop and bypass the detection faster than the next one can be developed and deployed.¹³ Reported behaviour in the same assessment includes agents modifying malware autonomously when a detection fired and iterating until evasion succeeded.¹³ If that pattern holds, the useful measure is not only mean time to contain but how long a newly shipped detection survives contact. Most teams do not currently track that.

A third consideration is practical rather than strategic. In at least one disclosed incident, responders found that commercial model guardrails initially refused to analyse genuine attack artefacts, and fell back to self-hosted models to complete the forensic work.¹⁴ Teams that expect to use AI assistance during response should establish before an incident whether their tooling will process hostile material.

8Limitations

No reliable public evidence currently proves AI attribution from logs alone.^{1, 2, 3, 6} Many indicators overlap with skilled human operators, traditional automation, red team tooling, and security operations activity.^{1, 3, 6} In practice, teams generally work from a confidence-based assessment supported by cross-domain telemetry, threat intelligence, observed timing, and clear caveats.^{1, 2, 5} An assessment of this kind reflects current common practice rather than a ceiling. Posture can continue to improve as baselines, telemetry coverage, and detection features mature.

The reframing in Section 1 carries its own limitation. The finding that a majority of documented operations were AI-orchestrated describes the set of AI-misuse cases one vendor detected on its own platform.¹³ It is not a measure of how often AI appears across intrusions generally. It should not be read as one.

The emerging techniques in Section 5 have so far been demonstrated in research settings and are likely to require additional work before they can be operationalised and supported in day-to-day operations.^{8, 10, 11} Organisations pursuing them may need new or expanded capability, though interview research with SOC practitioners suggests the binding constraint is less often specialist headcount than organisational readiness and the management of over-reliance, with adoption proceeding under competitive pressure regardless of assessed feasibility.¹⁶ Alternatively, organisations may prefer to adopt the commercial tooling expected to incorporate these techniques within 6 to 12 months.

9References

Numbered citations throughout this document refer to the sources below.

Confidence language throughout reflects the public evidence base as at 23 September 2026.