Hunt in the Cloud with Splunk

What this lab covers

You are handed a pattern of unexplained service outages and the CloudTrail logs covering them, and asked to determine whether anything malicious is behind it using Splunk.

The framing matters. You are not told there has been a breach, which is how these investigations actually start, and the first real question is whether the activity you are looking at is operational or hostile.

It assumes basic log analysis experience and no prior AWS knowledge, and suits analysts and engineers who own availability as much as security.

How to investigate cloud activity in Splunk

CloudTrail records one event per API call with the identity, source, action, parameters and outcome. In Splunk that becomes a searchable field-based dataset, which turns investigation from reading events into asking questions of the whole set.

The productive queries are aggregations rather than lookups. Which identities acted in this window, which actions are new for each of them, which calls failed and where they came from. Each narrows the population, and the sequence of narrowing is the investigation.

Distinguishing operational from hostile activity comes down to context. Automation acts constantly, from consistent sources, with a stable set of actions. A deployment role deleting resources is routine, and the same role deleting resources from an unfamiliar address, outside its usual window, with actions it has never issued before, is not.

How to detect this in your own environment

Build first-seen queries. Asking which API actions each principal performed for the first time in a period surfaces intrusions that no signature would match, and it is a search rather than a rule to maintain.

Alert on destructive actions by identities with no history of them. Deletion and modification are legitimate for some roles and not for most, and the population that legitimately destroys resources is small and knowable.

Keep failures. Access denied events reveal intent and boundaries in a way successes cannot, and they are frequently dropped from ingestion to reduce volume, which removes the clearest evidence of someone exploring their limits.

How to prevent it

This is a detection and response capability rather than a preventive control, and it shortens dwell time rather than stopping an intrusion.

The preventive value is indirect and real. Investigating a simulated incident reliably reveals what you cannot see, and every gap found this way is a logging configuration to fix before it matters.

Keep the log store outside the blast radius of the environment it monitors. Logs an attacker can alter or delete are not evidence, so a separate account with independent access control is the baseline.

Real-world context

Availability incidents and security incidents overlap more than most runbooks assume. Destruction of resources presents first as an outage, and the team investigating is usually operations rather than security, which delays the point at which anyone asks whether it was deliberate.

That delay is costly because it extends dwell time. The organizations that detect quickly are generally the ones where the outage investigation routinely includes the question of who made the change, rather than treating that as a separate security process.

The techniques correspond to detection of activity catalogd under MITRE ATT&CK T1485 Data Destruction, T1078.004 Cloud Accounts and T1580 Cloud Infrastructure Discovery.

Applying this at work

• Add the question who made this change to your outage runbook, so every availability investigation touches the audit log.
• Write and save a first-seen query for API actions per principal, which is one of the highest-yield hunts available.
• Alert on destructive API calls by identities with no history of them.
• Confirm access denied events survive your ingestion pipeline.
• Verify your log store sits in a separate account with access controls independent of the monitored environment.

Frequently asked questions

How do I tell an outage from an attack in CloudTrail?

By looking at who made the call rather than what broke. Automation acts from consistent sources with a stable set of actions, so the signal is a familiar identity behaving unlike itself, or acting from somewhere it never has.

What is a first-seen query?

A search that lists API actions each principal performed for the first time within a period. It surfaces unusual behavior without needing a signature, which makes it effective against techniques nobody has written a rule for.

Why keep access denied events?

Because they show intent and boundaries. An identity probing the edges of its permissions produces a distinctive pattern of failures, and that is often the earliest evidence available. Many pipelines drop them to save cost.

Do I need my own AWS account to practice this?

No. The lab runs in a live environment that Pwned Labs provisions for you, so there is nothing to set up and no risk to your own infrastructure.

Which certification covers this in depth?

The Amazon Cloud Red Team Professional (ACRTP) covers AWS attack and detection paths in full, including IAM abuse and privilege escalation, and is assessed hands-on in a live AWS account.