By Grant Sullivan (Lead Writer) | Vulnerability Research: Dr. Natalie Brooks | Threat Intelligence: Anton Novak | Network Defense: Adrian Ward | DFIR: Trevor Donovan | CISO Review: Katherine Hughes
EXECUTIVE THREAT SUMMARY & 30-SECOND OPERATOR TRIAGE
– Root-Cause Vector: Overly permissive IAM role trust relationships paired with unconstrained Instance Metadata Service access and unsegmented container network namespaces in SageMaker processing and training jobs.
– Exploitation Status: Active exploitation of wildcard IAM trust policies and Server-Side Request Forgery (SSRF) vulnerabilities in custom SageMaker container endpoints (CVSS v3.1: 8.8 High / Scope Changed).
– Am I Vulnerable? 1. Do your SageMaker AI training or processing jobs execute with IMDSv1 enabled or without explicitaws:SourceArnandaws:SourceAccountcondition keys? 2. Are SageMaker notebook instances or HyperPod worker nodes attached to subnets with direct internet gateways without VPC endpoint enforcement?
– Blast Radius: Cross-account IAM role assumption, temporary credential exfiltration from metadata services, unauthorized access to S3 training buckets, intellectual property theft of proprietary model weights, and lateral movement across enterprise AWS Organizations.
– Remediation SLA: 24 hours for IAM trust policy condition enforcement and mandatory IMDSv2 hop-limit configuration; 72 hours for VPC interface endpoint microsegmentation across all active AI execution enclaves.
Executive Architecture & Threat Overview
As cloud providers scale physical capacity to support enterprise demand, organizations are rapidly deploying specialized AI infrastructure across their Amazon Web Services (AWS) footprints. Recent operational guidance highlights the widespread adoption of multi-turn reinforcement learning and massive model training pipelines hosted on Amazon SageMaker AI. However, this aggressive expansion of high-density compute environments has exposed a severe architectural weakness: flawed identity delegation and loose network boundaries within automated model training pipelines.
When enterprises run complex machine learning jobs, containerized workloads require elevated permissions to read raw datasets from Amazon S3, log metrics to CloudWatch, and write serialized model artifacts back to secure repositories. Threat actors are actively targeting these processing environments. By exploiting SSRF vulnerabilities or code injection vectors inside custom training containers, adversaries retrieve temporary Security Token Service (STS) credentials from the underlying host and leverage misconfigured IAM role trust policies to pivot into broader cloud environments.
+-----------------------------------------------------------------------------------+
| ATTACKER CONTROLLED INTERNET |
+-----------------------------------------------------------------------------------+
|
| [1] Insecure Endpoint / SSRF Exploit
v
+-----------------------------------------------------------------------------------+
| AWS SAGEMAKER VPC WORKER CONTAINER |
| - Managed Container Execution Environment (/opt/ml) |
| - Unconstrained IMDS Access (169.254.169.254) |
+-----------------------------------------------------------------------------------+
|
| [2] Extract Temporary STS Credentials
v
+-----------------------------------------------------------------------------------+
| AWS INSTANCE METADATA SERVICE (IMDS) |
| - AWS Security Credentials / Role: SageMakerExecutionRole |
+-----------------------------------------------------------------------------------+
|
| [3] Assume Role via STS (Missing aws:SourceArn)
v
+-----------------------------------------------------------------------------------+
| TARGET ENTERPRISE AWS CROWN JEWEL ACCOUNT |
| - S3 Data Lakes, Production Databases, Active Directory / IAM Admin |
+-----------------------------------------------------------------------------------+
Securing these workloads requires moving beyond standard enterprise perimeter defenses. Security teams must enforce strict cryptographically bound identity delegation, mandatory IMDSv2 metadata limits, and zero-trust VPC endpoint policies across all SageMaker execution enclaves.
Root Cause Anatomy & Policy Diff Analysis
The primary vulnerability vector in cloud AI environments is the overuse of overly broad AssumeRole trust relationships combined with missing request condition keys. When SageMaker provisions compute instances on behalf of a tenant, the AWS service assumes an IAM execution role defined by the administrator.
If the trust policy of that role utilizes a broad Principal: { Service: "sagemaker.amazonaws.com" } block without enforcing aws:SourceArn or aws:SourceAccount condition locks, any authorized principal within the AWS region capable of initiating a SageMaker job can pass that execution role, resulting in cross-account privilege escalation and confused deputy exploitation.
Furthermore, if containers within the SageMaker pipeline permit unauthenticated outbound HTTP requests, an attacker who injects code into a training script can query http://169.254.169.254/latest/meta-data/iam/security-credentials/ to harvest the temporary access keys assigned to the compute node.
Vulnerable vs. Remediated IAM Trust Policy Architecture
The following code diff demonstrates an insecure SageMaker trust policy versus a hardened production policy that enforces condition keys and cryptographically isolates the execution role to specific training pipelines:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "sagemaker.amazonaws.com"
},
"Action": "sts:AssumeRole",
- "Condition": {}
+ "Condition": {
+ "StringEquals": {
+ "aws:SourceAccount": "123456789012"
+ },
+ "ArnLike": {
+ "aws:SourceArn": "arn:aws:sagemaker:us-east-1:123456789012:processing-job/prd-*"
+ }
+ }
}
]
}
Sysadmin Verification CLI Commands
Security operators must continuously audit their AWS environments to identify exposed SageMaker instances, unconstrained IAM roles, and legacy IMDS configurations.
Linux / Bash Terminal Verification
Run the following AWS CLI v2 script to identify all SageMaker notebook instances and training jobs operating without network isolation or using insecure defaults:
#!/usr/bin/env bash
set -euo pipefail
echo "[+] Checking SageMaker Notebook Instances for Direct Internet Access..."
aws sagemaker list-notebook-instances --query 'NotebookInstances[*].NotebookInstanceName' --output text | \
while read -r name; do
if [ -n "$name" ]; then
DIRECT_ACCESS=$(aws sagemaker describe-notebook-instance --notebook-instance-name "$name" --query 'DirectInternetAccess' --output text)
SUBNET=$(aws sagemaker describe-notebook-instance --notebook-instance-name "$name" --query 'SubnetId' --output text)
echo "Notebook: $name | DirectInternetAccess: $DIRECT_ACCESS | Subnet: $SUBNET"
fi
done
echo "[+] Auditing IAM Roles assumed by SageMaker for missing aws:SourceArn..."
aws iam list-roles --query 'Roles[?AssumeRolePolicyDocument.Statement[?Principal.Service==`sagemaker.amazonaws.com`]].RoleName' --output text | \
while read -r role; do
if [ -n "$role" ]; then
HAS_CONDITION=$(aws iam get-role --role-name "$role" --query 'Role.AssumeRolePolicyDocument.Statement[?Principal.Service==`sagemaker.amazonaws.com`].Condition' --output text)
if [ "$HAS_CONDITION" == "None" ] || [ -z "$HAS_CONDITION" ]; then
echo "[CRITICAL WARNING] Role '$role' allows sagemaker.amazonaws.com without condition keys!"
fi
fi
done
Windows / PowerShell Command Verification
Run the following PowerShell command to inspect local AWS credential profiles and search for active SageMaker compute resources missing explicit security group isolation:
# Audit SageMaker Processing Jobs for Network Isolation Status
Import-Module AWS.Tools.SageMaker
Write-Host "[+] Querying active SageMaker Processing Jobs..." -ForegroundColor Cyan
$Jobs = Get-SMPalProcessingJobList -MaxResults 50
foreach ($Job in $Jobs) {
$Detail = Get-SMPalProcessingJob -ProcessingJobName $Job.ProcessingJobName
$NetworkIso = $Detail.NetworkConfig.EnableNetworkIsolation
$VpcConfig = $Detail.NetworkConfig.VpcConfig.Subnets
if ($NetworkIso -ne $true) {
Write-Host "[!] ALARM: Job '$($Job.ProcessingJobName)' has EnableNetworkIsolation = FALSE" -ForegroundColor Red
} else {
Write-Host "[OK] Job '$($Job.ProcessingJobName)' is isolated." -ForegroundColor Green
}
}
Operational Workaround & Mitigation Warnings
Enforcing rigid security controls on live machine learning environments can disrupt active model pipelines if operational dependencies are not accounted for prior to policy enforcement.
OPERATIONAL WARNING: PERFORMANCE & FUNCTIONAL IMPACT
– Network Isolation Constraints: EnablingEnableNetworkIsolation = Trueon SageMaker processing or training jobs completely blocks all external outbound network calls from the execution container. If custom scripts rely on dynamic pip installs (e.g.,pip install transformersduring runtime) or download model weights directly from external repositories like HuggingFace, the job will fail immediately with connection timeouts.
– S3 Gateway Endpoint Requirements: When disabling direct internet access, security teams must provision AWS VPC Endpoints (Interface and Gateway endpoints) for S3, ECR, CloudWatch Logs, and SageMaker API/Runtime inside the target VPC. Failure to map routing tables correctly will prevent containers from pulling base Docker images from Amazon ECR.
– IMDSv2 Enforcement Overhead: MandatingHttpTokens=requiredwith anHttpPutResponseHopLimit=1blocks containers running in bridge or overlay networking modes from reaching instance metadata. Ensure containerized applications use AWS SDKs updated to support IMDSv2 token acquisition headers.
Attack Chain & MITRE ATT&CK Progression
The following breakdown illustrates how an adversary exploits loose IAM delegation and unsegmented SageMaker infrastructure to achieve multi-account operational persistence:
- Initial Access (T1190 – Exploit Public-Facing Application): The adversary identifies a vulnerable endpoint or supply chain dependency within a custom SageMaker inference container or web-facing Jupyter notebook instance.
- Execution (T1059.004 – Unix Shell): By tricking the application into executing arbitrary python code, the attacker opens an interactive reverse shell within the container filesystem (
/opt/ml/processing). - Credential Access (T1552.005 – Cloud Instance Metadata API): The attacker issues an HTTP request to
http://169.254.169.254/latest/meta-data/iam/security-credentials/to obtain temporary STS credentials assigned to the instance execution role. - Privilege Escalation (T1098 – Account Manipulation): Using the extracted STS credentials, the attacker invokes
sts:AssumeRoleagainst misconfigured secondary enterprise IAM roles that lackaws:SourceArnboundaries. - Lateral Movement & Exfiltration (T1537 – Transfer Data to Cloud Account): The adversary uses newly acquired cross-account privileges to dump proprietary training data from central S3 data lakes and exfiltrate machine learning model weights to an off-grid AWS account controlled by the threat actor.
Affected Infrastructure & Hardening Matrix
| Infrastructure Component | Vulnerable State / Feature | Hardened Architecture Requirement | Operational Mitigation Status |
|---|---|---|---|
| SageMaker Notebook Instances | DirectInternetAccess: Enabled without VPC endpoints |
Disable direct access; mandate customer VPC attachment with PrivateLink endpoints | Remediated via AWS Service Control Policy (SCP) |
| IAM Execution Roles | sts:AssumeRole with broad Service: sagemaker.amazonaws.com |
Enforce aws:SourceArn and aws:SourceAccount condition blocks in trust policy |
Emergency Patch Required ($\le 24 \text{h}$) |
| Container Networking | EnableNetworkIsolation: False on untrusted scripts |
Enforce EnableNetworkIsolation: True and mirror dependencies to private S3 |
Workaround Active (Pipeline update) |
| Instance Metadata Service | IMDSv1 Enabled (HttpTokens: Optional) |
Force IMDSv2 (HttpTokens: Required) with HttpPutResponseHopLimit: 1 |
Automated Remediation via Launch Template |
| SageMaker HyperPod Enclaves | Unsegmented Slurm cluster inter-node traffic | Enforce Transit Gateway microsegmentation and AWS Network Firewall inspection | Scheduled Architecture Upgrade |
Actionable Detection Engineering
Security Operations Centers (SOC) must deploy dedicated detection signatures to identify unauthorized STS token harvesting and cross-account privilege escalation originating from cloud AI enclaves.
Production-Grade Sigma Rule
title: SageMaker Cross-Account Role Assumption Without Source Context
id: a8f4e21b-93c1-4d3e-8212-b1122a009bc2
status: experimental
description: Detects sts:AssumeRole calls initiated by sagemaker.amazonaws.com lacking proper aws:SourceArn validation in CloudTrail logs.
author: Anton Novak
date: 2026/10/25
logsource:
product: aws
service: cloudtrail
detection:
selection:
eventName: 'AssumeRole'
userAgent|contains: 'sagemaker.amazonaws.com'
filter_valid_arn:
requestParameters.roleArn|contains:
- 'arn:aws:iam::123456789012:role/SageMaker'
condition: selection and not filter_valid_arn
falsepositives:
- Legitimate new SageMaker processing pipelines created without updated IAM condition templates.
level: high
tags:
- attack.initial_access
- attack.t1190
- attack.privilege_escalation
Production-Grade YARA Hunting Rule
rule Exploit_SageMaker_IMDS_Credential_Stealer {
meta:
description = "Detects quantitative models or container binaries attempting to query AWS IMDS endpoints from SageMaker workspace directories"
author = "Dr. Natalie Brooks"
date = "2026-10-25"
severity = "CRITICAL"
mitre_attck = "T1552.005"
strings:
$imds_ip = "169.254.169.254" ascii wide
$sagemaker_path = "/opt/ml/" ascii wide
$sts_token = "meta-data/iam/security-credentials/" ascii wide
$python_req = "requests.get" ascii wide
$curl_cmd = "curl" ascii wide
condition:
uint32(0) == 0x756c6923 /* #!/u */ or uint32(0) == 0x2123 /* !# */ or uint32(0) == 0x464c457f /* ELF */ or filesize < 5MB and
($imds_ip and $sts_token) and
($sagemaker_path or $python_req or $curl_cmd)
}
Indicators of Compromise (IOC) Artifact Table
The following defanged indicators represent malicious execution patterns observed during simulated cross-account compromise of cloud AI pipelines:
| Indicator Type | Defanged Value / Artifact | Observed Adversary Activity |
|---|---|---|
| Malicious User-Agent | SageMaker-Exfil-Agent/v1.0 (Linux x86_64) |
Unauthorized API calls pulling S3 datasets outside defined VPC |
| Network C2 Endpoint | 198.51.100[.]42 |
Exfiltration destination for serialized model weights |
| Domain Name | c2-ai-exfil[.]internal-telemetry[.]net |
External command and control host invoked via container shell |
| File SHA-256 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 |
Weaponized Jupyter notebook payload dropping reverse shell |
| IAM Policy Name | SageMakerUnconstrainedWildcardTrust |
Overly permissive IAM role trust policy template |
Jira / ServiceNow Ready Remediation Action Checklist
- [ ] **Immediate Containment (0-24 Hours):**
- [ ] Audit all active IAM role trust policies containing `sagemaker.amazonaws.com` principals.
- [ ] Append mandatory `aws:SourceAccount` and `aws:SourceArn` condition blocks to all SageMaker execution roles.
- [ ] Revoke active STS sessions generated by legacy SageMaker execution roles via AWS IAM CLI (`PutRolePolicy` with `AWSRevokeOlderSessions`).
- [ ] **Short-Term Remediation (24-72 Hours):**
- [ ] Enforce IMDSv2 (`HttpTokens: Required`) across all SageMaker notebook instances and underlying EC2 host clusters.
- [ ] Deploy Service Control Policies (SCPs) blocking the creation of SageMaker processing jobs that have `EnableNetworkIsolation` set to `False` in untrusted staging environments.
- [ ] Configure VPC Gateway Endpoints for Amazon S3 and Interface Endpoints for ECR, CloudWatch, and STS inside target subnets.
- [ ] **Strategic Hardening & Hunting (7+ Days):**
- [ ] Ingest AWS CloudTrail `AssumeRole` telemetry into SIEM and activate the provided Sigma detection rule.
- [ ] Execute automated YARA scans across container storage layers (`/opt/ml/`) within ECR repository registries.
- [ ] Establish centralized model registry pipelines using AWS RAM (Resource Access Manager) to prevent direct worker cross-account access.
Operator Q&A & Technical FAQ
Q1: How does enforcing aws:SourceArn in an IAM trust policy prevent confused deputy attacks in Amazon SageMaker?
A: When a service like SageMaker assumes a role on your behalf, AWS evaluates the trust policy of that target role. Without explicit condition keys, any caller in any AWS account can invoke a SageMaker job targeting your role ARN. By adding the aws:SourceArn condition set strictly to your specific SageMaker job or notebook ARN (e.g., arn:aws:sagemaker:us-east-1:123456789012:processing-job/*), AWS verifies that the assumption request originates exclusively from your defined resources within your designated account.
Q2: Why does HttpPutResponseHopLimit: 1 break certain SageMaker Docker containers, and how should it be configured?
A: When Docker containers run using standard bridge networking, packets destined for the host instance metadata service (169.254.169.254) traverse an internal network bridge. This extra network layer decrements the IP packet Time To Live (TTL) by 1. If the host metadata service enforces an HTTP PUT response hop limit of 1, the response packet is discarded before returning to the container. To secure the host while preserving container connectivity, set the hop limit to 2 if bridge networking is required, or execute containers in host networking mode (--net=host).
Q3: Can an attacker exfiltrate data from a SageMaker processing job if EnableNetworkIsolation is set to True?
A: No. When EnableNetworkIsolation is set to True, the AWS SageMaker container runtime disables all outbound network interfaces except for local loopback interfaces. The container cannot communicate with external internet endpoints, public S3 buckets, or external C2 servers. All input data must be mounted into /opt/ml/processing/input by the SageMaker control plane, and output artifacts must be written to /opt/ml/processing/output, which AWS securely syncs back to pre-configured S3 buckets.
Sources & References
- AWS Security Documentation: IAM Role Trust Policies and Condition Keys for SageMaker Workloads
- CISA Cloud Security Technical Reference Architecture: Securing Cloud Compute Enclaves and Machine Learning Workloads
- MITRE ATT&CK Cloud Matrix: T1552.005: Unsecured Credentials – Cloud Instance Metadata API
- NIST Special Publication 800-207: Zero Trust Architecture for Enterprise Cloud Infrastructures
