AWS Platform Operations
PCI-Compliant Infrastructure SRE at American Express
$200K+
Monthly cloud spend reduced
99.9999%
Uptime across 6 PCI accounts
50%
MTTD reduction
35%
MTTR reduction
situation
Shared AWS infrastructure at American Express spanned 6 PCI-supporting production accounts with no systematic cost controls. Cloud spend was inflated across three categories: oversized instances, idle resources (unattached volumes, zombie EKS nodes), and autoscaling gaps where services ran at full capacity off-hours.
challenge
Maintain 99.9999% uptime under 24×7 on-call in a regulated PCI environment while simultaneously identifying and eliminating over $200K/month in cloud waste - without disrupting live workloads.
solution
Built a Python rightsizing engine targeting all three waste categories: automated instance rightsizing from Datadog utilization metrics, idle-resource cleanup sweeps, and off-hours autoscaling schedules. Structured SLO tracking in Datadog held <0.01% error rates across all services. Ran quarterly DR exercises failing over Aurora/RDS and EKS within 8-hour RTO/RPO targets, including executing a live region-level failover during a data center incident.
KEY CONTRIBUTIONS
- Operated shared AWS services across 6 PCI-supporting production accounts: EKS, RDS/Aurora (PostgreSQL), IAM, S3, Lambda, Transit Gateway
- Python rightsizing engine cut $200K+/month by targeting oversized instances, idle resources, and autoscaling gaps simultaneously
- Led 24×7 on-call using Datadog, Dynatrace, and Splunk for incident detection and triage; held <0.01% error-rate SLO across all services
- Cut MTTD 50% and MTTR 35% through structured runbooks, SLO alerting, and correlated observability across all three platforms
- Planned and ran quarterly DR exercises, failing over Aurora/RDS and EKS within 8-hour RTO/RPO targets
EQUIPMENT MANIFEST