Kuberneteson AWS (EKS)

Amazon EKS overview

Amazon EKS is AWS-managed Kubernetes. The control plane (API server, scheduler, etcd) is operated by AWS, patched automatically, and deployed across at least three availability zones. You own the data plane — or, on EKS Auto Mode (GA November 2024), you delegate the data plane to AWS as well and consume Kubernetes as an almost-serverless service.

FactualMinds deploys EKS for teams that need Kubernetes portability (multi-cloud, on-prem via EKS Hybrid Nodes, or open-source ecosystem alignment) and for mid-market AWS-only teams that have outgrown ECS or plain Fargate. We default new 2026 clusters to Auto Mode on Kubernetes 1.36 with Graviton-first node pools unless a specific workload says otherwise.

What’s new on EKS in 2026

• EKS Auto Mode GA — fully managed data plane, managed add-ons (VPC CNI, kube-proxy, CoreDNS, EBS CSI, AWS Load Balancer Controller), and a managed Karpenter that provisions nodes within seconds of scheduling pressure.
• EKS Hybrid Nodes (GA November 2024) — register Linux hosts running on-prem or at the edge as EKS worker nodes governed by an AWS-hosted control plane. One kubectl surface for cloud and hybrid.
• Karpenter 1.0 (2024) — stable NodeClass/NodePool CRDs, disruption budgets, and consolidation-policy modes. Karpenter is the default on Auto Mode.
• Pod Identity — the ergonomic replacement for IRSA. No OIDC provider, no ServiceAccount annotation, no trust-policy gymnastics.
• Kubernetes 1.36 — current EKS/EKS Distro minor as of July 2026; upstream releases every ~4 months, EKS supports the current plus the previous three. Upgrade rollbacks revert control-plane minor upgrades within 7 days.
• ECR enhanced scanning — Inspector v2 scans images for OS and language-package CVEs with exploit-probability-index scoring; integrates with Security Hub.
• AWS Load Balancer Controller — managed install on Auto Mode; supports Gateway API, ALB and NLB target-group binding, and cross-zone health checks.
• Amazon EBS CSI driver managed add-on — Auto Mode handles install and upgrade; gp3 volumes by default.
• Cilium + Hubble / eBPF observability — supported via add-ons for teams that need deep network visibility without full-fat service mesh.

Why EKS

Kubernetes standard

• Standard kubectl, Helm, Kustomize, and standard manifests.
• Portable: workloads run on other clouds, on-premises (EKS Hybrid Nodes or EKS Anywhere), or upstream Kubernetes.
• Massive ecosystem (Prometheus, OpenTelemetry, Argo CD, Flux, Karpenter, Cilium, Istio, Linkerd).

AWS integration

• VPC CNI for pod networking with real AWS IP addresses.
• Pod Identity for pod-level IAM permissions without OIDC acrobatics.
• Native integrations with ALB/NLB, EFS, EBS, S3, RDS, DynamoDB, SQS, Kinesis, Bedrock.
• AWS Security Hub / GuardDuty EKS Protection for runtime threat detection.

Managed control plane

• Multi-AZ control plane included in the $0.10/hour price.
• AWS patches the control plane on a published minor-version cadence.
• SLA covers control-plane availability; you are responsible for workload availability.

EKS Architecture

Control plane (AWS managed)

• API server, scheduler, controller managers, etcd.
• Audit logs can be shipped to CloudWatch Logs; control-plane endpoints can be private, public, or public+private with IP allow-list.

Data plane (your choice)

• EKS Auto Mode — fully managed nodes, add-ons, networking, load balancing, and storage controllers.
• Managed node groups — EC2 instances you provision; AWS manages OS patching, drain, and replacement.
• Karpenter on self-managed nodes — for teams that want fine control over instance-type selection and disruption policy.
• AWS Fargate — serverless pods with no node management; higher per-pod price, best for bursty or sandbox workloads.

Networking

• AWS VPC CNI: each pod gets a real VPC IP (prefix delegation supported for IP density).
• Security groups for pods (SGFP) for per-pod network security.
• Cilium eBPF or Calico for network policy and observability.
• AWS Load Balancer Controller for ALB/NLB ingress.

EKS Auto Mode in practice

• AWS provisions, scales, and replaces nodes automatically based on pending pods.
• Managed Karpenter bin-packs across instance types, including Graviton by default.
• OS patching via node replacement on a rolling schedule; no in-place kernel updates.
• AWS manages the core add-ons (VPC CNI, kube-proxy, CoreDNS, AWS Load Balancer Controller, EBS CSI).
• Billed as EC2 + a small EKS Auto Mode management fee per vCPU-hour; typically net-neutral or cheaper versus self-managed node groups when labor is priced in.

Use Auto Mode when

• You want Kubernetes without node operations.
• Your security team can live with AWS-managed, regularly replaced AMIs.
• Your workloads do not require custom kernel modules or niche runtime options.

Prefer managed node groups when

• You need a regulated/approved AMI (e.g., STIG-hardened) maintained by your security team.
• You run custom kernel modules (BPF/eBPF extensions beyond what’s supported, niche drivers).
• You want fine-grained Spot pool control that the managed NodePool does not expose.

EKS Hybrid Nodes

• Register on-prem Linux hosts as EKS workers against an AWS-hosted control plane.
• Supports x86 and ARM; requires AWS Systems Manager connectivity from the on-prem host.
• Use for edge compute, data-gravity on-prem workloads, or manufacturing floor nodes that must stay physically on site but should be governed from AWS.
• Compare to EKS Anywhere: Hybrid Nodes share one control plane with AWS; EKS Anywhere runs its own on-site control plane.

Pod Identity vs IRSA

Pod Identity (2026 default)

aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace production \
  --service-account my-app \
  --role-arn arn:aws:iam::123456789:role/my-app-role

Pods using the my-app ServiceAccount in the production namespace automatically receive temporary credentials via the Pod Identity Agent. No annotation, no OIDC provider, no trust-policy StringEquals dance.

IRSA (legacy / niche)

• Still required for workloads that only accept token-file authentication, clusters older than 1.24, or EC2 workloads outside EKS.
• OIDC provider + annotated ServiceAccount + trust-policy condition on the OIDC subject.

Karpenter 1.0 patterns we deploy

• Graviton-first NodePool — allow arm64 architectures, prefer on-demand for baseline and Spot for scale-out.
• Consolidation policy — WhenUnderutilized for dev/staging, WhenEmpty for production to avoid disruption of long-running pods.
• Disruption budgets — cap how many nodes Karpenter can consolidate per hour, aligned with PDB.
• Per-namespace NodePool selection — heavy GPU workloads go to a dedicated NodePool with nvidia.com/gpu taints.

Observability stack

• CloudWatch Container Insights (enhanced observability) for the cluster, nodes, pods, and control plane.
• ADOT Collector DaemonSet forwarding traces, metrics, and logs to Managed Prometheus + Managed Grafana, or to Datadog / New Relic / Honeycomb.
• Cilium Hubble (or Pixie) for eBPF-level network visibility without a service mesh.
• EKS audit logs to CloudWatch Logs with 90-day retention and S3 archive behind Object Lock for SOC 2 / PCI evidence.

Graviton cost savings

• Graviton3 (m7g, c7g, r7g) and Graviton4 (m8g, c8g, r8g) typically deliver 30–40% better price-performance than comparable x86 for stateless microservices and JVM workloads.
• Build multi-arch images with docker buildx build --platform linux/amd64,linux/arm64 in CI; push both manifests to ECR.
• Karpenter on Auto Mode will pick ARM when it wins on price and pod fits.

Reference architecture (2026 default)

                    ┌──────────────────────────────────────────────┐
                    │  AWS-managed control plane (multi-AZ)        │
                    │  api / scheduler / controller-mgr / etcd     │
                    │  audit + authenticator + scheduler logs      │
                    └─────────────────┬────────────────────────────┘
                                      │ (private endpoint via PrivateLink)
                                      │
   ┌──────────────────────────────────┼──────────────────────────────────┐
   │ Data plane (Auto Mode)           │                                  │
   │  ├── managed Karpenter NodePool  │ ── Pod Identity Agent (per node) │
   │  ├── Graviton-first c8g/m8g/r8g  │ ── VPC CNI (prefix delegation)   │
   │  ├── consolidation policy        │ ── EBS CSI (gp3 default)         │
   │  └── disruption budgets          │ ── AWS LB Controller (ALB+NLB)   │
   └──────────────────────────────────┴──────────────────────────────────┘
                                      │
   Workloads ── ServiceAccount → PodIdentityAssociation → IAM Role
   Ingress  ── ALB (alb.ingress.k8s.aws/scheme: internet-facing)
   Storage  ── EBS gp3 PVCs / EFS for shared / S3 for objects
   Secrets  ── Secrets Store CSI / HashiCorp VSO → Vault / Secrets Manager
   Images   ── ECR (enhanced scanning, image signing) ← CI attestation
   Telemetry ─ CloudWatch Container Insights + ADOT → Datadog / AMP+AMG
   Audit    ── CloudWatch Logs (90d) + S3 Object Lock (compliance archive)

Failure modes & resilience

1. Karpenter consolidation evicting under-budgeted pods. Default consolidationPolicy: WhenUnderutilized will move pods aggressively. For long-running stateful workloads, set WhenEmpty on the NodePool and define a PodDisruptionBudget (minAvailable) so consolidation cannot violate availability. Disruption budgets at the NodePool level cap voluntary disruptions per hour.

2. Pod Identity Agent crash-loop. Symptom: pods using the ServiceAccount get 403 AccessDenied from STS. Causes: agent DaemonSet pod CrashLoopBackOff (check kubectl logs -n kube-system -l app=eks-pod-identity-agent), Pod Identity Association pointing at a non-existent IAM role, trust policy missing pods.eks.amazonaws.com principal, or IMDS hop limit too low on the node. Auto Mode handles the agent; on managed node groups confirm the agent add-on is healthy.

3. NodePool pinned to a single AZ. A zonal disruption (control-plane outage in one AZ, ELB endpoint flap) takes the workload with it. Always include topology.kubernetes.io/zone In [a, b, c] in NodePool requirements; combine with topologySpreadConstraints on Deployments.

4. gp3 volume detach during node replacement. Auto Mode replaces nodes — StatefulSets with volumeClaimTemplates should explicitly set persistentVolumeReclaimPolicy: Retain and a storageClass with volumeBindingMode: WaitForFirstConsumer. Otherwise an in-flight reschedule can race with detach and the pod stays ContainerCreating for several minutes.

5. --max-unavailable vs PDB collisions. A Deployment’s RollingUpdate strategy plus a strict PDB (minAvailable: 100%) deadlocks the rollout. Always set PDB minAvailable such that replicas - minAvailable >= maxUnavailable.

6. Cluster Autoscaler vs Karpenter coexistence. Running both in the same cluster causes thrash. Pick one. Karpenter for new clusters; Cluster Autoscaler only if a vendor product hard-requires it.

7. EKS minor-version upgrade window. AWS supports current + 3 prior minors (~14 months). Letting a cluster slip to N-4 forces emergency upgrade across multiple breaking changes. Schedule quarterly minor upgrades; test in a staging cluster first.

Observability runbook

Enable control-plane logs at cluster creation:

aws eks update-cluster-config \
  --region eu-west-1 \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

Alarms we ship:

Debug path: “Pod stuck Pending”:

1. kubectl describe pod <name> → Events. Most common: 0/N nodes are available: insufficient memory or node(s) didn't match Pod's node affinity.
2. If insufficient resources: confirm Karpenter is provisioning (kubectl get nodeclaims); check NodePool requirements allow the pod’s architecture and instance family.
3. If affinity mismatch: check NodePool labels match pod’s nodeSelector / affinity.
4. If FailedScheduling on Pod Identity SA: confirm PodIdentityAssociation exists for (cluster, namespace, serviceAccount).

Debug path: “Node not ready”:

1. kubectl describe node <node> → Conditions section. MemoryPressure, DiskPressure, PIDPressure are first signals.
2. CloudWatch Container Insights → node detail → kubelet logs.
3. VPC CNI: kubectl logs -n kube-system -l k8s-app=aws-node for IP exhaustion or ENI attach failures.
4. If on Auto Mode, the node will be replaced automatically — confirm replacement is in progress before manual intervention.

When EKS is NOT the right call

• Small, simple container workload with 1–3 services and a team unfamiliar with Kubernetes — Amazon ECS on Fargate has a fraction of the operational surface and is often the better first step.
• Entirely event-driven or short-lived workload — AWS Lambda or ECS Fargate spot often costs less and simplifies ops.
• You have no plans to leverage Kubernetes portability or ecosystem — the $73/month per-cluster plus learning-curve tax is real.
• You need air-gapped operation with no AWS dependency — evaluate EKS Anywhere or upstream Kubernetes on bare metal.

EKS best practices

Resource management

• Always set requests and limits. Use Vertical Pod Autoscaler recommendations to size requests.
• Use pod disruption budgets; align with Karpenter disruption budgets to avoid correlated voluntary disruptions.

Auto-scaling

• Karpenter (or Auto Mode’s managed Karpenter) preferred over Cluster Autoscaler for new clusters.
• HPA on CPU/memory/custom metrics (Prometheus) for pod-level scaling.
• KEDA for event-driven autoscaling (SQS, Kinesis, Kafka lag).

Security

• Pod Identity for pod-level IAM.
• Network policies via Cilium/Calico; restrict egress by default.
• Kubernetes secrets encrypted with a customer-managed KMS key.
• Pair with HashiCorp Vault Secrets Operator or AWS Secrets Manager + Secrets Store CSI driver for application secrets.
• ECR enhanced scanning + image signing verified at admission.

Reliability

• Multi-AZ NodePools; never pin a production NodePool to a single AZ.
• Backups of cluster state (Velero) for stateful apps or CRD-heavy control-plane configuration.
• Routine disaster-recovery tests of cluster re-creation from IaC.

Related reading

• ECS vs EKS: container orchestration decision guide
• Karpenter vs Cluster Autoscaler on EKS: cost optimization
• How to deploy EKS with Karpenter for cost-optimized autoscaling

Related services

• AWS Application Modernization
• DevOps Pipeline Setup
• Hire a Dedicated AWS Expert