Kubernetes Interview Questions & Answers part 05
70+ Kubernetes interview questions and answers from basic to advanced — covering Pods, Deployments, Services, Networking, RBAC, Helm, Autoscaling, Security, and real-world troubleshooting scenarios.
Kubernetes & EKS Interview Questions & Answers for AWS Cloud and AWS DevOps Engineer
A comprehensive guide covering Basic, Intermediate, and Advanced topics for Kubernetes and Amazon EKS interviews.
Answer:
Kubernetes (K8s) is an open-source container orchestration platform originally developed by Google and now maintained by the CNCF (Cloud Native Computing Foundation). It automates the deployment, scaling, and management of containerized applications.
Key capabilities:
- Automated rollouts and rollbacks — deploy changes and roll them back if something goes wrong
- Service discovery and load balancing — expose containers using DNS names or IP addresses
- Storage orchestration — automatically mount storage systems (local, cloud, NFS, etc.)
- Self-healing — restarts failed containers, replaces containers, kills containers that don’t respond to health checks
- Secret and config management — manage sensitive information without rebuilding images
- Horizontal scaling — scale applications up or down with a command or automatically
# Check Kubernetes version
kubectl version --short
# Get cluster info
kubectl cluster-info
Answer:
Kubernetes follows a master-worker architecture:
Control Plane (Master) Components:
| Component | Role |
|---|---|
API Server (kube-apiserver) | Frontend for Kubernetes control plane; all communication goes through it |
| etcd | Consistent and highly-available key-value store for all cluster data |
Scheduler (kube-scheduler) | Watches for newly created Pods and assigns them to nodes |
Controller Manager (kube-controller-manager) | Runs controller processes (Node, Replication, Endpoint controllers, etc.) |
| Cloud Controller Manager | Links cluster to cloud provider APIs |
Worker Node Components:
| Component | Role |
|---|---|
| kubelet | Agent on each node that ensures containers are running in a Pod |
| kube-proxy | Maintains network rules on nodes for Pod communication |
| Container Runtime | Software to run containers (containerd, CRI-O, Docker) |
┌──────────────────────────────────────────────┐
│ Control Plane │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │API Server│ │Scheduler │ │ Controller │ │
│ └──────────┘ └──────────┘ │ Manager │ │
│ │ └──────────────┘ │
│ ┌─────▼──────────────────────────────────┐ │
│ │ etcd │ │
│ └────────────────────────────────────────┘ │
└──────────────────────────────────────────────┘
│
┌──────────▼───────────┐
│ Worker Node │
│ ┌────────────────┐ │
│ │ kubelet │ │
│ ├────────────────┤ │
│ │ kube-proxy │ │
│ ├────────────────┤ │
│ │Container Runtime│ │
│ └────────────────┘ │
└──────────────────────┘
Answer:
A Pod is the smallest and most basic deployable unit in Kubernetes. It represents a single instance of a running process and can contain one or more tightly coupled containers that share the same network namespace, IP address, and storage.
Key characteristics:
- Each Pod gets a unique IP address within the cluster
- Containers inside a Pod share
localhostnetworking - Pods are ephemeral — they are not self-healing by themselves
- They are typically managed by higher-level controllers (Deployments, StatefulSets)
# Example Pod manifest
apiVersion: v1
kind: Pod
metadata:
name: my-app-pod
labels:
app: my-app
spec:
containers:
- name: app-container
image: nginx:1.21
ports:
- containerPort: 80
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "200m"
memory: "256Mi"
# Get all pods in all namespaces
kubectl get pods -A
# Describe a pod
kubectl describe pod my-app-pod
# Get pod logs
kubectl logs my-app-pod
Answer:
A Node is a physical or virtual machine that runs workloads (Pods) in a Kubernetes cluster. Every node is managed by the control plane and contains the services needed to run Pods.
Node components:
kubelet— communicates with the API server and manages Pod lifecyclekube-proxy— handles networking rules for Service routing- Container runtime — runs the containers (e.g., containerd)
Node types:
- Master Node — runs control plane components (in older setups)
- Worker Node — runs application workloads
# List all nodes
kubectl get nodes
# Get node details
kubectl describe node <node-name>
# Check node resource usage
kubectl top nodes
Answer:
Namespaces provide a mechanism to isolate groups of resources within a single cluster. They are ideal for multi-team or multi-project environments where resource quotas and access control need to be separated.
Default namespaces:
default— for objects with no other namespacekube-system— for system objects created by Kuberneteskube-public— readable by all users; used for public cluster infokube-node-lease— holds Lease objects for node heartbeats
# List namespaces
kubectl get namespaces
# Create a namespace
kubectl create namespace my-team
# Get pods in a specific namespace
kubectl get pods -n my-team
# Set default namespace for kubectl
kubectl config set-context --current --namespace=my-team
# ResourceQuota for a namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: team-quota
namespace: my-team
spec:
hard:
pods: "20"
requests.cpu: "4"
requests.memory: 8Gi
limits.cpu: "8"
limits.memory: 16Gi
Answer:
A Deployment is a higher-level abstraction that manages a ReplicaSet and provides declarative updates for Pods. It ensures the desired number of Pod replicas are running and handles rollouts and rollbacks.
Features:
- Declarative updates — you describe the desired state
- Rolling updates — updates Pods gradually to avoid downtime
- Rollback — easily revert to previous versions
- Pause and resume deployments
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: my-app:v2
ports:
- containerPort: 8080
# Create deployment
kubectl apply -f deployment.yaml
# Check rollout status
kubectl rollout status deployment/my-app
# Rollback to previous version
kubectl rollout undo deployment/my-app
# View rollout history
kubectl rollout history deployment/my-app
Answer:
A ReplicaSet ensures that a specified number of Pod replicas are running at any given time. It replaces Pods that fail, are deleted, or are terminated. Deployments manage ReplicaSets, so in practice you rarely create a ReplicaSet directly.
How it works:
- ReplicaSet defines a label selector to identify which Pods it manages
- It continuously monitors Pod counts against the desired count
- It creates or deletes Pods to match the desired state
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: my-replicaset
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: nginx
Answer:
A Service is an abstraction that defines a logical set of Pods and a policy to access them. Since Pods are ephemeral and their IPs change, a Service provides a stable IP address and DNS name to access them.
How it works:
- Services use label selectors to find matching Pods
kube-proxymaintains network rules to route traffic- Each Service gets a ClusterIP and a DNS entry (e.g.,
my-service.default.svc.cluster.local)
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: my-app
ports:
- protocol: TCP
port: 80
targetPort: 8080
type: ClusterIP
Answer:
| Type | Description | Use Case |
|---|---|---|
| ClusterIP | Default; exposes Service on internal cluster IP | Internal microservice communication |
| NodePort | Exposes Service on each Node’s IP at a static port (30000-32767) | External access in development |
| LoadBalancer | Exposes Service externally using a cloud load balancer | Production external access |
| ExternalName | Maps Service to a DNS name (e.g., external DB) | Connecting to external services |
# LoadBalancer Service Example (EKS)
apiVersion: v1
kind: Service
metadata:
name: my-lb-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
selector:
app: my-app
ports:
- port: 80
targetPort: 8080
type: LoadBalancer
Answer:
A ConfigMap stores non-confidential configuration data as key-value pairs, decoupling configuration from container images. This allows you to change application behavior without rebuilding images.
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
APP_ENV: "production"
APP_PORT: "8080"
config.json: |
{
"logLevel": "info",
"retries": 3
}
Using ConfigMap in a Pod:
spec:
containers:
- name: app
image: my-app
envFrom:
- configMapRef:
name: app-config
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: app-config
Answer:
A Secret stores sensitive data such as passwords, tokens, and SSH keys. Data is stored base64-encoded (not encrypted by default, but can be encrypted at rest with KMS).
# Create a secret from literal values
kubectl create secret generic db-secret \
--from-literal=username=admin \
--from-literal=password=s3cr3t
apiVersion: v1
kind: Secret
metadata:
name: db-secret
type: Opaque
data:
username: YWRtaW4= # base64("admin")
password: czNjcjN0 # base64("s3cr3t")
# Using secrets as environment variables
spec:
containers:
- name: app
image: my-app
env:
- name: DB_USERNAME
valueFrom:
secretKeyRef:
name: db-secret
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-secret
key: password
Answer:
kubectl is the command-line tool for interacting with Kubernetes clusters. It communicates with the Kubernetes API server to create, update, delete, and inspect resources.
Common commands:
# Cluster info
kubectl cluster-info
kubectl get nodes
# Pod management
kubectl get pods -n <namespace>
kubectl describe pod <pod-name>
kubectl logs <pod-name> -f
kubectl exec -it <pod-name> -- /bin/bash
# Apply/delete manifests
kubectl apply -f manifest.yaml
kubectl delete -f manifest.yaml
# Scale deployment
kubectl scale deployment my-app --replicas=5
# Port forwarding
kubectl port-forward pod/my-pod 8080:80
# Resource usage
kubectl top pods
kubectl top nodes
Answer:
A DaemonSet ensures that a copy of a Pod runs on all (or some selected) nodes. When a new node joins the cluster, the DaemonSet controller automatically adds a Pod to it.
Common use cases:
- Log collectors (Fluentd, Filebeat)
- Monitoring agents (Prometheus Node Exporter, Datadog)
- Network plugins (Calico, Cilium)
- Storage daemons
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
spec:
selector:
matchLabels:
name: node-exporter
template:
metadata:
labels:
name: node-exporter
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
containers:
- name: node-exporter
image: prom/node-exporter:latest
ports:
- containerPort: 9100
Answer:
A StatefulSet manages stateful applications that require stable, unique network identifiers, persistent storage, and ordered deployment/scaling/deletion. Unlike Deployments, StatefulSets give each Pod a unique, stable hostname.
When to use:
- Databases (MySQL, PostgreSQL, Cassandra)
- Message brokers (Kafka, RabbitMQ)
- Any app needing stable network identity
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 3
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:14
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
Pods are named with ordinal suffixes: postgres-0, postgres-1, postgres-2.
Answer:
Job: Creates one or more Pods and ensures they complete successfully. It is used for batch processing or one-time tasks.
apiVersion: batch/v1
kind: Job
metadata:
name: data-migration
spec:
completions: 1
parallelism: 1
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: my-migration-tool
command: ["./migrate.sh"]
CronJob: Creates Jobs on a recurring schedule using cron syntax.
apiVersion: batch/v1
kind: CronJob
metadata:
name: daily-backup
spec:
schedule: "0 2 * * *" # Every day at 2 AM
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: backup
image: my-backup-tool
command: ["./backup.sh"]
Answer:
Amazon Elastic Kubernetes Service (EKS) is a fully managed Kubernetes service from AWS that simplifies running Kubernetes by handling the control plane infrastructure, upgrades, patches, and high availability.
Key benefits:
- AWS manages the Kubernetes control plane across multiple Availability Zones
- Certified Kubernetes conformant — compatible with standard K8s tooling
- Integrates natively with AWS services (IAM, VPC, ALB, ECR, CloudWatch)
- Supports EC2 nodes, Fargate, and EKS Anywhere
- Automated Kubernetes version upgrades
# Create EKS cluster using eksctl
eksctl create cluster \
--name my-cluster \
--region us-east-1 \
--nodegroup-name standard-nodes \
--node-type t3.medium \
--nodes 3 \
--nodes-min 1 \
--nodes-max 5 \
--managed
# Update kubeconfig
aws eks update-kubeconfig --name my-cluster --region us-east-1
Answer:
| Feature | EKS | Self-Managed Kubernetes |
|---|---|---|
| Control Plane | Fully managed by AWS | You manage it |
| Upgrades | Simplified with one-click | Manual and complex |
| HA | Multi-AZ by default | Must configure manually |
| etcd backup | Managed by AWS | Your responsibility |
| Cost | $0.10/hour per cluster + node costs | Only node costs |
| AWS Integration | Native (IAM, VPC, ALB) | Manual configuration |
| Flexibility | Less control over control plane | Full control |
Answer:
Node Groups in EKS are collections of EC2 instances (worker nodes) that share the same configuration. There are two types:
Managed Node Groups:
- AWS provisions, registers, and terminates nodes automatically
- Supports EC2 Auto Scaling Groups
- Handles AMI updates and node lifecycle
- Easier to maintain; recommended for most cases
Self-Managed Node Groups:
- You manage the EC2 instances manually
- More control over AMIs and configurations
- Required for specialized hardware (GPU, custom kernel)
# Create managed node group
eksctl create nodegroup \
--cluster my-cluster \
--name gpu-nodes \
--node-type p3.2xlarge \
--nodes 2 \
--managed
# List node groups
eksctl get nodegroup --cluster my-cluster
Answer:
AWS Fargate is a serverless compute engine for containers. With EKS + Fargate, you can run Pods without provisioning or managing EC2 instances — AWS manages the underlying infrastructure automatically.
Key points:
- No node management; pay per Pod CPU/memory
- Uses Fargate Profiles to define which Pods run on Fargate
- Each Pod gets its own isolated micro-VM (enhanced security)
- DaemonSets are NOT supported on Fargate
# Fargate Profile via eksctl
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: my-cluster
region: us-east-1
fargateProfiles:
- name: default
selectors:
- namespace: default
- namespace: kube-system
# Create fargate profile
eksctl create fargateprofile \
--cluster my-cluster \
--name my-profile \
--namespace my-namespace
Answer:
EKS uses AWS IAM for authentication and Kubernetes RBAC for authorization.
Authentication flow:
kubectlcalls the AWS CLI/SDK to get a pre-signed token via STS- The token is passed to the Kubernetes API server
- The EKS cluster verifies the token with AWS IAM
- Kubernetes RBAC is checked for authorization
# Configure kubectl for EKS
aws eks update-kubeconfig \
--name my-cluster \
--region us-east-1 \
--role-arn arn:aws:iam::123456789:role/eks-admin-role
# Verify access
kubectl auth can-i get pods
# Check kubeconfig
kubectl config view
Answer:
| Feature | Deployment | StatefulSet |
|---|---|---|
| Pod identity | Interchangeable (random names) | Stable, unique (pod-0, pod-1) |
| Storage | Shared or ephemeral | Dedicated PVC per Pod |
| Scaling | Any order | Ordered (0, 1, 2…) |
| DNS | Single service endpoint | Individual headless DNS per Pod |
| Use case | Stateless apps | Stateful apps (DBs, queues) |
| Rolling update | Parallel or rolling | Sequential (n-1 to 0) |
# StatefulSet pod DNS format:
# <pod-name>.<service-name>.<namespace>.svc.cluster.local
# Example: postgres-0.postgres.default.svc.cluster.local
Answer:
PersistentVolume (PV): A piece of storage in the cluster provisioned by an admin or dynamically via StorageClass. It exists independently of Pods.
PersistentVolumeClaim (PVC): A request for storage by a user. It binds to a PV that matches its requirements (size, access mode, StorageClass).
Access Modes:
ReadWriteOnce (RWO)— can be mounted by a single nodeReadOnlyMany (ROX)— can be mounted by many nodes in read-only modeReadWriteMany (RWX)— can be mounted by many nodes in read/write mode
# PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: gp2
# Using PVC in a Pod
spec:
containers:
- name: app
volumeMounts:
- mountPath: /data
name: my-storage
volumes:
- name: my-storage
persistentVolumeClaim:
claimName: my-pvc
Answer:
A StorageClass defines the “class” of storage (e.g., SSD, HDD, network storage) and enables dynamic provisioning of PersistentVolumes when a PVC is created.
# EKS GP3 StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "3000"
throughput: "125"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
Reclaim policies:
Delete— automatically delete PV when PVC is deletedRetain— keep PV after PVC deletion for manual reclamation
Answer:
A rolling update gradually replaces old Pod instances with new ones to ensure zero downtime.
Parameters:
maxSurge— max Pods above desired count during updatemaxUnavailable— max Pods that can be unavailable during update
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Allow 1 extra pod
maxUnavailable: 0 # No downtime; always keep all pods running
Process:
- Create a new ReplicaSet with the updated Pod template
- Scale up new ReplicaSet by
maxSurgePods - Scale down old ReplicaSet by
maxUnavailablePods - Repeat until all Pods are updated
# Update image (triggers rolling update)
kubectl set image deployment/my-app my-app=my-app:v2
# Monitor rollout
kubectl rollout status deployment/my-app
# Pause and resume
kubectl rollout pause deployment/my-app
kubectl rollout resume deployment/my-app
Answer:
Liveness Probe: Determines if a container is running. If it fails, the kubelet restarts the container.
Readiness Probe: Determines if a container is ready to accept traffic. If it fails, the Pod is removed from Service endpoints (no traffic sent to it).
Startup Probe: Checks if an application has started. Useful for slow-starting containers; disables liveness/readiness until startup succeeds.
spec:
containers:
- name: app
image: my-app
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
startupProbe:
httpGet:
path: /healthz
port: 8080
failureThreshold: 30
periodSeconds: 10
Probe types: httpGet, tcpSocket, exec (command)
Answer:
HPA automatically scales the number of Pods in a Deployment or StatefulSet based on observed CPU/memory utilization or custom metrics.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: my-app-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
# Create HPA
kubectl autoscale deployment my-app --cpu-percent=70 --min=2 --max=10
# Check HPA status
kubectl get hpa
Note: HPA requires the metrics-server to be installed in the cluster.
Answer:
The kube-scheduler is a control plane component that watches for newly created Pods with no assigned node and selects a node for them to run on.
Scheduling process (two phases):
- Filtering (Predicates): Finds nodes that are feasible for the Pod (resource availability, taints/tolerations, node selectors, etc.)
- Scoring (Priorities): Ranks feasible nodes based on scoring functions (resource balance, affinity rules, etc.)
Factors considered:
- Resource requests and limits
- Node labels and selectors
- Taints and tolerations
- Affinity and anti-affinity rules
- Pod topology spread constraints
- Node pressure (DiskPressure, MemoryPressure)
# Force Pod to a specific node
spec:
nodeSelector:
kubernetes.io/hostname: ip-192-168-1-100
# Or use nodeName directly
spec:
nodeName: ip-192-168-1-100
Answer:
Taints are applied to nodes to repel Pods that don’t explicitly tolerate the taint. Tolerations are applied to Pods to allow them to be scheduled on tainted nodes.
Taint effects:
NoSchedule— Pods without toleration are not scheduled on the nodePreferNoSchedule— Kubernetes tries to avoid scheduling Pods without tolerationNoExecute— Existing Pods without toleration are evicted
# Add a taint to a node
kubectl taint nodes node1 dedicated=gpu:NoSchedule
# Remove a taint
kubectl taint nodes node1 dedicated=gpu:NoSchedule-
# Pod with toleration
spec:
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
Use cases:
- Dedicated nodes for specific workloads (GPU nodes, prod nodes)
- Preventing non-system Pods on control plane nodes
Answer:
Node Affinity: Constrains which nodes a Pod can be scheduled on based on node labels. More expressive than nodeSelector.
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: # hard rule
nodeSelectorTerms:
- matchExpressions:
- key: instance-type
operator: In
values: ["m5.xlarge", "m5.2xlarge"]
preferredDuringSchedulingIgnoredDuringExecution: # soft rule
- weight: 1
preference:
matchExpressions:
- key: zone
operator: In
values: ["us-east-1a"]
Pod Affinity / Anti-Affinity: Schedules Pods relative to other Pods.
# Anti-affinity: spread pods across nodes
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: my-app
topologyKey: kubernetes.io/hostname
Answer:
Role-Based Access Control (RBAC) regulates access to Kubernetes resources based on the roles of users or service accounts.
Key objects:
| Object | Scope | Purpose |
|---|---|---|
Role | Namespace | Grants permissions within a namespace |
ClusterRole | Cluster-wide | Grants permissions across all namespaces |
RoleBinding | Namespace | Binds Role to user/group/service account |
ClusterRoleBinding | Cluster-wide | Binds ClusterRole cluster-wide |
# Role — allows reading pods in default namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods-binding
namespace: default
subjects:
- kind: ServiceAccount
name: my-service-account
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Answer:
An Ingress resource defines HTTP/HTTPS routing rules to Services. An Ingress Controller implements those rules (e.g., NGINX, Traefik, AWS ALB Ingress Controller).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- myapp.example.com
secretName: tls-secret
rules:
- host: myapp.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 80
In EKS, the AWS Load Balancer Controller creates ALBs automatically from Ingress resources using annotations.
Answer:
etcd is a distributed, consistent key-value store used as Kubernetes’ backing store for all cluster data. Every API object (Pods, Services, ConfigMaps, Secrets, etc.) is stored in etcd.
Key properties:
- Consistency: Uses Raft consensus algorithm for leader election and data replication
- High availability: Typically run as a 3 or 5-node cluster (odd number for quorum)
- Watch API: Enables Kubernetes controllers to watch for changes
Important facts:
- All communication with etcd goes through the API server
- Backing up etcd is critical for disaster recovery
- In EKS, etcd is fully managed by AWS
# In a self-managed cluster — backup etcd
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Answer:
A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using custom controllers and CRDs. Operators encode operational knowledge (how to deploy, scale, upgrade, backup) into software.
Operator pattern:
- Define a CRD (e.g.,
PostgreSQLCluster) - Implement a Controller that watches the CRD
- Controller reconciles the actual state with the desired state
Popular Operators:
- Prometheus Operator
- PostgreSQL Operator (Zalando or CrunchyData)
- Cert-Manager
- ArgoCD
# Example: Using the Prometheus Operator CRD
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
replicas: 2
retention: 30d
storage:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 50Gi
Answer:
| Feature | Role | ClusterRole |
|---|---|---|
| Scope | Single namespace | Cluster-wide |
| Use case | Namespace-scoped resources | Cluster-scoped resources (Nodes, PVs) or all namespaces |
| Bound by | RoleBinding | ClusterRoleBinding (or RoleBinding for namespace scope) |
A ClusterRole can be bound in two ways:
- ClusterRoleBinding → grants access across all namespaces
- RoleBinding → grants ClusterRole access within a specific namespace only
This is useful when you want to reuse a ClusterRole definition across multiple namespaces.
Answer:
Kubernetes uses CoreDNS as the cluster DNS server. Every Pod gets a /etc/resolv.conf pointing to the CoreDNS service IP. Services and Pods are accessible via DNS.
DNS naming format:
# Service
<service-name>.<namespace>.svc.cluster.local
# Pod
<pod-ip-dashes>.<namespace>.pod.cluster.local
# Example: 10-244-1-5.default.pod.cluster.local
# Headless service Pods (StatefulSet)
<pod-name>.<service-name>.<namespace>.svc.cluster.local
# Example: mysql-0.mysql.default.svc.cluster.local
# Test DNS from inside a Pod
kubectl exec -it my-pod -- nslookup kubernetes.default
kubectl exec -it my-pod -- curl http://my-service.my-namespace.svc.cluster.local
Answer:
The EKS Control Plane includes the Kubernetes API server, scheduler, controller manager, and etcd. AWS:
- Runs the control plane across 3 Availability Zones for HA
- Manages etcd backups automatically
- Handles security patches and control plane upgrades
- Provides a dedicated API server endpoint for each cluster
- Monitors and auto-replaces unhealthy control plane nodes
Customers are responsible for:
- Worker nodes and node groups
- Application deployments
- Kubernetes version upgrades (with AWS assistance)
# Check EKS cluster status
aws eks describe-cluster --name my-cluster --region us-east-1
# List EKS clusters
aws eks list-clusters --region us-east-1
Answer:
EKS uses a webhook token authenticator that verifies AWS IAM identities:
kubectlrequests a pre-signed STS token viaaws eks get-token- The token is sent to the Kubernetes API server
- The API server passes it to the AWS IAM Authenticator webhook
- The webhook calls STS to validate the token and returns the IAM identity
- Kubernetes maps the IAM identity to a Kubernetes RBAC user/group via the
aws-authConfigMap (or EKS Access Entries)
# Get token manually (for debugging)
aws eks get-token --cluster-name my-cluster
# Check what identity kubectl uses
kubectl auth whoami
aws sts get-caller-identity
Answer:
The aws-auth ConfigMap in the kube-system namespace maps AWS IAM principals (users, roles) to Kubernetes RBAC users and groups. This controls who can access the cluster and with what permissions.
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::123456789:role/eks-node-group-role
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::123456789:role/eks-admin-role
username: admin
groups:
- system:masters
mapUsers: |
- userarn: arn:aws:iam::123456789:user/john
username: john
groups:
- developers
Note: AWS now recommends EKS Access Entries (API-based) as the preferred alternative to the aws-auth ConfigMap.
Answer:
The Amazon VPC CNI (Container Network Interface) plugin is the default networking plugin for EKS. It assigns real VPC IP addresses to Pods from the node’s subnet, enabling direct communication between Pods and other AWS resources.
Key features:
- Each Pod gets a real VPC IP address (not a virtual overlay network)
- Pods can communicate directly with RDS, ElastiCache, and other AWS services
- Security Groups can be applied directly to Pods (
SecurityGroupPolicy) - Supports IPv4 and IPv6
# Check VPC CNI version
kubectl describe daemonset aws-node -n kube-system | grep Image
# Check IP address allocation per node
kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,MAX_PODS:.status.capacity.pods'
IP address calculation:
Each EC2 instance type has a limit on ENIs and IPs per ENI. Max Pods = (ENIs × (IPs per ENI - 1)) + 2
Answer:
Methods to scale EKS nodes:
Cluster Autoscaler (CA): Automatically adjusts the number of nodes in an Auto Scaling Group based on pending Pods.
Karpenter: AWS-native node autoscaler that provisions optimal EC2 instances on demand (faster and more flexible than CA).
Manual scaling: Update the desired count in the ASG or via eksctl.
# Manual scaling with eksctl
eksctl scale nodegroup \
--cluster my-cluster \
--name standard-nodes \
--nodes 5 \
--nodes-min 2 \
--nodes-max 10
# Cluster Autoscaler deployment annotation
spec:
template:
spec:
containers:
- name: cluster-autoscaler
command:
- ./cluster-autoscaler
- --cloud-provider=aws
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/my-cluster
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
Answer:
Karpenter is an open-source, high-performance node autoscaler for Kubernetes, originally built by AWS. It provisions the right EC2 instance types for your workloads in seconds, rather than minutes.
Advantages over Cluster Autoscaler:
- Provisions nodes directly (no ASG required)
- Supports diverse instance types automatically (spot + on-demand mix)
- Consolidates underutilized nodes automatically
- Node provisioning in ~60 seconds vs 2-5 minutes for CA
# NodePool (Karpenter v0.30+)
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: default
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
- key: kubernetes.io/arch
operator: In
values: ["amd64"]
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
disruption:
consolidationPolicy: WhenUnderutilized
limits:
cpu: 1000
Answer:
EKS Anywhere allows you to create and manage Kubernetes clusters on your own infrastructure (on-premises, VMware vSphere, bare metal, or other cloud providers) using the same EKS tools and configurations used in AWS.
Use cases:
- Regulatory requirements that prevent cloud usage
- Data sovereignty requirements
- Hybrid cloud architectures
- Air-gapped environments
Key features:
- Uses same EKS configuration API
- Supports curated packages (CoreDNS, Cilium, etc.)
- Optionally connect to AWS via EKS Connector for management in the AWS Console
Answer:
The AWS Load Balancer Controller is a controller that manages AWS Elastic Load Balancers for Kubernetes clusters. It provisions:
- Application Load Balancers (ALBs) for Ingress resources
- Network Load Balancers (NLBs) for Service type LoadBalancer
# ALB Ingress via AWS Load Balancer Controller
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
alb.ingress.kubernetes.io/ssl-redirect: "443"
spec:
rules:
- host: myapp.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service
port:
number: 80
# Install AWS Load Balancer Controller via Helm
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
Answer:
EKS Add-ons are operational software components that extend the functionality of Kubernetes. AWS manages their lifecycle (installation, updates, conflict resolution).
Available add-ons:
kube-proxy— network proxycoredns— cluster DNSvpc-cni— Pod networkingaws-ebs-csi-driver— EBS storageaws-efs-csi-driver— EFS storageadot— AWS Distro for OpenTelemetryamazon-cloudwatch-observability— CloudWatch monitoringeks-pod-identity-agent— Pod identity
# List available add-ons
aws eks describe-addon-versions --kubernetes-version 1.29
# Install an add-on
aws eks create-addon \
--cluster-name my-cluster \
--addon-name aws-ebs-csi-driver \
--service-account-role-arn arn:aws:iam::123456789:role/ebs-csi-role
# List installed add-ons
aws eks list-addons --cluster-name my-cluster
Answer:
Options for secret management in EKS:
- Kubernetes Secrets (base64 encoded; encrypt with AWS KMS for security)
- AWS Secrets Manager + Secrets Store CSI Driver (mount secrets as volumes)
- AWS Systems Manager Parameter Store (same CSI driver)
- External Secrets Operator (sync external secrets to Kubernetes Secrets)
# Using Secrets Store CSI Driver with AWS Secrets Manager
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets
spec:
provider: aws
parameters:
objects: |
- objectName: "prod/myapp/db-password"
objectType: secretsmanager
objectAlias: db-password
---
spec:
containers:
- name: app
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: aws-secrets
# Enable EKS secrets encryption with KMS
aws eks create-cluster \
--name my-cluster \
--encryption-config "resources=[secrets],provider={keyArn=arn:aws:kms:...}"
Answer:
Kubernetes enforces a flat networking model with these requirements:
- All Pods can communicate with each other without NAT
- All Nodes can communicate with all Pods without NAT
- The IP a Pod sees for itself is the same IP others see
Network layers:
- Pod-to-Pod — via CNI plugin (VPC CNI, Calico, Cilium, Flannel)
- Pod-to-Service — via kube-proxy (iptables or IPVS rules)
- External-to-Service — via NodePort, LoadBalancer, or Ingress
CNI Plugin responsibilities:
- Assign IP addresses to Pods
- Set up routing rules
- Handle network policy enforcement (Calico, Cilium)
# Inspect CNI config on a node
cat /etc/cni/net.d/10-aws.conflist
# Trace network path
kubectl exec -it my-pod -- traceroute 10.100.0.1
Answer:
A CRD extends the Kubernetes API by defining new resource types. Once a CRD is registered, you can create instances of it using kubectl like any built-in resource.
# Define a CRD
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: databases.mycompany.io
spec:
group: mycompany.io
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
engine:
type: string
enum: [postgres, mysql]
replicas:
type: integer
minimum: 1
scope: Namespaced
names:
plural: databases
singular: database
kind: Database
# Use the custom resource
apiVersion: mycompany.io/v1
kind: Database
metadata:
name: my-db
spec:
engine: postgres
replicas: 3
Answer:
VPA automatically adjusts the CPU and memory requests/limits of containers based on actual usage. Unlike HPA (which scales replicas), VPA scales resources per container.
VPA modes:
Off— only provides recommendations (no changes)Initial— sets resources only at Pod creationAuto— updates resources and evicts/restarts Pods to apply changes
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: my-app-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: my-app
updatePolicy:
updateMode: "Auto"
resourcePolicy:
containerPolicies:
- containerName: my-app
minAllowed:
cpu: 100m
memory: 128Mi
maxAllowed:
cpu: 2
memory: 2Gi
Note: VPA and HPA should not be used together on the same metric (e.g., both on CPU). HPA + VPA on different metrics (e.g., HPA on custom metrics, VPA on CPU/memory) can work together.
Answer:
A PDB limits the number of Pods of a replicated application that are down simultaneously during voluntary disruptions (node drains, cluster upgrades, evictions).
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: my-app-pdb
spec:
selector:
matchLabels:
app: my-app
minAvailable: 2 # OR use maxUnavailable
# maxUnavailable: 1 # Maximum 1 Pod can be unavailable
Use cases:
- Ensures minimum replicas during node drain for upgrades
- Protects stateful applications from data loss during disruptions
- Works with the Cluster Autoscaler and Karpenter
# Check PDB status
kubectl get pdb
# During cluster upgrade — drain blocks if PDB would be violated
kubectl drain node1 --ignore-daemonsets --delete-emptydir-data
Answer:
Kubernetes implements service discovery in two ways:
1. DNS-based (recommended):
- CoreDNS resolves Service names to ClusterIPs
<service>.<namespace>.svc.cluster.local
2. Environment variables:
- At Pod start, Kubernetes injects env vars for all Services in the namespace
- e.g.,
MY_SERVICE_SERVICE_HOST,MY_SERVICE_SERVICE_PORT - Limitation: only works for Services created before the Pod
Headless Services (for StatefulSets):
- Set
clusterIP: None - DNS returns individual Pod IPs instead of a single VIP
- Enables direct Pod addressing
# Headless service
apiVersion: v1
kind: Service
metadata:
name: my-stateful-svc
spec:
clusterIP: None
selector:
app: my-stateful-app
ports:
- port: 5432
Answer:
A Service Mesh is a dedicated infrastructure layer that manages service-to-service communication (traffic management, observability, security) using sidecar proxies without changing application code.
Istio architecture:
- Data Plane: Envoy sidecar proxies (injected automatically into Pods) handle all traffic
- Control Plane (Istiod): Manages proxy configuration, certificate lifecycle, and traffic policies
Key Istio features:
- Traffic management: canary deployments, circuit breaking, retries, timeouts
- mTLS: automatic mutual TLS between services
- Observability: distributed tracing (Jaeger), metrics (Prometheus), logging
- Authorization policies: fine-grained L7 access control
# VirtualService — traffic splitting (canary)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-app
spec:
hosts:
- my-app
http:
- route:
- destination:
host: my-app
subset: v1
weight: 90
- destination:
host: my-app
subset: v2
weight: 10
Answer:
Admission Controllers are plugins that intercept API server requests after authentication and authorization but before persisting objects to etcd. They can validate or mutate requests.
Two types:
- Mutating Admission Webhooks: Modify the request (e.g., inject sidecar, add labels/defaults)
- Validating Admission Webhooks: Allow or reject the request (e.g., enforce policies)
Built-in admission controllers:
LimitRanger— enforces resource limitsResourceQuota— enforces namespace quotasPodSecurity— enforces Pod security standardsMutatingAdmissionWebhook— calls external webhook for mutationsValidatingAdmissionWebhook— calls external webhook for validation
# Validating Webhook configuration
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: my-policy-webhook
webhooks:
- name: validate.mycompany.io
clientConfig:
service:
name: policy-service
namespace: kube-system
path: /validate
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["apps"]
resources: ["deployments"]
admissionReviewVersions: ["v1"]
sideEffects: None
Answer:
Network Policies are Kubernetes resources that control traffic flow at the IP/port level between Pods, namespaces, and external endpoints. They require a CNI plugin that supports them (Calico, Cilium, Weave).
# Deny all ingress, allow only from specific namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-from-frontend
namespace: backend
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: frontend
podSelector:
matchLabels:
role: api
ports:
- protocol: TCP
port: 5432
egress:
- to:
- ipBlock:
cidr: 10.0.0.0/8
Default behavior: Without a NetworkPolicy, all traffic is allowed. Once a NetworkPolicy selects a Pod, that Pod follows the policy’s rules.
Answer:
Kubernetes is not inherently multi-tenant but can be made so using multiple isolation mechanisms:
Soft multi-tenancy (shared cluster):
- Namespaces — logical isolation
- RBAC — access control per team/namespace
- ResourceQuotas — limit resource consumption per namespace
- LimitRanges — default/max resources per Pod in a namespace
- Network Policies — isolate network traffic between namespaces
- Pod Security Standards — enforce security contexts
Hard multi-tenancy (strong isolation):
- Separate clusters per tenant (VCluster, separate EKS clusters)
- VCluster — virtual Kubernetes clusters inside a namespace
- Capsule / HNC — multi-tenancy frameworks
# ResourceQuota per tenant namespace
apiVersion: v1
kind: ResourceQuota
metadata:
name: tenant-a-quota
namespace: tenant-a
spec:
hard:
requests.cpu: "10"
requests.memory: 20Gi
limits.cpu: "20"
limits.memory: 40Gi
pods: "50"
services: "10"
Answer:
kube-proxy manages network rules on nodes for Service routing. It supports three modes:
| Feature | iptables | IPVS |
|---|---|---|
| Routing | Sequential rule matching | Hash table lookup |
| Performance | Degrades at scale (O(n)) | Constant time (O(1)) |
| Load balancing algorithms | Round-robin only | RR, least connections, source hash, etc. |
| Scale | Good up to ~1000 Services | Scales to 10,000+ Services |
| Health checking | Limited | Built-in |
# Check current kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
# Switch to IPVS mode (via configmap)
kubectl edit configmap kube-proxy -n kube-system
# Set: mode: "ipvs"
For large clusters (>1000 Services), IPVS mode is strongly recommended. EKS also supports IPVS mode.
rk?
Answer:
Kubernetes garbage collection automatically removes objects that are no longer needed:
1. Owner References & Cascading Deletion:
- Resources have
ownerReferencespointing to their owner (e.g., Pod → ReplicaSet → Deployment) - When an owner is deleted, dependents are deleted via Foreground or Background cascading deletion
# Delete with cascade (default: background)
kubectl delete deployment my-app
# Orphan dependents (don't delete ReplicaSet/Pods)
kubectl delete deployment my-app --cascade=orphan
2. Image Garbage Collection:
kubeletremoves unused container images when disk usage exceedsimageGCHighThresholdPercent(default 85%)
3. Container Garbage Collection:
- Removes terminated containers based on
MaxContainerCountandMaxDeadContainerAge
4. API Resource GC:
- Removes completed Jobs, finished Pods (based on
ttlSecondsAfterFinished)
# Auto-delete Job after 60 seconds
spec:
ttlSecondsAfterFinished: 60
Answer:
Cluster Autoscaler (CA) automatically adjusts the size of a Kubernetes cluster by adding or removing nodes based on Pod scheduling needs.
Scale-up logic:
- Pod is Pending because no node has enough resources
- CA simulates which node group could schedule the Pod
- CA increases the ASG desired count
- New node joins, Pod is scheduled
Scale-down logic:
- CA checks if a node’s utilization drops below 50% for 10+ minutes
- Verifies all Pods can be rescheduled on other nodes
- Drains the node and terminates the EC2 instance
# EKS — ASG tags required for CA
Tags:
k8s.io/cluster-autoscaler/enabled: "true"
k8s.io/cluster-autoscaler/<cluster-name>: "owned"
Key CA flags:
--scale-down-delay-after-add=10m # Wait after scale-up before scale-down check
--scale-down-unneeded-time=10m # Node must be unneeded for this long
--skip-nodes-with-local-storage=false # Allow scale-down of nodes with local storage
Answer:
GitOps is an operational framework where Git is the single source of truth for infrastructure and application configuration. Changes are made via Git commits/PRs, and automated agents reconcile the cluster state to match.
GitOps tools for Kubernetes:
- ArgoCD — declarative, Git-based continuous delivery
- Flux CD — lightweight, GitOps toolkit for Kubernetes
ArgoCD workflow:
- Developer commits Kubernetes manifests to Git
- ArgoCD detects the diff between Git and cluster state
- ArgoCD syncs the cluster (applies manifests)
- Health status is reported back
# ArgoCD Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/my-org/my-app
targetRevision: main
path: k8s/
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
Answer:
CrashLoopBackOff means a container is repeatedly crashing and Kubernetes is backing off before restarting it.
Step-by-step troubleshooting:
# 1. Check Pod status and events
kubectl get pods
kubectl describe pod <pod-name>
# 2. Check current logs
kubectl logs <pod-name>
# 3. Check previous container logs (if container already crashed)
kubectl logs <pod-name> --previous
# 4. Check exit code (tells you why the container exited)
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# 5. Debug with a shell (if container has bash)
kubectl debug -it <pod-name> --image=busybox --target=<container-name>
Common exit codes:
| Exit Code | Meaning |
|---|---|
0 | Success (no crash — liveness probe failing?) |
1 | Application error |
137 | OOMKilled (out of memory) |
139 | Segmentation fault |
143 | Graceful termination (SIGTERM) |
Common root causes:
- Wrong command or entrypoint in the container
- Missing environment variables or secrets
- Application fails healthcheck (liveness probe)
- OOMKilled — increase memory limits
- Bad image or missing dependencies
Answer:
OPA Gatekeeper is a policy engine for Kubernetes built on Open Policy Agent (OPA). It uses validating admission webhooks and CRDs to enforce custom policies.
Key concepts:
- ConstraintTemplate — defines the policy logic in Rego
- Constraint — an instance of a ConstraintTemplate with specific parameters
# ConstraintTemplate — enforce required labels
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: requirelabels
spec:
crd:
spec:
names:
kind: RequireLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requirelabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("Missing required labels: %v", [missing])
}
---
# Constraint — apply the policy
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireLabels
metadata:
name: must-have-env-label
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
parameters:
labels: ["env", "team", "app"]
Answer:
IRSA (IAM Roles for Service Accounts) allows Kubernetes Pods to assume AWS IAM roles using Kubernetes Service Accounts. This replaces the old pattern of assigning IAM roles to EC2 nodes.
How it works:
- EKS cluster has an OIDC provider configured
- IAM role has a trust policy allowing the OIDC provider and specific service account
- Pod uses a service account annotated with the IAM role ARN
- The EKS Pod Identity Webhook injects AWS credential env vars into the Pod
- AWS SDK in the Pod automatically fetches temporary credentials via OIDC token
# Create OIDC provider for EKS cluster
eksctl utils associate-iam-oidc-provider \
--cluster my-cluster \
--approve
# Create IAM role for service account
eksctl create iamserviceaccount \
--cluster my-cluster \
--namespace my-namespace \
--name my-service-account \
--attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess \
--approve
# Service Account with IRSA annotation
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-service-account
namespace: my-namespace
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/my-pod-role
Answer:
EKS Pod Identity is a newer, simpler mechanism for granting AWS permissions to Pods. It uses a dedicated Pod Identity Agent DaemonSet instead of OIDC webhooks.
Comparison:
| Feature | IRSA | EKS Pod Identity |
|---|---|---|
| Mechanism | OIDC + webhook | Pod Identity Agent DaemonSet |
| IAM trust policy | Complex (OIDC condition) | Simple (pods.eks.amazonaws.com) |
| Cross-account | Supported | Supported |
| Cluster config | OIDC provider required | Agent add-on required |
| Simplicity | More complex setup | Simpler setup |
# Enable Pod Identity add-on
aws eks create-addon \
--cluster-name my-cluster \
--addon-name eks-pod-identity-agent
# Create Pod Identity association
aws eks create-pod-identity-association \
--cluster-name my-cluster \
--namespace my-namespace \
--service-account my-service-account \
--role-arn arn:aws:iam::123456789:role/my-pod-role
Answer:
Multi-cluster architectures improve availability, separate concerns, and meet compliance requirements.
Common patterns:
1. Active-Active (Global Load Balancing):
- Multiple EKS clusters in different regions
- Route 53 latency/geolocation routing between clusters
- Data synchronization via CRDTs or database replication
2. Active-Passive (Disaster Recovery):
- Primary cluster in one region, standby in another
- Velero for backup/restore
- Route 53 failover routing
3. Hub-Spoke (Management Cluster):
- Central management cluster running ArgoCD/Flux
- Spoke clusters receive workloads from the hub
Tools for multi-cluster:
- ArgoCD — multi-cluster GitOps
- Cluster API (CAPI) — manage cluster lifecycle
- AWS App Mesh — cross-cluster service mesh
- Velero — backup and DR
# Register multiple clusters in ArgoCD
argocd cluster add --kubeconfig ./cluster2-kubeconfig arn:aws:eks:us-west-2:123456789:cluster/cluster2
Answer:
Cost optimization strategies:
1. Right-size workloads:
- Use VPA recommendations to set appropriate resource requests
- Avoid over-provisioning CPU/memory
2. Spot Instances:
- Use Karpenter or CA with mixed instance types and Spot
- Design apps to handle interruptions gracefully (2-minute notice)
3. Node consolidation:
- Enable Karpenter’s
consolidationpolicy to bin-pack Pods
4. Fargate for variable workloads:
- Only pay for actual Pod CPU/memory
5. Cluster Autoscaler / Karpenter:
- Scale down idle nodes automatically
6. Savings Plans & Reserved Instances:
- Commit to 1 or 3 years for baseline workloads
# Karpenter — prefer Spot, fall back to On-Demand
spec:
template:
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
disruption:
consolidationPolicy: WhenUnderutilized
consolidateAfter: 30s
# Monitor costs with Kubecost
helm install kubecost cost-analyzer \
--repo https://kubecost.github.io/cost-analyzer/ \
--namespace kubecost --create-namespace
Answer:
Amazon EKS Distro (EKS-D) is the same Kubernetes distribution that powers Amazon EKS, made available for you to use anywhere. It is a free, open-source distribution of Kubernetes that includes:
- The same Kubernetes version and patches used in EKS
- Extended support timelines for older K8s versions
- Same versions of dependencies: etcd, CoreDNS, metrics-server, etc.
- Amazon-tested and signed binaries
Use cases:
- Run the same Kubernetes distribution on-premises as in EKS
- Consistent behavior across hybrid environments
- Foundation for EKS Anywhere
Answer:
Blue/Green deployment runs two identical environments (blue = current, green = new) and switches traffic instantaneously.
Method 1: Kubernetes Services + Label Switching
# Switch traffic from blue to green by updating service selector
kubectl patch service my-service \
-p '{"spec":{"selector":{"version":"green"}}}'
Method 2: AWS ALB Weighted Target Groups
# Ingress with traffic splitting (AWS Load Balancer Controller)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.blue-green: |
{
"type": "forward",
"forwardConfig": {
"targetGroups": [
{"serviceName": "blue-service", "servicePort": 80, "weight": 0},
{"serviceName": "green-service", "servicePort": 80, "weight": 100}
]
}
}
Method 3: ArgoCD Rollouts (Argo Rollouts)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
blueGreen:
activeService: my-active-service
previewService: my-preview-service
autoPromotionEnabled: false
Answer:
In EKS, etcd is fully managed by AWS. You do not have direct access to etcd. AWS automatically handles:
- etcd backups (multiple times per day)
- Multi-AZ replication for etcd
- Automatic etcd recovery
For application-level DR:
- Velero — backs up Kubernetes resources and PersistentVolumes to S3
# Install Velero with AWS S3 backend
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.8.0 \
--bucket my-velero-bucket \
--backup-location-config region=us-east-1 \
--snapshot-location-config region=us-east-1 \
--secret-file ./credentials-velero
# Backup all resources in a namespace
velero backup create my-backup --include-namespaces production
# Schedule daily backups
velero schedule create daily-backup \
--schedule="0 1 * * *" \
--include-namespaces production
# Restore from backup
velero restore create --from-backup my-backup
Answer:
1. IAM and RBAC:
- Use IRSA or Pod Identity instead of node-level IAM roles
- Apply least-privilege IAM policies
- Use EKS Access Entries instead of aws-auth ConfigMap
- Regularly audit RBAC bindings
2. Network Security:
- Enable private API server endpoint
- Use Security Groups for Pods
- Implement Network Policies (Calico or Cilium)
- Use VPC endpoints for AWS service traffic
3. Secrets Management:
- Encrypt Kubernetes Secrets with KMS at rest
- Use AWS Secrets Manager via CSI driver or External Secrets Operator
4. Pod Security:
- Enforce Pod Security Standards (
Restrictedprofile) - Disable privilege escalation:
allowPrivilegeEscalation: false - Run containers as non-root users
- Use read-only root filesystems
5. Runtime Security:
- Enable Amazon GuardDuty for EKS (runtime threat detection)
- Use Falco for real-time runtime security
6. Image Security:
- Scan images with Amazon ECR image scanning (or Trivy/Snyk)
- Use immutable image tags
- Sign images with Notary/Cosign
# Restricted Pod Security Standard
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
Answer:
Three pillars of observability: Metrics, Logs, Traces
Metrics:
- Amazon CloudWatch Container Insights — native AWS monitoring for EKS
- Prometheus + Grafana — open-source, highly flexible
- Datadog / New Relic — enterprise observability platforms
Logs:
- Fluent Bit (DaemonSet) → CloudWatch Logs / OpenSearch
- Fluentd — more plugins, slightly heavier
- EKS control plane logging: enable in AWS Console/CLI
Traces:
- AWS X-Ray — native AWS distributed tracing
- OpenTelemetry (ADOT) — standard collection pipeline
- Jaeger / Tempo — open-source tracing
# Enable EKS Control Plane logging
aws eks update-cluster-config \
--name my-cluster \
--logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'
# Install Prometheus stack via Helm
helm install kube-prometheus-stack \
prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace \
--set grafana.adminPassword=admin123
# CloudWatch agent as a DaemonSet (Container Insights)
# Install via add-on
aws eks create-addon \
--cluster-name my-cluster \
--addon-name amazon-cloudwatch-observability
Answer:
EKS upgrade process (recommended steps):
Phase 1: Preparation
# 1. Review EKS release notes and deprecated APIs
# 2. Test upgrade in lower environments first
# 3. Backup with Velero
# Check current version
aws eks describe-cluster --name my-cluster --query cluster.version
# Check deprecated API usage
kubectl convert --help
# Use Pluto to detect deprecated APIs
pluto detect-all-in-cluster --target-versions k8s=v1.29.0
Phase 2: Upgrade the Control Plane
# Upgrade control plane (15-25 min, no downtime)
aws eks update-cluster-version \
--name my-cluster \
--kubernetes-version 1.29
# Wait for completion
aws eks wait cluster-active --name my-cluster
Phase 3: Upgrade Add-ons
# Update EKS add-ons (vpc-cni, coredns, kube-proxy)
aws eks update-addon \
--cluster-name my-cluster \
--addon-name vpc-cni \
--resolve-conflicts OVERWRITE
Phase 4: Upgrade Node Groups
# For Managed Node Groups
aws eks update-nodegroup-version \
--cluster-name my-cluster \
--nodegroup-name standard-nodes
# The process: new nodes → cordon old nodes → drain → terminate
# PodDisruptionBudgets are respected during drain
Phase 5: Validate
kubectl get nodes
kubectl get pods -A
kubectl get events -A | grep Warning
Key tip: Upgrade one minor version at a time (e.g., 1.27 → 1.28 → 1.29). Skipping versions is not supported.
kubectl expose deployment main-app --type=LoadBalancer --port=80 --target-port=8080 so it gets a cloud load balancer with a public IP.
2) Expose the database with kubectl expose deployment database --type=ClusterIP --port=5432 --target-port=5432 so it only gets an internal cluster IP and DNS name.
3) The main app connects to the database using the internal service name,
e.g. database:5432, which Kubernetes resolves via its internal DNS.
This separation keeps the database private and secure while making only the necessary application public-facing.Answer:
EKS security spans four layers — identity, network, workload, and data:
1. Identity & Access (who can do what):
# Prefer EKS Access Entries (or aws-auth ConfigMap) with least-privilege IAM roles
# Map each team/CI role to a scoped Kubernetes RBAC group — never system:masters for humans
aws eks create-access-entry --cluster-name my-cluster \
--principal-arn arn:aws:iam::123456789:role/dev-team-role \
--kubernetes-groups developers
- Use IRSA (IAM Roles for Service Accounts) or EKS Pod Identity so Pods get scoped AWS permissions — never mount broad node-level IAM roles into every Pod
- Enforce RBAC with least-privilege Roles/ClusterRoles; audit with
kubectl auth can-i --list
2. Network:
# Default-deny NetworkPolicy, then explicitly allow required traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny }
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
- Set the EKS API endpoint to private (or restrict public access to known CIDRs)
- Use Security Groups for Pods to control traffic at the ENI level for sensitive workloads
3. Workload (Pod Security Standards):
# Enforce restricted Pod Security Standard at the namespace level
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
pod-security.kubernetes.io/enforce: restricted
- Run containers as non-root, read-only root filesystem, drop all Linux capabilities by default
- Scan images in the pipeline (Trivy/ECR scan) and block CRITICAL vulnerabilities before deploy
4. Data & Audit:
- Enable secrets encryption with a customer-managed KMS key at cluster creation
- Enable control plane audit logging to CloudWatch Logs and alert on suspicious API calls (e.g.,
execinto prod Pods, RBAC changes) - Keep the EKS version and add-ons (
vpc-cni,coredns,kube-proxy) patched — AWS regularly ships CVE fixes for these
Interview summary line: “I’d treat EKS security as defense-in-depth: IRSA/Pod Identity instead of broad node roles, default-deny network policies, restricted Pod Security Standards enforced per namespace, KMS-encrypted secrets, and audit logging wired to alerts — so no single misconfiguration is enough to compromise the cluster.”
Answer:
CrashLoopBackOff means the container starts, exits, and Kubernetes keeps restarting it with exponential backoff. Work through it systematically:
# Step 1: See the restart count and last exit reason at a glance
kubectl get pod my-pod
# STATUS: CrashLoopBackOff RESTARTS: 7
# Step 2: Describe the pod — check Events at the bottom and the exit code
kubectl describe pod my-pod
# Look for: "Last State: Terminated, Reason: Error, Exit Code: 1"
# Step 3: Check logs from the CURRENT crashed attempt
kubectl logs my-pod
# Step 4: Check logs from the PREVIOUS attempt (often more useful —
# the current container may have crashed before logging anything)
kubectl logs my-pod --previous
# Step 5: Check resource limits — was it OOMKilled?
kubectl describe pod my-pod | grep -A5 "Last State"
# Exit Code 137 = OOMKilled → increase memory limits or fix a leak
# Step 6: Check readiness/liveness probes — a misconfigured probe can
# kill an otherwise-healthy container
kubectl get pod my-pod -o yaml | grep -A10 livenessProbe
# Step 7: Verify the image, command, and config are correct
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].image}'
kubectl get configmap,secret -n <namespace> # confirm referenced ones exist
# Step 8: If logs are empty, shell in via a debug container (image itself won't stay up)
kubectl debug my-pod -it --image=busybox --target=my-container
Common root causes, ranked by frequency: application error on startup (bad config/missing env var) → OOMKilled (limits too low) → failing liveness probe (too aggressive initialDelaySeconds) → missing ConfigMap/Secret referenced in the Pod spec → wrong command/entrypoint in the image.
Answer:
Since it’s failing from one specific pod and not every replica, the issue is almost always scoped to that pod’s node, network path, or local state — not the database or the Service itself.
# Step 1: Confirm it's really isolated to this pod, not intermittent everywhere
kubectl get pods -o wide -l app=myapp
# Note which node this pod is scheduled on vs. the healthy pods
# Step 2: Test DNS resolution from inside the failing pod
kubectl exec -it <failing-pod> -- nslookup mydb.namespace.svc.cluster.local
# If this fails but works from a healthy pod → CoreDNS or node-local DNS cache issue
# Step 3: Test raw TCP connectivity to the DB
kubectl exec -it <failing-pod> -- nc -zv mydb.namespace.svc.cluster.local 5432
# Connection refused vs. timeout tells you a lot:
# - refused → reaching the DB, but DB is rejecting (auth, max connections, pg_hba.conf)
# - timeout → not reaching the DB at all (network policy, security group, routing)
# Step 4: Compare NetworkPolicy exposure — is this pod missing a required label?
kubectl get pod <failing-pod> --show-labels
kubectl get networkpolicy -n <namespace> -o yaml
# A NetworkPolicy selector matching on labels can silently exclude one pod
# if its labels drifted (e.g., deployed from a different manifest/branch)
# Step 5: Check the NODE the pod is on for node-level issues
kubectl describe node <node-name> | grep -A5 Conditions
# Node-level security group / CNI issues affect every pod on that specific node
# Step 6: Check for connection pool / max_connections exhaustion on the DB side
# If 99 pods share a connection pool limit and this pod is the "one too many"
# request, the DB itself may be rejecting only the newest connections
# Step 7: Check the pod's actual runtime env vars — a stale ConfigMap/Secret
# mount is one of the most common "works everywhere except this pod" causes
kubectl exec -it <failing-pod> -- env | grep -i db
kubectl get pod <failing-pod> -o jsonpath='{.spec.containers[0].envFrom}'
Most common root causes, in order of likelihood: a NetworkPolicy that matches other pods but not this one (label drift), the pod scheduled on a node with a different Security Group/CNI config, a stale ConfigMap/Secret mounted before a rolling update rolled out to other pods, or the pod being the connection that tips the DB over max_connections.
Answer:
ClusterIP is the default Kubernetes Service type — it gives a Service a stable, virtual IP address that’s only reachable from inside the cluster, never from outside.
apiVersion: v1
kind: Service
metadata:
name: backend-svc
spec:
selector:
app: backend
ports:
- port: 80 # port the Service listens on
targetPort: 8080 # port the Pod's container actually listens on
type: ClusterIP # default — can be omitted
How traffic actually reaches a Pod:
kube-proxyon every node watches the API server for Services/Endpoints and programs local iptables (or IPVS) rules- When another Pod sends traffic to the ClusterIP, those rules intercept it and load-balance it across the Service’s healthy backend Pods (matched via the
selectorlabels) - The ClusterIP itself is virtual — it doesn’t exist on any real network interface, it’s purely an iptables/IPVS forwarding rule
DNS: CoreDNS automatically creates a record so other Pods can reach it by name instead of IP:
backend-svc.default.svc.cluster.local → resolves to the ClusterIP
Headless variant (clusterIP: None): skips the virtual IP and load-balancing entirely — DNS returns the individual Pod IPs directly instead. This is what StatefulSets use, since each Pod needs its own stable, addressable identity (e.g., postgres-0.postgres.default.svc.cluster.local) rather than being load-balanced as an interchangeable group.
When to use it: internal-only communication — a backend API that only the frontend Service should call, or a database that should never be reachable from outside the cluster. For external access you’d put a LoadBalancer or Ingress in front of it instead.
Answer:
Kubernetes has no built-in “user” object — identity comes from an external source (a client certificate, an OIDC token, or in EKS’s case, IAM), and RBAC controls what that identity can do once authenticated. The process has two distinct halves:
1. Give the person an identity (self-managed cluster — client certificate approach):
# Generate a private key and CSR for the new user
openssl genrsa -out john.key 2048
openssl req -new -key john.key -out john.csr -subj "/CN=john/O=developers"
# Submit as a Kubernetes CertificateSigningRequest and approve it
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: john-csr
spec:
request: $(cat john.csr | base64 | tr -d '\n')
signerName: kubernetes.io/kube-apiserver-client
usages: ["client auth"]
EOF
kubectl certificate approve john-csr
# Build a kubeconfig for John using the signed certificate
kubectl config set-credentials john --client-certificate=john.crt --client-key=john.key
On EKS, use IAM instead — map an IAM identity to a Kubernetes username via Access Entries (or the legacy aws-auth ConfigMap):
aws eks create-access-entry --cluster-name my-cluster \
--principal-arn arn:aws:iam::123456789:user/john \
--kubernetes-groups developers
2. Grant SPECIFIC authorization (both cases — this is where RBAC comes in):
# A narrow Role — only what "developers" should be able to do, nothing more
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: developer-access
namespace: staging
rules:
- apiGroups: ["", "apps"]
resources: ["pods", "deployments", "services", "configmaps"]
verbs: ["get", "list", "watch", "create", "update"]
# Notably NOT "delete" or access to "secrets" — least privilege
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: john-developer-binding
namespace: staging
subjects:
- kind: User
name: john # matches CN in the cert, or the mapped IAM username
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: developer-access
apiGroup: rbac.authorization.k8s.io
# Verify what John can actually do
kubectl auth can-i delete pods --as=john -n staging # should be "no"
kubectl auth can-i create deployments --as=john -n staging # should be "yes"
Key point for an interview: “adding a user” and “authorizing a user” are two separate steps — Kubernetes authenticates via an external identity source, then RBAC (Role/RoleBinding, scoped to a namespace like above) is what actually defines “specific authorization.” Always scope with a Role+RoleBinding (namespace-limited) rather than a ClusterRole+ClusterRoleBinding unless the person genuinely needs cluster-wide access.
Answer:
This is a scoping question interviewers use to gauge real hands-on scale — there’s no universal “correct” answer, but a strong response is specific and structured rather than vague. Example of how to structure it:
Scale: “I manage 3 EKS clusters — dev, staging, and production — each in a separate AWS account for blast-radius isolation.”
Configuration, be ready to specify:
Cluster version: EKS 1.29, upgraded one minor version at a time
Node groups: Managed node groups (m5.xlarge) for steady-state workloads,
Karpenter for burst/batch workloads on Spot
Networking: VPC CNI, private API endpoint, 3-AZ spread
Add-ons: vpc-cni, coredns, kube-proxy, aws-load-balancer-controller,
cluster-autoscaler (or Karpenter), external-dns
Ingress: AWS Load Balancer Controller provisioning ALBs from Ingress
Observability: Prometheus + Grafana for metrics, Fluent Bit → OpenSearch for logs
Secrets: Secrets Store CSI Driver backed by AWS Secrets Manager
GitOps/Deploys: ArgoCD, syncing from a dedicated gitops repo per environment
Scale (rough numbers): ~40 nodes, ~300 pods in production at peak
Why the structured version is stronger than a one-liner: it shows you actually operate the cluster day-to-day (know the add-ons, autoscaler choice, and observability stack) rather than having only deployed application workloads onto a cluster someone else configured. If you’re newer to Kubernetes, it’s fine to answer honestly at whatever scale you’ve actually worked with (even a single local/minikube cluster or one small EKS cluster) — specificity about what you did configure matters more than the raw cluster count.
Answer:
“Pods are running” only means the container process is alive — it doesn’t mean the Service is actually routing traffic to them. Work outward from the Service:
# Step 1: "Running" isn't the same as "Ready" — check READY column, not just STATUS
kubectl get pods -l app=myapp
# NAME READY STATUS RESTARTS
# myapp-abc123 0/1 Running 0 ← Running, but NOT Ready = excluded from Service
# Step 2: If Pods show 0/1 Ready, the readiness probe is failing — check why
kubectl describe pod myapp-abc123 | grep -A5 Readiness
kubectl logs myapp-abc123
# Common cause: app takes longer to warm up than initialDelaySeconds allows
# Step 3: Check the Service actually has endpoints
kubectl get endpoints myapp-svc
# If this is EMPTY, the Service's label selector doesn't match any Pod's labels
# (classic cause: a recent Deployment change altered labels without updating the Service)
# Step 4: Compare the Service selector to the Pod's actual labels
kubectl get svc myapp-svc -o jsonpath='{.spec.selector}'
kubectl get pods --show-labels
# Step 5: If using Ingress/ALB — check the Ingress Controller / Load Balancer
# Controller logs, and confirm target group health (on EKS with ALB Ingress)
kubectl logs -n kube-system deploy/aws-load-balancer-controller
# Step 6: Check for a PodDisruptionBudget or rollout blocking enough Ready replicas
kubectl get pdb
kubectl rollout status deployment/myapp
Most common root cause: Pods are Running but not Ready — a failing or too-aggressive readiness probe removes them from the Service’s endpoint list even though the container itself never crashed. The second most common cause is a label selector mismatch after a Deployment template change, leaving the Service pointing at zero matching Pods.
Answer:
All three manage a set of Pods, but they answer different questions about identity, placement, and lifecycle:
| Aspect | Deployment | StatefulSet | DaemonSet |
|---|---|---|---|
| Pod identity | Interchangeable — any replica can serve any request | Stable, unique (app-0, app-1, …) | One per matching node, tied to that node |
| Scaling | Any order, in parallel | Ordered — one at a time (0, 1, 2…) | Automatic — one Pod per node as nodes join/leave |
| Storage | Shared or ephemeral, no per-Pod guarantee | Dedicated PVC per Pod, follows the Pod on rescheduling | Usually none, or hostPath for node-local data |
| Networking | Single Service endpoint, any Pod | Stable per-Pod DNS (app-0.app.default.svc...) | Runs on the node’s own network typically |
| “How many replicas?” | You choose (2, 3, 10…) | You choose | Exactly one per (matching) node — not independently settable |
Practical use cases:
Deployment → Stateless web/API servers
Example: 5 replicas of a REST API behind a Service — any replica can
answer any request, and Kubernetes can kill/reschedule any of them freely.
StatefulSet → Databases, message brokers, anything needing stable identity + storage
Example: a 3-node PostgreSQL or Kafka cluster where postgres-0 is always
the same Pod with the same PVC — losing that identity would mean losing
which node is primary vs replica.
DaemonSet → Node-level infrastructure agents
Example: a log collector (Fluent Bit) or metrics agent (node_exporter)
that must run on every single node to collect that node's data — adding
a new node automatically gets its own copy, no manual scaling needed.
Quick interview framing: “I reach for Deployment by default. I only reach for StatefulSet when Pods genuinely can’t be interchangeable — usually because of local storage or ordering requirements. I use DaemonSet only for infrastructure that must exist on every node, not application workloads.”
Answer:
If the name/tag is confirmed correct, the problem is almost always authentication, network reachability, or registry-side state — not the manifest itself:
1. Registry authentication / pull permissions:
# For ECR: does the NODE's IAM role actually have ecr:GetAuthorizationToken +
# ecr:BatchGetImage + ecr:GetDownloadUrlForLayer permissions?
aws iam get-role-policy --role-name eks-node-role --policy-name ecr-pull
# For a private registry needing a Secret: is it actually attached to the Pod
# (or the ServiceAccount), and not just created and forgotten?
kubectl get pod <pod> -o jsonpath='{.spec.imagePullSecrets}'
kubectl get secret <regcred-name> -o yaml # confirm it actually exists, isn't expired
# Describe the pod — the Events section usually names the exact auth failure
kubectl describe pod <pod> | grep -A5 Events
# "unauthorized: authentication required" = credentials issue, not a typo
2. Network path from the NODE to the registry:
# Nodes in a private subnet need a route to the registry — for ECR specifically,
# either a NAT Gateway (internet route) OR an ECR VPC Interface Endpoint
aws ec2 describe-vpc-endpoints --filters Name=service-name,Values=com.amazonaws.us-east-1.ecr.api
# Test reachability directly from a node (via SSM, not SSH)
aws ssm start-session --target <node-instance-id>
curl -v https://<account-id>.dkr.ecr.us-east-1.amazonaws.com/v2/
3. Registry-side / image-side state:
# Does the image tag actually exist in the registry — was it deleted by a
# lifecycle policy, or never successfully pushed by a failed CI job?
aws ecr describe-images --repository-name my-app --image-ids imageTag=v1.2.3
# Rate limiting — public Docker Hub images can hit anonymous pull rate limits
# if many nodes pull simultaneously without authentication
kubectl describe pod <pod> | grep -i "toomanyrequests"
Interview framing: the fact that the image name/tag is confirmed correct should immediately redirect the answer away from the YAML and toward the three layers above the manifest — IAM/credentials, network path, and whether the image genuinely exists and is pullable at that moment, roughly in that order of likelihood.
Answer: This is an experience question — a strong answer names a specific strategy, ties it to a concrete reason, and shows you considered the trade-off, not just recited the options.
Example of how to structure it: “For our main API service, I used rolling updates with maxSurge: 1, maxUnavailable: 0 — it’s the default, requires no extra infrastructure, and zero-downtime was the main requirement, not instant rollback. Readiness probes were critical here — without a well-tuned readiness probe, Kubernetes will happily route traffic to a Pod that’s technically running but not yet ready to serve, which caused a handful of 502s before we tightened it.”
spec:
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
template:
spec:
containers:
- name: api
readinessProbe:
httpGet: { path: /ready, port: 8080 }
initialDelaySeconds: 10
periodSeconds: 5
“For a higher-risk service — our payments API — we used canary via Argo Rollouts instead: 10% traffic to the new version, automated analysis against error-rate and latency metrics from Prometheus for 10 minutes, auto-promote on pass or auto-rollback on fail. The extra tooling was worth it there specifically because a bad payments deploy is much more expensive than a bad deploy anywhere else in the system.”
apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 10m }
- analysis:
templates: [{ templateName: success-rate }]
What makes this a strong answer: naming different strategies for different services based on their actual risk profile — not applying the same strategy everywhere out of habit — is what separates real production experience from a memorized definition of blue-green/canary/rolling.
Answer:
OOMKilled is a cgroup-enforced hard limit, not something the Kubernetes scheduler or kubelet decides in real time — it happens at the Linux kernel level, and Kubernetes just reports what already occurred.
The actual sequence:
1. You set a memory LIMIT on the container:
resources:
limits:
memory: "512Mi"
2. The container runtime (containerd) creates a cgroup for that container
and sets memory.max (cgroups v2) to 512Mi — this is a KERNEL-enforced ceiling
3. The container's process(es) allocate memory normally... until the
cgroup's total usage would exceed 512Mi on the next allocation
4. At that exact moment, the Linux kernel's OOM killer fires — but
SCOPED TO THAT CGROUP, not the whole node (this is the key difference
from a node-level OOM, which is a different, worse scenario)
5. The kernel picks a process to kill within that cgroup (usually the
largest memory consumer, via oom_score_adj) and sends it SIGKILL —
immediate termination, no graceful shutdown, no chance to catch the signal
6. containerd detects the container exited due to OOM and reports it
to the kubelet, which sets:
Reason: OOMKilled
Exit Code: 137 (128 + SIGKILL's signal number 9)
7. The Pod's restart policy (default: Always) kicks in — kubelet
restarts the container, which is why OOMKilled often shows up
together with CrashLoopBackOff if the memory pressure recurs quickly
# Confirm it was OOMKilled specifically (not a normal crash)
kubectl describe pod <pod> | grep -A3 "Last State"
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137
# Check what the limit actually was vs. what was likely being used
kubectl get pod <pod> -o jsonpath='{.spec.containers[0].resources}'
Important distinction for an interview: this is different from a node-level OOM (where the node itself runs out of memory across ALL pods, and the kubelet’s node-pressure eviction — or the kernel OOM killer at the node scope — starts killing pods based on QoS class, lowest-priority BestEffort pods first). A single container hitting its own memory.max only ever kills processes inside that container’s cgroup — it doesn’t affect other Pods on the node at all.
Fix: raise the memory limit if the usage is legitimate, fix a memory leak if it’s not, or set memory requests accurately so the scheduler places the Pod on a node that actually has enough headroom in the first place.
Answer:
Every Service gets a DNS record, and the namespace determines how much of the name you need to specify — same-namespace lookups can use a short name, cross-namespace lookups need at least the namespace included:
Full form (works from ANY namespace):
backend-svc.orders.svc.cluster.local
From WITHIN the same namespace (orders), the short form also resolves:
backend-svc ← works, thanks to the Pod's search domain
From a DIFFERENT namespace (e.g., a Pod in "frontend" calling "orders"):
backend-svc ← FAILS — resolves to nothing in this namespace
backend-svc.orders ← works — namespace included
backend-svc.orders.svc.cluster.local ← always works, fully qualified
Why the short name only works locally: every Pod’s /etc/resolv.conf has a search list that includes its own namespace (e.g., orders.svc.cluster.local), so an unqualified name gets that suffix appended automatically. A Pod in a different namespace doesn’t have that suffix in its search path, so the same short name resolves to nothing.
# Check a Pod's actual search domains
kubectl exec -it mypod -n frontend -- cat /etc/resolv.conf
# search frontend.svc.cluster.local svc.cluster.local cluster.local ...
What can break cross-namespace DNS specifically:
# 1. A NetworkPolicy blocking egress to kube-dns/CoreDNS on port 53
# — extremely common cause once teams start locking down NetworkPolicies
kubectl get networkpolicy -n frontend -o yaml
# Fix: always allow egress to the kube-system namespace on port 53 (UDP+TCP)
# 2. CoreDNS pods themselves unhealthy or under-resourced
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# 3. Custom Corefile misconfiguration (stub domains, forward rules edited
# incorrectly) breaking resolution for specific zones
kubectl get configmap coredns -n kube-system -o yaml
# 4. The TARGET service genuinely doesn't exist in that namespace, or its
# selector matches zero Pods (Service exists, but has no Endpoints —
# DNS resolves fine, but connecting still fails)
kubectl get endpoints backend-svc -n orders
# 5. dnsPolicy overridden on the Pod itself (rare, but breaks everything)
kubectl get pod mypod -o jsonpath='{.spec.dnsPolicy}'
# Should be "ClusterFirst" (default) for normal in-cluster resolution
Most common root cause in practice: a NetworkPolicy that locks down egress traffic for security but forgets to explicitly allow DNS (port 53) to kube-system — the Pod can’t even resolve the name to attempt a connection, which often gets misdiagnosed as “the other service is down” when it’s actually a DNS-level block.
📌 Quick Reference
Kubectl Cheat Sheet
# Context management
kubectl config get-contexts
kubectl config use-context <context>
# Resource management
kubectl get all -n <namespace>
kubectl apply -f <file>
kubectl delete -f <file>
kubectl edit <resource> <name>
# Debugging
kubectl describe <resource> <name>
kubectl logs <pod> -c <container> -f --previous
kubectl exec -it <pod> -- /bin/sh
kubectl top pods --sort-by=memory
# Port forwarding
kubectl port-forward svc/<service> 8080:80
# Rollout management
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name> --to-revision=2
Happy interviewing! 🚀 This guide covers the most commonly asked Kubernetes and EKS interview questions across all experience levels.
Add More Questions to This Guide
Know a question that should be here? Share it and help the community!
Open Google Form