Production Secrets & Configuration

In Lesson 5, you deployed your Task API to DOKS with a basic configuration. But production services need more than hardcoded values. Your AI agent needs database credentials, API keys for external services, and environment-specific settings that change between staging and production.

Storing these directly in Deployment manifests creates security risks and operational headaches. Every configuration change requires a new container image or manifest edit. Worse, secrets end up in version control where anyone with repository access can read them.

Kubernetes solves this with two primitives: Secrets for sensitive data and ConfigMaps for non-sensitive configuration. This lesson teaches you to use both effectively, including the critical image pull secret that authenticates your cluster to pull images from GitHub Container Registry.

What You'll Learn

This lesson covers 5 concepts organized into three groups:

Concept Group	Concepts	Focus
Secret Fundamentals	1-2	Creating secrets, types of secrets (Opaque, docker-registry)
Configuration Management	3-4	ConfigMaps, configuration hierarchy patterns
Verification	5	Mounting secrets, verifying injection, troubleshooting

Prerequisites:

Lesson 5 completed (Task API deployed to DOKS)
kubectl configured and connected to your cluster
GitHub personal access token with read:packages scope

Time Estimate: 25 minutes

Concepts: 15 minutes
Exercises: 8 minutes
Try With AI: 2 minutes

Concept 1: Kubernetes Secrets

A Kubernetes Secret stores sensitive data like passwords, tokens, and certificates. Unlike ConfigMaps, Secrets are base64-encoded (not encrypted by default, but protected from casual viewing) and can be restricted with RBAC.

Creating a Secret from Literal Values

kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=prod-super-secret-password \
  --from-literal=OPENAI_API_KEY=sk-proj-your-api-key-here

Output:

secret/task-api-secrets created

Viewing Secret Structure (Not Values)

kubectl get secret task-api-secrets -o yaml

Output:

apiVersion: v1
kind: Secret
metadata:
  name: task-api-secrets
  namespace: default
type: Opaque
data:
  DATABASE_PASSWORD: cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=
  OPENAI_API_KEY: c2stcHJvai15b3VyLWFwaS1rZXktaGVyZQ==

The values are base64-encoded, not encrypted. Anyone with kubectl get secret access can decode them:

echo "cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=" | base64 -d

Output:

prod-super-secret-password

Security Note: Base64 is encoding, not encryption. For true at-rest encryption, enable Kubernetes secrets encryption or use a secrets manager like HashiCorp Vault or cloud provider secrets (AWS Secrets Manager, GCP Secret Manager).

Concept 2: Image Pull Secrets for Private Registries

Your Task API container lives in GitHub Container Registry (GHCR), which requires authentication. Without an image pull secret, Kubernetes cannot pull your private images.

Creating the Image Pull Secret

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN \
  --docker-email=your-email@example.com

Output:

secret/ghcr-secret created

Where:

$GITHUB_USERNAME is your GitHub username
$GITHUB_TOKEN is a personal access token with read:packages scope

Verifying the Secret

kubectl get secret ghcr-secret -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

Output:

{
  "auths": {
    "ghcr.io": {
      "username": "your-username",
      "password": "ghp_xxxxxxxxxxxxxxxxxxxx",
      "email": "your-email@example.com",
      "auth": "eW91ci11c2VybmFtZTpnaHBfeHh4eHh4eHh4eHh4eHh4eHh4eHh4"
    }
  }
}

Referencing in Deployment

Add imagePullSecrets to your Deployment spec:

# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest

Without this reference, pod creation fails with ImagePullBackOff:

kubectl describe pod task-api-xxxxx | grep -A5 "Events:"

Output (without secret):

Events:
  Warning  Failed   12s  kubelet  Failed to pull image "ghcr.io/your-username/task-api:latest":
           rpc error: code = Unknown desc = failed to pull and unpack image:
           failed to resolve reference: unexpected status code 401 Unauthorized

Concept 3: ConfigMaps for Non-Sensitive Configuration

ConfigMaps store configuration data that isn't sensitive: feature flags, service URLs, logging levels. Unlike Secrets, ConfigMaps appear in plain text and are easier to inspect.

Creating a ConfigMap

kubectl create configmap task-api-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=API_BASE_URL=https://api.example.com \
  --from-literal=ENABLE_METRICS=true

Output:

secret/task-api-config created

Creating ConfigMap from a File

For complex configuration, use a file:

# config.env
LOG_LEVEL=info
API_BASE_URL=https://api.example.com
ENABLE_METRICS=true
MAX_CONNECTIONS=100
REQUEST_TIMEOUT_MS=30000

kubectl create configmap task-api-config --from-env-file=config.env

Output:

configmap/task-api-config created

Viewing ConfigMap Contents

kubectl get configmap task-api-config -o yaml

Output:

apiVersion: v1
kind: ConfigMap
metadata:
  name: task-api-config
  namespace: default
data:
  API_BASE_URL: https://api.example.com
  ENABLE_METRICS: "true"
  LOG_LEVEL: info
  MAX_CONNECTIONS: "100"
  REQUEST_TIMEOUT_MS: "30000"

Concept 4: Configuration Hierarchy Pattern

Production applications need layered configuration: defaults that work everywhere, environment overrides for staging vs production, and secrets that are never committed to version control.

The Three-Layer Pattern

┌─────────────────────────────────────┐
│           Application               │
├─────────────────────────────────────┤
│  Layer 3: Secrets (highest priority)│  ← DATABASE_PASSWORD, API_KEY
│  kubectl create secret              │
├─────────────────────────────────────┤
│  Layer 2: ConfigMap (environment)   │  ← LOG_LEVEL, API_URL
│  kubectl create configmap           │
├─────────────────────────────────────┤
│  Layer 1: Container defaults        │  ← Baked into image
│  Dockerfile ENV statements          │
└─────────────────────────────────────┘

Environment variables from higher layers override lower layers.

Implementing in Deployment

# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest

          # Layer 1: Defaults (optional, usually in Dockerfile)
          env:
            - name: NODE_ENV
              value: "production"

          # Layer 2: ConfigMap (all non-sensitive config)
          envFrom:
            - configMapRef:
                name: task-api-config

          # Layer 3: Secrets (sensitive values, highest priority)
            - secretRef:
                name: task-api-secrets

Output (kubectl describe pod):

Environment Variables from:
  task-api-config   ConfigMap  Optional: false
  task-api-secrets  Secret     Optional: false
Environment:
  NODE_ENV:  production

Why This Order Matters

If LOG_LEVEL appears in both ConfigMap and as a hardcoded env var, the ConfigMap value wins because envFrom is processed after env. Secrets in envFrom are processed last, so they override everything.

Concept 5: Verifying Secret and ConfigMap Injection

Deploying configuration is only half the battle. You must verify that containers receive the expected values.

Check Pod Environment Variables

kubectl exec -it deployment/task-api -- env | grep -E "(DATABASE|OPENAI|LOG_LEVEL)"

Output:

DATABASE_PASSWORD=prod-super-secret-password
OPENAI_API_KEY=sk-proj-your-api-key-here
LOG_LEVEL=info

Verify All Environment Variables

kubectl exec -it deployment/task-api -- printenv | sort

Output:

API_BASE_URL=https://api.example.com
DATABASE_PASSWORD=prod-super-secret-password
ENABLE_METRICS=true
LOG_LEVEL=info
MAX_CONNECTIONS=100
NODE_ENV=production
OPENAI_API_KEY=sk-proj-your-api-key-here
REQUEST_TIMEOUT_MS=30000
...

Troubleshooting Missing Variables

If expected variables are missing, check:

ConfigMap/Secret exists:

kubectl get configmap task-api-config
kubectl get secret task-api-secrets

Deployment references are correct:

kubectl get deployment task-api -o yaml | grep -A10 envFrom

Pod events for errors:

kubectl describe pod -l app=task-api | grep -A20 Events

Output (missing ConfigMap):

Events:
  Warning  Failed  12s  kubelet  Error: configmap "task-api-config" not found

Common Mistakes

1. Forgetting imagePullSecrets for Private Registries

# WRONG - Missing imagePullSecrets
spec:
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest  # Will fail with 401

Fix: Always include imagePullSecrets when using private registries:

spec:
  imagePullSecrets:
    - name: ghcr-secret
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest

2. Putting Secrets in ConfigMaps

# WRONG - API keys in ConfigMap (visible to anyone)
kubectl create configmap task-api-config \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx

Fix: Use Secrets for sensitive data:

kubectl create secret generic task-api-secrets \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx

3. Using Wrong Secret Type for Docker Registry

# WRONG - Opaque secret won't work for imagePullSecrets
kubectl create secret generic ghcr-secret \
  --from-literal=password=ghp_xxxxx

Fix: Use docker-registry type:

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN

4. Forgetting to Restart Pods After Secret Update

Updating a Secret doesn't automatically restart pods:

kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=new-password \
  --dry-run=client -o yaml | kubectl apply -f -
# Pod still has OLD password!

Fix: Restart the deployment:

kubectl rollout restart deployment/task-api

Exercise 6.1: Create Application Secrets

Create secrets for your Task API with database credentials and an API key.

Instructions:

Create the secret:

kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=my-secure-password \
  --from-literal=API_KEY=your-test-api-key

Verify creation:
```
kubectl get secret task-api-secrets
```

Expected output: Secret listed with Opaque type and 2 data items.

Exercise 6.2: Create Image Pull Secret for GHCR

Set up authentication for GitHub Container Registry.

Instructions:

Create a GitHub personal access token with read:packages scope at https://github.com/settings/tokens

Create the secret (replace with your values):

export GITHUB_USERNAME=your-username
export GITHUB_TOKEN=ghp_your_token_here

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN

Verify:

kubectl get secret ghcr-secret -o jsonpath='{.type}'

Expected output: kubernetes.io/dockerconfigjson

Exercise 6.3: Create ConfigMap and Deploy with Full Configuration

Create a ConfigMap and update your deployment to use the three-layer configuration pattern.

Instructions:

Create ConfigMap:

kubectl create configmap task-api-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=ENABLE_METRICS=true

Apply updated deployment with all references (save as task-api-full.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: task-api
  template:
    metadata:
      labels:
        app: task-api
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: task-api-config
            - secretRef:
                name: task-api-secrets

Apply and verify:

kubectl apply -f task-api-full.yaml
kubectl exec -it deployment/task-api -- printenv | grep -E "(LOG_LEVEL|DATABASE)"

Expected output: Both ConfigMap and Secret values visible in pod environment.

Try With AI

Part 1: Initial Request

Your Task API needs additional secrets for a new feature: Redis connection string and a webhook signing secret. Ask AI for help:

"I need to add Redis and webhook secrets to my Kubernetes deployment. Create kubectl commands to add REDIS_URL and WEBHOOK_SECRET to my existing task-api-secrets."

What you're learning: How to extend existing secrets without recreating them from scratch.

Part 2: Critical Evaluation

Review AI's response. Ask yourself:

Did it use kubectl create secret with --dry-run=client -o yaml | kubectl apply -f - to update existing secrets?
Or did it suggest deleting and recreating (which causes downtime)?
Does the approach preserve existing secret values?

Part 3: Configuration Hierarchy

Ask AI about a more advanced pattern:

"I want to use external-secrets operator to pull secrets from AWS Secrets Manager instead of creating them with kubectl. Is this better than what I'm doing now?"

What you're learning: The difference between simple kubectl-based secrets (good for learning) and enterprise patterns (external-secrets operator).

Part 4: Validation

Verify the secrets work by checking your pod:

kubectl exec -it deployment/task-api -- env | grep REDIS

If the variable appears with the expected value, your secret management is working.

Reflect on Your Skill

You built a multi-cloud-deployer skill in Lesson 0. Test and improve it based on what you learned.

Test Your Skill

Using my multi-cloud-deployer skill, set up secrets for a production deployment.
Does my skill include creating application secrets and image pull secrets?

Identify Gaps

Ask yourself:

Does my skill cover creating Kubernetes secrets with kubectl?
Does it include image pull secrets for private registries like GHCR?
Does it demonstrate the three-layer configuration pattern?
Does it include verification steps using kubectl exec?

Improve Your Skill

If you found gaps:

My multi-cloud-deployer skill is missing secrets management patterns.
Update it to include:
- Creating application secrets (kubectl create secret generic)
- Creating image pull secrets (kubectl create secret docker-registry)
- ConfigMap creation for non-sensitive configuration
- The three-layer hierarchy (defaults -> configmap -> secrets)
- Verification with kubectl exec to check environment variables
- Common mistakes like forgetting imagePullSecrets reference

Continue to Personal Cloud Lab - Hetzner + K3s →

What You'll Learn​

Concept 1: Kubernetes Secrets​

Creating a Secret from Literal Values​

Viewing Secret Structure (Not Values)​

Concept 2: Image Pull Secrets for Private Registries​

Creating the Image Pull Secret​

Verifying the Secret​

Referencing in Deployment​

Concept 3: ConfigMaps for Non-Sensitive Configuration​

Creating a ConfigMap​

Creating ConfigMap from a File​

Viewing ConfigMap Contents​

Concept 4: Configuration Hierarchy Pattern​

The Three-Layer Pattern​

Implementing in Deployment​

Why This Order Matters​

Concept 5: Verifying Secret and ConfigMap Injection​

Check Pod Environment Variables​

Verify All Environment Variables​

Troubleshooting Missing Variables​

Common Mistakes​

Exercise 6.1: Create Application Secrets​

Exercise 6.2: Create Image Pull Secret for GHCR​

Exercise 6.3: Create ConfigMap and Deploy with Full Configuration​

Try With AI​

Reflect on Your Skill​

Test Your Skill​

Identify Gaps​

Improve Your Skill​

What You'll Learn

Concept 1: Kubernetes Secrets

Creating a Secret from Literal Values

Viewing Secret Structure (Not Values)

Concept 2: Image Pull Secrets for Private Registries

Creating the Image Pull Secret

Verifying the Secret

Referencing in Deployment

Concept 3: ConfigMaps for Non-Sensitive Configuration

Creating a ConfigMap

Creating ConfigMap from a File

Viewing ConfigMap Contents

Concept 4: Configuration Hierarchy Pattern

The Three-Layer Pattern

Implementing in Deployment

Why This Order Matters

Concept 5: Verifying Secret and ConfigMap Injection

Check Pod Environment Variables

Verify All Environment Variables

Troubleshooting Missing Variables

Common Mistakes

Exercise 6.1: Create Application Secrets

Exercise 6.2: Create Image Pull Secret for GHCR

Exercise 6.3: Create ConfigMap and Deploy with Full Configuration

Try With AI

Reflect on Your Skill

Test Your Skill

Identify Gaps

Improve Your Skill