Kubernetes (K3s)

Cluster Overview

The homelab runs a highly available K3s cluster with multiple master and worker nodes.

Cluster Specifications

• Distribution: K3s (Lightweight Kubernetes)
• Version: v1.28.5+k3s1
• Architecture: Multi-master HA setup
• CNI: Flannel (default)
• Ingress: NGINX Ingress Controller
• Load Balancer: MetalLB

Node Configuration

Control Plane Nodes

Worker Nodes

Cluster Access

API Endpoint

• URL: https://192.168.1.210:6443
• Authentication: Token-based

kubectl Configuration

# Export kubeconfig
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Or copy to local machine
scp ubuntu@192.168.1.210:/etc/rancher/k3s/k3s.yaml ~/.kube/k3s-config

# Use with kubectl
kubectl --kubeconfig ~/.kube/k3s-config get nodes

Storage Configuration

Storage Classes

Persistent Volumes

Major persistent volumes in use:

• Gitea data (10GB iSCSI)
• Bitcoin Core blockchain (10TB iSCSI)
• Home Assistant data (100GB iSCSI)
• Prometheus metrics (50GB iSCSI)
• Media storage (NFS shares)

Networking

Service Mesh

• Direct pod-to-pod communication
• No additional service mesh (Istio/Linkerd) currently deployed

Load Balancing

• MetalLB provides LoadBalancer services
• IP Pool: 192.168.1.150-192.168.1.199

Ingress Configuration

# Example ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - app.home.dodwell.us
    secretName: app-tls
  rules:
  - host: app.home.dodwell.us
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

GitOps with ArgoCD

ArgoCD Configuration

• URL: https://argocd.home.dodwell.us
• Sync Policy: Automatic with pruning
• Repository: Gitea private repo

Application Structure

k3s/
├── apps/           # Application manifests
├── bootstrap/      # ArgoCD app-of-apps
└── envs/local/     # Environment-specific apps

Security

RBAC

• Service accounts for each application
• Minimal permissions principle
• No default admin access

Network Policies

• Namespace isolation by default
• Explicit ingress/egress rules
• Zero-trust networking approach

Secrets Management

• SOPS for GitOps secrets
• Vault for dynamic secrets
• Encrypted at rest

Monitoring

Metrics

• Prometheus scrapes all services
• Node Exporter on each node
• kube-state-metrics for cluster metrics

Key Metrics to Monitor

• Node resource utilization
• Pod resource requests/limits
• PVC usage
• API server latency
• etcd performance

Common Operations

Node Management

# Drain node for maintenance
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Uncordon node after maintenance
kubectl uncordon <node-name>

# Get node details
kubectl describe node <node-name>

Troubleshooting

# Check pod issues
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>

# Check events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

# Resource usage
kubectl top nodes
kubectl top pods -n <namespace>

Backup and Restore

• etcd snapshots taken daily
• Persistent volumes backed up to NAS
• GitOps provides declarative recovery