Troubleshooting Guide

Comprehensive troubleshooting procedures for common issues

General Troubleshooting Methodology

Problem Resolution Process

  1. Identify the Problem: Gather symptoms and error messages
  2. Check Monitoring: Review Grafana dashboards and Prometheus alerts
  3. Isolate the Issue: Determine affected components and scope
  4. Apply Solution: Use documented procedures or escalate
  5. Verify Fix: Confirm resolution and monitor for stability
  6. Document: Update runbooks with lessons learned

Diagnostic Commands

System Status Overview:

# Check all pod status
kubectl get pods -A | grep -v Running

# Check node status
kubectl get nodes -o wide

# Check cluster events
kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp | tail -20

# Check resource usage
kubectl top nodes
kubectl top pods -A

Infrastructure Issues

Proxmox Problems

VM Not Starting:

# Check VM status
qm status VMID

# Check VM configuration
qm config VMID

# Check storage availability
pvesm status

# Check cluster status
pvecm status

# Common fixes
qm start VMID
qm reset VMID
qm unlock VMID

Storage Issues:

# Check storage status
pvesm status

# Check ZFS pool health
zpool status

# Check disk space
df -h
zfs list -o space

# Check iSCSI targets
iscsiadm -m session

Network Connectivity:

# Check bridge status
ip link show

# Check firewall rules
iptables -L

# Check DHCP leases
cat /var/lib/dhcp/dhcpd.leases

# Test connectivity
ping -c 4 192.168.1.1
traceroute 192.168.1.212

K3s Cluster Issues

Node Not Ready:

# Check node status
kubectl describe node k3s-worker-01

# Check node logs
journalctl -u k3s -f

# Check kubelet status
systemctl status k3s

# Common fixes
systemctl restart k3s
systemctl restart k3s-agent

Network Problems:

# Check CNI status
kubectl get pods -n kube-system | grep flannel

# Check coredns
kubectl get pods -n kube-system | grep coredns

# Test DNS resolution
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

# Check service endpoints
kubectl get endpoints -A

Storage Problems:

# Check storage classes
kubectl get storageclass

# Check persistent volumes
kubectl get pv

# Check volume mounts
kubectl describe pod POD_NAME | grep -A 10 Volumes

# Check iSCSI connections
iscsiadm -m session

Application Issues

Pod Failures

Pod Stuck in Pending:

# Check pod events
kubectl describe pod POD_NAME

# Check node resources
kubectl top nodes

# Check resource quotas
kubectl describe quota -n NAMESPACE

# Check node affinity/anti-affinity
kubectl get pod POD_NAME -o yaml | grep -A 20 affinity

Pod Stuck in Init:

# Check init container logs
kubectl logs POD_NAME -c INIT_CONTAINER_NAME

# Check init container status
kubectl describe pod POD_NAME

# Common issues: ConfigMap/Secret not available
kubectl get configmap -n NAMESPACE
kubectl get secret -n NAMESPACE

Pod CrashLoopBackOff:

# Check pod logs
kubectl logs POD_NAME --previous

# Check liveness/readiness probes
kubectl describe pod POD_NAME | grep -A 5 Liveness
kubectl describe pod POD_NAME | grep -A 5 Readiness

# Check resource limits
kubectl describe pod POD_NAME | grep -A 5 Limits

Service Discovery Issues

Service Not Reachable:

# Check service endpoints
kubectl get endpoints SERVICE_NAME

# Check service configuration
kubectl describe service SERVICE_NAME

# Test from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -qO- http://SERVICE_NAME:PORT

# Check network policies
kubectl get networkpolicy -n NAMESPACE

DNS Resolution Problems:

# Check CoreDNS pods
kubectl get pods -n kube-system | grep coredns

# Check CoreDNS logs
kubectl logs -n kube-system deployment/coredns

# Test DNS resolution
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup SERVICE_NAME.NAMESPACE.svc.cluster.local

# Check DNS configuration
kubectl get configmap coredns -n kube-system -o yaml

ArgoCD Issues

Sync Problems

Application Out of Sync:

# Check application status
argocd app get APPLICATION_NAME

# Check sync policy
argocd app get APPLICATION_NAME -o yaml | grep -A 10 syncPolicy

# Manual sync
argocd app sync APPLICATION_NAME --force

# Check sync history
argocd app history APPLICATION_NAME

Repository Access Issues:

# Check repository connection
argocd repo list

# Test repository access
argocd repo get https://gitea.home.dodwell.us/exodus/homelab.git

# Check credentials
kubectl get secret -n argocd | grep repo

Resource Hooks Failing:

# Check hook status
kubectl get job -n argocd | grep hook

# Check hook logs
kubectl logs job/HOOK_JOB_NAME -n argocd

# Delete failed hooks
kubectl delete job HOOK_JOB_NAME -n argocd

Monitoring and Alerting

Prometheus Issues

Metrics Not Scraped:

# Check Prometheus targets
curl http://prometheus.home.dodwell.us/api/v1/targets

# Check ServiceMonitor
kubectl get servicemonitor -A

# Check service labels
kubectl get service SERVICE_NAME -o yaml | grep -A 5 labels

# Check Prometheus configuration
kubectl get configmap prometheus-config -o yaml

High Memory Usage:

# Check Prometheus memory usage
kubectl top pod prometheus-xxx -n prometheus

# Check retention settings
kubectl get prometheus -o yaml | grep retention

# Check query complexity
curl http://prometheus.home.dodwell.us/api/v1/query_range?query=up

# Restart Prometheus
kubectl rollout restart deployment prometheus -n prometheus

Grafana Issues

Dashboard Not Loading:

# Check Grafana logs
kubectl logs deployment/grafana -n grafana

# Check data source connectivity
curl -H "Authorization: Bearer $TOKEN" http://grafana.home.dodwell.us/api/datasources

# Check dashboard configuration
kubectl get configmap grafana-dashboards -o yaml

# Test query directly
curl -X POST http://prometheus.home.dodwell.us/api/v1/query -d 'query=up'

Authentication Problems:

# Check Authelia logs
kubectl logs deployment/authelia -n authelia

# Check OIDC configuration
kubectl get configmap authelia-config -o yaml | grep -A 20 oidc

# Test authentication endpoint
curl -v https://auth.home.dodwell.us/api/health

Storage Issues

Persistent Volume Problems

PVC Stuck in Pending:

# Check PVC status
kubectl describe pvc PVC_NAME

# Check storage class
kubectl get storageclass

# Check available PVs
kubectl get pv | grep Available

# Check node capacity
kubectl describe node NODE_NAME | grep -A 5 Capacity

Mount Failures:

# Check mount events
kubectl describe pod POD_NAME | grep -A 5 Mount

# Check iSCSI connection
iscsiadm -m session | grep TARGET

# Check filesystem
kubectl exec POD_NAME -- df -h

# Check permissions
kubectl exec POD_NAME -- ls -la /data

TrueNAS Issues

iSCSI Target Not Available:

# Check target status on TrueNAS
sudo systemctl status istgt

# Check target configuration
sudo cat /etc/istgt/istgt.conf

# Test from initiator
iscsiadm -m discovery -t st -p 192.168.1.6:3260

# Check network connectivity
ping 192.168.1.6
telnet 192.168.1.6 3260

ZFS Pool Issues:

# Check pool status
zpool status -v

# Check for errors
zpool events

# Run scrub
zpool scrub tank

# Check capacity
zfs list -o space

Network Issues

Ingress Problems

Service Not Accessible Externally:

# Check ingress controller
kubectl get pods -n ingress-nginx

# Check ingress configuration
kubectl describe ingress INGRESS_NAME

# Check certificate
kubectl get certificate

# Test from inside cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- wget -qO- http://SERVICE_NAME

SSL Certificate Issues:

# Check cert-manager
kubectl get pods -n cert-manager

# Check certificate request
kubectl get certificaterequest

# Check certificate status
kubectl describe certificate CERT_NAME

# Check Let's Encrypt logs
kubectl logs -n cert-manager deployment/cert-manager

Load Balancer Issues

HAProxy Not Responding:

# Check HAProxy status
sudo systemctl status haproxy

# Check HAProxy configuration
sudo haproxy -f /etc/haproxy/haproxy.cfg -c

# Check backend health
echo "show stat" | sudo socat unix-connect:/var/run/haproxy.sock stdio

# Check logs
sudo journalctl -u haproxy -f

Keepalived Issues:

# Check keepalived status
sudo systemctl status keepalived

# Check VIP assignment
ip addr show | grep 192.168.1.200

# Check priority
sudo journalctl -u keepalived | grep priority

# Test failover
sudo systemctl stop keepalived

Security Issues

Authentication Failures

Authelia Not Working:

# Check Authelia logs
kubectl logs deployment/authelia -n authelia

# Check LDAP connectivity
kubectl exec -it deployment/authelia -n authelia -- ldapsearch -x -H ldaps://openldap.openldap.svc.cluster.local:636

# Check Redis connectivity
kubectl exec -it deployment/authelia -n authelia -- redis-cli -h redis ping

# Test authentication
curl -v https://auth.home.dodwell.us/api/health

LDAP Issues:

# Check OpenLDAP logs
kubectl logs deployment/openldap -n openldap

# Test LDAP bind
ldapsearch -x -H ldaps://openldap.home.dodwell.us -D "cn=admin,dc=home,dc=dodwell,dc=us" -W

# Check certificate
openssl s_client -connect openldap.home.dodwell.us:636 -showcerts

Certificate Problems

Certificate Expired:

# Check certificate expiry
kubectl get certificate

# Check certificate details
kubectl describe certificate CERT_NAME

# Force renewal
kubectl delete certificaterequest CERT_REQUEST_NAME

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

Performance Issues

High CPU Usage

Identify High CPU Pods:

# Check pod CPU usage
kubectl top pods -A --sort-by=cpu

# Check node CPU usage
kubectl top nodes

# Check specific pod metrics
kubectl describe pod POD_NAME | grep -A 5 Limits

# Scale deployment
kubectl scale deployment DEPLOYMENT_NAME --replicas=3

Memory Issues

Out of Memory:

# Check memory usage
kubectl top pods -A --sort-by=memory

# Check memory limits
kubectl describe pod POD_NAME | grep -A 5 Limits

# Check node memory
free -h
kubectl describe node NODE_NAME | grep -A 5 Capacity

# Adjust memory limits
kubectl patch deployment DEPLOYMENT_NAME -p '{"spec":{"template":{"spec":{"containers":[{"name":"CONTAINER_NAME","resources":{"limits":{"memory":"1Gi"}}}]}}}}'

Disk Space Issues

Low Disk Space:

# Check disk usage
df -h

# Check pod disk usage
kubectl exec POD_NAME -- df -h

# Check log sizes
du -sh /var/log/
journalctl --disk-usage

# Clean up
docker system prune -a
journalctl --vacuum-time=7d

Emergency Procedures

Cluster Down

Complete Cluster Failure:

# Check all nodes
kubectl get nodes

# Check etcd
kubectl get pods -n kube-system | grep etcd

# Restart k3s on all nodes
sudo systemctl restart k3s

# Check cluster health
kubectl get componentstatus

Service Outage

Critical Service Down:

# Check service status
kubectl get deployment SERVICE_NAME

# Check recent events
kubectl get events --sort-by=.metadata.creationTimestamp | tail -20

# Restart service
kubectl rollout restart deployment SERVICE_NAME

# Check logs
kubectl logs deployment/SERVICE_NAME -f

Security Incident

Suspected Breach:

# Check authentication logs
kubectl logs deployment/authelia -n authelia | grep -i fail

# Check access logs
kubectl logs deployment/nginx-ingress-controller -n ingress-nginx | grep -i 401

# Block IP address
# Add to firewall rules on router
ssh admin@192.168.1.1 '/ip firewall address-list add list=blocked_ips address=X.X.X.X'

# Rotate credentials
kubectl delete secret SECRET_NAME
kubectl create secret generic SECRET_NAME --from-literal=key=new_value

Escalation Procedures

When to Escalate

  1. Data Loss: Potential or confirmed data loss
  2. Security Breach: Suspected unauthorized access
  3. Extended Outage: Service down for >30 minutes
  4. Infrastructure Failure: Multiple node failures
  5. Unknown Issues: Problems not covered in documentation

Escalation Steps

  1. Gather Information: Collect logs, metrics, and symptoms
  2. Document: Record timeline and actions taken
  3. Notify: Alert relevant stakeholders
  4. Provide Access: Ensure responders can access systems
  5. Support: Assist with resolution efforts

Emergency Contacts

  • Infrastructure: Primary administrator
  • Security: Security team contact
  • Vendor Support: For critical hardware/software issues

Documentation Updates

After resolving issues:

  1. Update Runbooks: Add new procedures
  2. Improve Monitoring: Add alerts for early detection
  3. Enhance Automation: Automate repetitive fixes
  4. Training: Share knowledge with team
  5. Review: Conduct post-incident review

Troubleshooting Tools

Useful Commands

kubectl shortcuts:

# Create aliases
alias k='kubectl'
alias kgp='kubectl get pods'
alias kgs='kubectl get svc'
alias kgd='kubectl get deploy'

# Useful functions
function klogs() { kubectl logs -f deployment/$1 -n $2; }
function kexec() { kubectl exec -it $1 -- /bin/bash; }

Monitoring shortcuts:

# Check all unhealthy pods
kubectl get pods -A | grep -v Running | grep -v Completed

# Check resource usage
kubectl top pods -A --sort-by=memory | head -20

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp -A | tail -20

External Tools

  • k9s: Terminal-based Kubernetes dashboard
  • kubectx/kubens: Context and namespace switching
  • stern: Multi-pod log tailing
  • kustomize: Configuration management
  • argocd cli: GitOps management
  • helm: Package management

Prevention

Proactive Measures

  1. Regular Updates: Keep systems patched
  2. Monitoring: Comprehensive alerting
  3. Backup Testing: Regular restore tests
  4. Documentation: Keep procedures current
  5. Training: Regular knowledge sharing
  6. Automation: Reduce manual errors

Health Checks

Daily Checks:

#!/bin/bash
# Daily health check script
echo "=== Cluster Health Check ==="
kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed
kubectl top nodes
kubectl get pv | grep -v Bound

Weekly Checks:

  • Certificate expiry review
  • Backup verification
  • Security patch status
  • Capacity planning review
  • Performance trend analysis

This troubleshooting guide provides a comprehensive reference for resolving common issues in the homelab environment. Regular practice with these procedures will improve response times and reduce service disruption.