Monitoring Stack

Overview

The homelab uses a complete Prometheus-based monitoring stack for infrastructure and application observability.

Components

Prometheus

• URL: https://prometheus.home.dodwell.us
• Purpose: Metrics collection and storage
• Retention: 30 days
• Storage: 50GB iSCSI volume
• Features:
  • Service discovery
  • Alert rules
  • HA configuration
  • Remote write capability

Grafana

• URL: https://grafana.home.dodwell.us
• Purpose: Metrics visualization
• Features:
  • Pre-built dashboards
  • Custom dashboards
  • Alert visualization
  • Multi-datasource support

Exporters

Node Exporter

• Deployment: Every node (Proxmox & K8s)
• Metrics: System-level metrics
• Port: 9100

Blackbox Exporter

• Purpose: Endpoint monitoring
• Checks:
  • HTTP/HTTPS endpoints
  • TCP services
  • ICMP (ping)
  • DNS queries

Custom Exporters

• AdGuard Exporter: DNS query metrics
• Bitcoin Exporter: Blockchain metrics
• Squid Exporter: Proxy metrics

Dashboards

Infrastructure Dashboards

1. Proxmox Overview

   • Node health
   • VM resource usage
   • Storage utilization
   • Network throughput

2. Kubernetes Cluster

   • Node resources
   • Pod metrics
   • API server performance
   • etcd health

3. Network Overview

   • Bandwidth usage
   • DNS query rates
   • Proxy statistics
   • Ingress metrics

Application Dashboards

1. Home Assistant

   • Entity counts
   • Automation performance
   • Database size
   • API latency

2. Media Stack

   • Download queues
   • Storage usage
   • API response times
   • Error rates

3. Security Services

   • Authentication attempts
   • Certificate expiry
   • Vault seal status
   • Security events

Alert Rules

Critical Alerts

# Node down
- alert: NodeDown
  expr: up{job="node"} == 0
  for: 2m
  severity: critical
  
# Disk space
- alert: DiskSpaceLow
  expr: node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes < 0.1
  for: 5m
  severity: warning

# High CPU
- alert: HighCPUUsage
  expr: (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)) > 0.85
  for: 10m
  severity: warning

Service Alerts

# Service down
- alert: ServiceDown
  expr: probe_success == 0
  for: 3m
  severity: critical

# Certificate expiry
- alert: CertificateExpiringSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
  for: 1h
  severity: warning

ServiceMonitor Configuration

Example ServiceMonitor for applications:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Metrics Retention

Query Examples

Useful PromQL Queries

# CPU usage by node
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Disk I/O rate
rate(node_disk_read_bytes_total[5m]) + rate(node_disk_written_bytes_total[5m])

# Container restart rate
rate(kube_pod_container_status_restarts_total[15m]) > 0

# HTTP error rate
rate(nginx_ingress_controller_requests{status=~"5.."}[5m])

Best Practices

1. Label Management

   • Use consistent labeling
   • Avoid high-cardinality labels
   • Document label meanings

2. Query Optimization

   • Use recording rules for complex queries
   • Aggregate before graphing
   • Limit time ranges

3. Alert Tuning

   • Avoid alert fatigue
   • Set appropriate thresholds
   • Use inhibition rules

Troubleshooting

Common Issues

High Memory Usage

# Check cardinality
curl -s http://prometheus:9090/api/v1/label/__name__/values | jq '. | length'

# Top memory consumers
curl -s http://prometheus:9090/api/v1/query?query=prometheus_tsdb_symbol_table_size_bytes | jq

Slow Queries

# Enable query logging
curl -X POST http://prometheus:9090/api/v1/admin/tsdb/snapshot

# Check slow query log
kubectl logs -n monitoring prometheus-0 | grep "query took"