Storage

Storage architecture, configuration, and management

Storage Architecture

The homelab implements a multi-tiered storage strategy combining high-performance local storage with centralized network storage for different workload requirements.

Storage Tiers

Storage Architecture
  1. Primary Storage: TrueNAS ZFS with iSCSI
  2. Object Storage: MinIO S3-compatible storage
  3. Local Storage: VM local disks for OS and temp data
  4. Backup Storage: Encrypted off-site backups

TrueNAS Configuration

ZFS Pool Configuration

Primary Pool: tank

# Pool composition
Pool: tank
  State: ONLINE
  Scan: scrub repaired 0B in 0 days 02:15:32
  
  NAME        STATE     READ WRITE CKSUM
  tank        ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
      sdc     ONLINE       0     0     0

ZFS Features:

  • Compression: LZ4 enabled
  • Deduplication: Disabled (performance)
  • Snapshots: Automated hourly/daily
  • Scrubbing: Weekly scheduled

iSCSI Target Configuration

Target Creation:

# Create iSCSI target
sudo iscsi target create iqn.2024-01.us.dodwell.home:target1

# Create extent
sudo iscsi extent create extent1 /dev/zvol/tank/iscsi/target1

# Associate target with extent
sudo iscsi targetextent create target1 extent1

Active iSCSI Targets:

Target NameSizePurposePortal
iqn.2024-01.us.dodwell.home:gitea10GBGitea data192.168.1.6:3260
iqn.2024-01.us.dodwell.home:bitcoin10TBBitcoin Core192.168.1.6:3260
iqn.2024-01.us.dodwell.home:home-assistant100GBHome Assistant192.168.1.6:3260
iqn.2024-01.us.dodwell.home:prometheus500GBPrometheus TSDB192.168.1.6:3260
iqn.2024-01.us.dodwell.home:vault50GBVault storage192.168.1.6:3260

NFS Shares

NFS Export Configuration:

# /etc/exports
/mnt/tank/nfs/static-web    192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)
/mnt/tank/nfs/authelia      192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)
/mnt/tank/nfs/prometheus    192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)
/mnt/tank/nfs/squid         192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=1000,anongid=1000)

NFS Permissions:

  • User mapping: All squash to uid/gid 1000
  • Access: Read/write for K3s cluster
  • Sync mode: Synchronous writes for consistency

Kubernetes Storage

Storage Classes

TrueNAS iSCSI Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-iscsi
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  fsType: ext4

TrueNAS NFS Storage Class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: truenas-nfs
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - vers=4.1
  - rsize=262144
  - wsize=262144
  - hard
  - proto=tcp
  - timeo=600
  - noatime

Persistent Volume Management

PV Creation Pattern:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: truenas-iscsi
  iscsi:
    targetPortal: 192.168.1.6:3260
    iqn: iqn.2024-01.us.dodwell.home:example
    lun: 0
    fsType: ext4

PVC Usage:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: truenas-iscsi

Volume Snapshots

Snapshot Class:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: truenas-snapshot-class
driver: csi.truenas.com
deletionPolicy: Delete

Taking Snapshots:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: example-snapshot
spec:
  volumeSnapshotClassName: truenas-snapshot-class
  source:
    persistentVolumeClaimName: example-pvc

MinIO Object Storage

MinIO Configuration

MinIO Setup:

# MinIO Helm values
mode: standalone
persistence:
  enabled: true
  existingClaim: minio-data
  
resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
    
buckets:
  - name: harbor
    policy: none
    purge: false
  - name: wiki
    policy: public
    purge: false
  - name: backups
    policy: none
    purge: false

Access Configuration:

  • Root User: minio
  • Console Access: https://minio.home.dodwell.us
  • API Access: https://minio-api.home.dodwell.us

S3 API Integration

Harbor Registry Integration:

# Harbor using MinIO as storage backend
storage:
  s3:
    endpoint: http://minio:9000
    bucket: harbor
    accesskey: harbor-access-key
    secretkey: harbor-secret-key
    secure: false

Application Integration:

# Configure MinIO client
mc alias set homelab http://minio-api.home.dodwell.us minio minio123

# Create bucket
mc mb homelab/application-data

# Upload files
mc cp /path/to/file homelab/application-data/

Backup Strategy

Backup Tiers

Tier 1: Application-level Backups

  • Frequency: Hourly for critical data
  • Retention: 24 hours locally
  • Method: Application-specific tools

Tier 2: Volume Snapshots

  • Frequency: Daily ZFS snapshots
  • Retention: 30 days
  • Method: Automated ZFS snapshot scripts

Tier 3: Off-site Backups

  • Frequency: Weekly full backups
  • Retention: 3 months
  • Method: Encrypted backup to external storage

Backup Procedures

ZFS Snapshot Automation:

#!/bin/bash
# /usr/local/bin/zfs-snapshot.sh

POOL="tank"
SNAPSHOT_PREFIX="auto"
RETENTION_DAYS=30

# Create snapshot
zfs snapshot ${POOL}@${SNAPSHOT_PREFIX}-$(date +%Y%m%d_%H%M%S)

# Clean old snapshots
zfs list -t snapshot -H -o name | \
grep "${POOL}@${SNAPSHOT_PREFIX}" | \
head -n -${RETENTION_DAYS} | \
xargs -r -n 1 zfs destroy

Kubernetes Backup:

#!/bin/bash
# Backup etcd
kubectl get all --all-namespaces -o yaml > cluster-state-$(date +%Y%m%d).yaml

# Backup persistent volumes
for pv in $(kubectl get pv -o name); do
    kubectl get $pv -o yaml > backup-$pv-$(date +%Y%m%d).yaml
done

# Backup secrets
kubectl get secrets --all-namespaces -o yaml > secrets-$(date +%Y%m%d).yaml

Disaster Recovery

Recovery Procedures:

  1. Data Loss: Restore from most recent snapshot
  2. Volume Corruption: Create new volume and restore data
  3. Storage Failure: Replace hardware and restore from backup
  4. Site Disaster: Restore from off-site encrypted backups

Recovery Testing:

# Test snapshot restore
zfs clone tank@auto-20240101_120000 tank/test-restore

# Test backup integrity
sha256sum -c backup-checksums.txt

# Test application restore
kubectl apply -f backup-manifests/

Performance Optimization

Storage Performance

ZFS Optimization:

# ARC (Adaptive Replacement Cache) tuning
echo 'vfs.zfs.arc_max="8G"' >> /etc/sysctl.conf

# L2ARC (SSD cache) configuration
zpool add tank cache /dev/nvme0n1

# ZIL (ZFS Intent Log) optimization
zpool add tank log /dev/nvme0n2

iSCSI Performance:

# Network optimization
echo 'net.core.rmem_max = 16777216' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 16777216' >> /etc/sysctl.conf

# iSCSI target tuning
echo 'lio_target.iscsi_target_max_queues = 8' >> /etc/modprobe.d/iscsi.conf

Monitoring and Alerting

Storage Metrics:

# Prometheus rules for storage
groups:
- name: storage
  rules:
  - alert: ZFSPoolDegraded
    expr: zfs_pool_state != 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "ZFS pool {{ $labels.pool }} is degraded"
      
  - alert: HighDiskUsage
    expr: (zfs_dataset_used_bytes / zfs_dataset_quota_bytes) > 0.9
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Dataset {{ $labels.dataset }} is {{ $value | humanizePercentage }} full"

Grafana Dashboards:

  • ZFS pool health and performance
  • iSCSI target utilization
  • Storage capacity trending
  • Backup success/failure rates

Storage Operations

Daily Operations

Health Checks:

# ZFS pool status
zpool status -v

# Check iSCSI sessions
iscsiadm -m session

# Verify NFS exports
showmount -e localhost

# Check disk usage
df -h
zfs list -o space

Maintenance Tasks:

# ZFS scrub (weekly)
zpool scrub tank

# Clean old snapshots
zfs destroy tank@snapshot-to-delete

# Monitor I/O performance
iostat -x 1 5

Troubleshooting

Common Issues:

iSCSI Connection Problems:

# Check target discovery
iscsiadm -m discovery -t st -p 192.168.1.6:3260

# Login to target
iscsiadm -m node -l

# Check session status
iscsiadm -m session -P 3

NFS Mount Issues:

# Check NFS exports
showmount -e 192.168.1.6

# Test mount
mount -t nfs4 192.168.1.6:/mnt/tank/nfs/test /mnt/test

# Check mount status
mount | grep nfs

ZFS Performance Issues:

# Check ARC statistics
cat /proc/spl/kstat/zfs/arcstats

# Monitor pool I/O
zpool iostat tank 1

# Check for errors
zpool events -v

Storage Documentation References