Service Deployment

Service deployment strategies, GitOps workflows, and automation

Deployment Architecture

The homelab uses a GitOps-based deployment strategy with ArgoCD managing all application deployments from Git repositories.

GitOps Workflow

Code Changes: Push to Git repository
ArgoCD Sync: Automatic detection and deployment
Health Checks: Application health monitoring
Rollback: Automated rollback on failure

ArgoCD Configuration

ArgoCD Setup

ArgoCD Installation:

# Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Access ArgoCD UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

ArgoCD Access:

UI: https://argocd.home.dodwell.us
CLI: argocd login argocd.home.dodwell.us
Authentication: OIDC with Authelia

Project Structure

ArgoCD Projects:

# System Critical Project
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: system-critical
  namespace: argocd
spec:
  description: Critical infrastructure services
  destinations:
  - namespace: '*'
    server: https://kubernetes.default.svc
  sourceRepos:
  - '*'
  roles:
  - name: admin
    policies:
    - p, proj:system-critical:admin, applications, *, system-critical/*, allow
    - p, proj:system-critical:admin, repositories, *, *, allow

Application Projects:

system-critical: Core infrastructure (ArgoCD, cert-manager, ingress)
monitoring: Prometheus, Grafana, exporters
security: Authelia, Vault, SOPS
applications: User-facing applications
mail: Email infrastructure
telephony: VoIP services

Application Deployment

Application Example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: grafana
  namespace: argocd
spec:
  project: monitoring
  source:
    repoURL: https://gitea.home.dodwell.us/exodus/homelab.git
    targetRevision: HEAD
    path: k3s/apps/grafana/overlays/local
  destination:
    server: https://kubernetes.default.svc
    namespace: grafana
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Kustomize Configuration

Directory Structure

k3s/apps/
├── application-name/
│   ├── base/
│   │   ├── kustomization.yaml
│   │   ├── namespace.yaml
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ...
│   └── overlays/
│       └── local/
│           ├── kustomization.yaml
│           ├── ingress.yaml
│           └── patches/

Base Configuration

Base Kustomization:

# k3s/apps/grafana/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- namespace.yaml
- deployment.yaml
- service.yaml
- config-map.yaml

commonLabels:
  app: grafana
  component: monitoring
  environment: base

Environment Overlays

Local Environment:

# k3s/apps/grafana/overlays/local/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base
- ingress.yaml

patchesStrategicMerge:
- deployment-patch.yaml

commonLabels:
  environment: local

images:
- name: grafana/grafana
  newTag: 10.2.3

Deployment Patch:

# deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  template:
    spec:
      priorityClassName: critical-workload
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: In
                values:
                - "true"

Helm Integration

Helm Chart Management

Helm Chart Structure:

# k3s/apps/minio/base/helm/values.yaml
fullnameOverride: minio
persistence:
  enabled: true
  existingClaim: minio-data
mode: standalone
resources:
  requests:
    memory: "128Mi"
    cpu: "100m"

Helm Chart Generation:

# Generate manifests from Helm chart
helm template minio bitnami/minio \
  --values helm/values.yaml \
  --namespace minio \
  --output-dir ./manifests

# Apply with Kustomize
kustomize build overlays/local | kubectl apply -f -

Chart Updates

Update Process:

Version Check: Check for chart updates
Values Update: Update values.yaml if needed
Manifest Generation: Regenerate manifests
Testing: Test in development environment
Deployment: Commit and let ArgoCD sync

Container Registry

Harbor Registry

Harbor Configuration:

# Harbor with MinIO backend
storage:
  s3:
    endpoint: http://minio:9000
    bucket: harbor
    accesskey: harbor
    secretkey: harbor123
    secure: false
    
database:
  type: external
  external:
    host: postgresql
    port: 5432
    username: harbor
    password: harbor123
    database: harbor

Registry Access:

UI: https://harbor.home.dodwell.us
Docker: docker login harbor.home.dodwell.us
Kubernetes: Image pull secrets

Image Management

Image Scanning:

# Scan image for vulnerabilities
curl -X POST "https://harbor.home.dodwell.us/api/v2.0/projects/homelab/repositories/app/artifacts/latest/scan"

# Get scan results
curl "https://harbor.home.dodwell.us/api/v2.0/projects/homelab/repositories/app/artifacts/latest/scan/results"

Image Signing:

# Sign image with Cosign
cosign sign harbor.home.dodwell.us/homelab/app:latest

# Verify signature
cosign verify harbor.home.dodwell.us/homelab/app:latest

Deployment Strategies

Rolling Updates

Deployment Strategy:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  template:
    spec:
      containers:
      - name: app
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

Blue-Green Deployment

Blue-Green Configuration:

# Blue deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue

---
# Green deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green

Canary Deployment

Canary with Flagger:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: app-canary
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  progressDeadlineSeconds: 60
  service:
    port: 8080
  analysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      threshold: 99
    - name: request-duration
      threshold: 500

Configuration Management

Secret Management

SOPS Integration:

# Encrypted secret
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  username: ENC[AES256_GCM,data:Qn4XzQ==,iv:123...,tag:abc...,type:str]
  password: ENC[AES256_GCM,data:Xy2Bvw==,iv:456...,tag:def...,type:str]

Vault Integration:

# Vault secret injection
apiVersion: v1
kind: Secret
metadata:
  name: vault-secret
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "myapp"
    vault.hashicorp.com/agent-inject-secret-config: "secret/myapp/config"

ConfigMap Management

Configuration Versioning:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config-v1
  labels:
    version: v1
data:
  config.yaml: |
    database:
      host: postgresql
      port: 5432
    redis:
      host: redis
      port: 6379

Monitoring and Observability

Deployment Monitoring

ArgoCD Metrics:

# ArgoCD application metrics
argocd_app_info{dest_server="https://kubernetes.default.svc",health_status="Healthy",name="grafana",namespace="argocd",operation="",project="monitoring",repo="https://gitea.home.dodwell.us/exodus/homelab.git",sync_status="Synced"}

# Sync metrics
argocd_app_sync_total{dest_server="https://kubernetes.default.svc",name="grafana",namespace="argocd",project="monitoring",repo="https://gitea.home.dodwell.us/exodus/homelab.git"}

Deployment Health:

# Kubernetes deployment metrics
kube_deployment_status_replicas_available{deployment="grafana",namespace="grafana"}
kube_deployment_status_replicas_unavailable{deployment="grafana",namespace="grafana"}

Alerting

Deployment Alerts:

groups:
- name: deployment
  rules:
  - alert: ApplicationSyncFailed
    expr: argocd_app_sync_total{phase="Failed"} > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Application {{ $labels.name }} sync failed"
      
  - alert: DeploymentReplicasMismatch
    expr: kube_deployment_status_replicas_available != kube_deployment_spec_replicas
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: "Deployment {{ $labels.deployment }} has {{ $value }} replicas available"

Testing and Validation

Pre-deployment Testing

Manifest Validation:

# Validate Kubernetes manifests
kubectl apply --dry-run=client -f manifests/

# Validate with kubeval
kubeval manifests/*.yaml

# Security scanning
kube-score score manifests/

Integration Testing:

# Test service endpoints
curl -f https://grafana.home.dodwell.us/api/health

# Test authentication
curl -H "Authorization: Bearer $TOKEN" https://grafana.home.dodwell.us/api/user

Post-deployment Validation

Health Checks:

# Check deployment status
kubectl get deployments -A

# Check pod status
kubectl get pods -A | grep -v Running

# Check service endpoints
kubectl get endpoints -A

Smoke Tests:

# Test critical paths
./smoke-tests/test-authentication.sh
./smoke-tests/test-database-connectivity.sh
./smoke-tests/test-external-services.sh

Troubleshooting

Common Issues

ArgoCD Sync Issues:

# Check application status
argocd app get grafana

# Manual sync
argocd app sync grafana --force

# Check sync history
argocd app history grafana

Deployment Failures:

# Check deployment events
kubectl describe deployment grafana -n grafana

# Check pod logs
kubectl logs -f deployment/grafana -n grafana

# Check resource constraints
kubectl top pods -n grafana

Image Pull Issues:

# Check image pull secrets
kubectl get secrets -o yaml | grep dockerconfigjson

# Test image pull
kubectl run test --image=harbor.home.dodwell.us/homelab/app:latest --rm -it

# Check registry connectivity
curl -v https://harbor.home.dodwell.us/v2/

Debugging Commands

ArgoCD CLI:

# Login to ArgoCD
argocd login argocd.home.dodwell.us

# List applications
argocd app list

# Get application details
argocd app get grafana -o yaml

# Watch application sync
argocd app sync grafana --watch

Kubectl Commands:

# Check all resources in namespace
kubectl get all -n grafana

# Describe problematic resource
kubectl describe pod grafana-xxx -n grafana

# Check events
kubectl get events -n grafana --sort-by=.metadata.creationTimestamp

Deployment Documentation References

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.