hermes-skills/devops/rke2-cluster-administration/references/influxdb-k8s-migration-2026-07.md

InfluxDB K8s Migration — Session Detail (2026-07-14)

Context

Migrating InfluxDB v2 from CT109 (hot, 10.0.30.109) + CT134 (archive, 10.0.30.110) to a single K8s StatefulSet. Design was developed through the Compound Engineering brainstorming process (brainstorming → writing-plans → execution).

Design doc: docs/plans/2026-07-14-influxdb-k8s-migration-design.md Implementation plan: docs/plans/2026-07-14-influxdb-k8s-migration-plan.md

Key Decisions

• Fresh Deploy + Replication (Option B) over Backup/Restore (Option A)
• Single instance with both buckets + local downsampling task (not two instances)
• Dual-Write Phase cutover: CT109 stays as fallback for 1-2 weeks
• Daily backup to Norris S3 via CronJob (same pattern as CNPG)

1Password Item Creation

Vault: "Kubernetes ESO" (ID: 334ykdtj5kar3jlpcrztjvx2fu)

Created two items using the K8s ESO service account token (extracted from onepassword-token secret in external-secrets namespace):

1. influxdb-k8s-admin (Login category, ID: pc7olfgnbfc34ljjcnafrbf4si)

   • username: dominik
   • password: generated via openssl rand -base64 24
   • url: http://10.0.30.204:8086

2. influxdb-k8s-token (API Credential category, ID: ec7u4yb2zumae7ewq3yxciynhy)

   • TWO-STEP required: op item create then op item edit <ID> credential="<token>"

ExternalSecret Key Mapping

data:
  - secretKey: DOCKER_INFLUXDB_INIT_USERNAME
    remoteRef:
      key: influxdb-k8s-admin/username       # Login category → "username" field
  - secretKey: DOCKER_INFLUXDB_INIT_PASSWORD
    remoteRef:
      key: influxdb-k8s-admin/password       # Login category → "password" field
  - secretKey: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
    remoteRef:
      key: influxdb-k8s-token/credential     # API Credential → "credential" field

Deployment Details

Resource Allocation

• VIP: 10.0.30.204 (Cilium L2 LB IPAM, pool .200-.250)
• PVC data: 30Gi on ceph-hdd-replica
• PVC config: 1Gi on ceph-hdd-replica
• Resources: 512Mi-2Gi RAM, 250m-2000m CPU

Init Mode Behavior

• DOCKER_INFLUXDB_INIT_MODE=setup creates Org homelab + Bucket ha_hot_90d (90d retention)
• Admin token from 1Password is set as the initial all-access token
• PostSync Job creates: ha_archive_5y_5m bucket (5y/1825d retention) + downsampling task

K8s Org ID differs from CT109

• CT109 org homelab: ID d2aeab314f4408ee
• K8s org homelab: ID 430395547892375d
• Replication --remote-org-id must use the DESTINATION (K8s) org ID

Replication Setup

# On CT109: create remote pointing to K8s
influx remote create \
  --name k8s-influxdb \
  --remote-url http://10.0.30.204:8086 \
  --remote-api-token <K8S_TOKEN> \
  --remote-org-id 430395547892375d \
  --org homelab

# Create replication for hot bucket
# CRITICAL: flags are --local-bucket-id and --remote-bucket-id (NOT --local-bucket)
influx replication create \
  --name ha-hot-to-k8s \
  --remote-id <REMOTE_ID> \
  --local-bucket-id 3ca435503b6d5f59 \
  --remote-bucket-id 7a2ec96a41099ee8 \
  --org homelab

Verification: influx replication list shows Latest Status Code: 204 (success).

Backfill via Flux to()

Chunked backfill (3 parallel ranges) from CT109 to K8s:

from(bucket: "ha_hot_90d")
  |> range(start: 2026-04-15T00:00:00Z, stop: 2026-05-15T00:00:00Z)
  |> to(bucket: "ha_hot_90d", host: "http://10.0.30.204:8086",
        token: "<K8S_TOKEN>", org: "homelab")

Result: 4,253,134 rows transferred. Disk footprint only 51 MB on K8s (vs 4.1 GB on CT109) — InfluxDB v2 columnar compression is extremely effective on fresh shards.

Downsampling Task — Pitfall and Fix

Error 1: Missing option task header

400 Bad Request: invalid options: no task options defined

Fix: Add option task = { name: "archive_5m_to_hdd_90d", every: 1h } at the top.

Error 2: String fields crash mean()

HA writes icon_str, state_class_str, attribution_str alongside numeric value. Fix: |> filter(fn: (r) => r._field == "value") before aggregation.

Final working Flux

option task = { name: "archive_5m_to_hdd_90d", every: 1h }

from(bucket: "ha_hot_90d")
  |> range(start: -91d, stop: -90d)
  |> filter(fn: (r) => r._field == "value")
  |> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
  |> to(bucket: "ha_archive_5y_5m", org: "homelab")

ArgoCD ConfigMap Update Failure

After fixing the Flux script in the ConfigMap and pushing to Git, ArgoCD showed OutOfSync/Healthy but the live ConfigMap retained the old script content. The PostSync Hook Job ran with the OLD ConfigMap and failed again.

Workaround

1. kubectl delete cm influxdb-init -n influxdb
2. Hard refresh ArgoCD app
3. Created the task manually via kubectl exec (faster for one-time fixes)

HA Cutover — Home Assistant OS Access

Discovery

HA was NOT on 10.0.30.100 (Docker host). Found by checking CT109 inbound connections:

ssh root@10.0.30.109 'ss -tnp | grep ":8086"'
# Found: 10.0.30.10 connecting to CT109:8086

HA OS SSH Access

• Port 22222 (NOT 22 — port 22 is refused)
• ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10
• HA OS is a minimal Alpine-based system (no hostname, no python3)

HA Configuration Path

• Config at /mnt/data/supervisor/homeassistant/configuration.yaml
• NOT at /config/ (which is empty)
• Supervisor metadata at /mnt/data/supervisor/homeassistant.json (contains access_token and refresh_token for HA Supervisor API)

InfluxDB Config in HA

influxdb:
  api_version: 2
  ssl: false
  host: 10.0.30.204       # Changed from 10.0.30.109
  port: 8086
  token: <K8S_TOKEN>     # Changed from CT109 token
  organization: homelab
  bucket: ha_hot_90d
  tags:
    source: HA
  tags_attributes:
    - friendly_name
  default_measurement: units

Applying Changes

Used sed -i (HA OS has no python3):

ssh -p 22222 root@10.0.30.10 \
  "sed -i 's/host: 10.0.30.109/host: 10.0.30.204/' /mnt/data/supervisor/homeassistant/configuration.yaml && \
   sed -i 's/token: 92ec.../token: <K8S_TOKEN>/' /mnt/data/supervisor/homeassistant/configuration.yaml"

Restart HA:

ssh -p 22222 root@10.0.30.10 'docker restart homeassistant'

Verification

After 90s, queried K8s InfluxDB for recent data — confirmed HA writing directly to K8s with source: HA tag and timestamps within the last 3 minutes.

Backup CronJob — Multi-Container Pattern

Image Limitations Discovered

• influxdb:2.7 — has tar but NO python3, aws, rclone, working apt-get
• amazon/aws-cli:2 — tag DOES NOT EXIST, must use specific version (e.g. 2.35.22)
• amazon/aws-cli:2.35.22 — has aws but NO tar

Working Pattern: init-container + upload-container

initContainers:
  - name: backup
    image: influxdb:2.7
    command: [/bin/sh, -c, |
      influx backup /shared/backup --host http://influxdb:8086 \
        --token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
      cd /shared && tar czf backup.tar.gz backup/]
    envFrom:
      - secretRef: { name: influxdb-admin }
    volumeMounts:
      - { name: shared, mountPath: /shared }
containers:
  - name: upload
    image: amazon/aws-cli:2.35.22
    command: [/bin/sh, -c, |
      aws s3 cp /shared/backup.tar.gz s3://homelab-influxdb-backup/$(date +%Y-%m-%d)/backup.tar.gz \
        --endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
    env:  # MUST be uppercase AWS_* — envFrom with lowercase fails
      - name: AWS_ACCESS_KEY_ID
        valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_access_key_id } }
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_secret_access_key } }
    volumeMounts:
      - { name: shared, mountPath: /shared }
volumes:
  - { name: shared, emptyDir: {} }

S3 Credentials

Reused existing postgres-s3-backup 1Password item (same Norris S3 account). ExternalSecret references postgres-s3-backup/access_key_id and postgres-s3-backup/secret_access_key.

S3 Bucket Creation — CRITICAL: --region us-east-1 not nsc-nbg

Bucket homelab-influxdb-backup must be created before first backup run. CRITICAL: aws s3 mb with --region nsc-nbg fails with InvalidLocationConstraint: The nsc-nbg location constraint is not valid. Must use --region us-east-1 for bucket creation. nsc-nbg works for subsequent aws s3 cp operations but NOT for s3 mb. Use a one-off pod:

kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"

Backup Verified Successful ✅

InfluxDB backup CronJob ran successfully on 2026-07-14:

• 77 shards backed up via influx backup
• backup.tar.gz = 25.4 MB (compressed)
• Uploaded to s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz
• Total time: ~45 seconds
• Both S3 buckets created: homelab-influxdb-backup + homelab-gitea-backup

Commits

• c2840f4 — Design + Implementation Plan docs
• 85be11f — InfluxDB StatefulSet + all 6 manifests
• 03808c1 — Fix: option task header in Flux script
• a4ef1fe — Backup CronJob + S3 ExternalSecret
• f4208d3 — Fix: rclone/aws/boto3 fallback (attempted)
• 2728f2e — Fix: init-container + aws-cli container split
• 39879ed — Fix: correct aws-cli image tag 2.35.22
• ebd1f5c — Fix: move tar to backup container
• be26461 — Fix: uppercase AWS env vars

Remaining Steps

1. Create S3 bucket homelab-influxdb-backup ✅ Done
2. Trigger backup test run ✅ Verified (25.4 MB uploaded)
3. Verify backup uploaded to Norris S3 ✅ Confirmed
4. After 1-2 weeks stable: decommission CT109 + CT134
   • Stop replication on CT109
   • Stop InfluxDB services on CT109 + CT134
   • Stilllege CTs in Proxmox