# InfluxDB K8s Migration — Session Detail (2026-07-14) ## Context Migrating InfluxDB v2 from CT109 (hot, 10.0.30.109) + CT134 (archive, 10.0.30.110) to a single K8s StatefulSet. Design was developed through the Compound Engineering brainstorming process (brainstorming → writing-plans → execution). Design doc: `docs/plans/2026-07-14-influxdb-k8s-migration-design.md` Implementation plan: `docs/plans/2026-07-14-influxdb-k8s-migration-plan.md` ## Key Decisions - **Fresh Deploy + Replication** (Option B) over Backup/Restore (Option A) - **Single instance** with both buckets + local downsampling task (not two instances) - **Dual-Write Phase** cutover: CT109 stays as fallback for 1-2 weeks - **Daily backup** to Norris S3 via CronJob (same pattern as CNPG) ## 1Password Item Creation ### Vault: "Kubernetes ESO" (ID: 334ykdtj5kar3jlpcrztjvx2fu) Created two items using the K8s ESO service account token (extracted from `onepassword-token` secret in `external-secrets` namespace): 1. **influxdb-k8s-admin** (Login category, ID: pc7olfgnbfc34ljjcnafrbf4si) - username: dominik - password: generated via `openssl rand -base64 24` - url: http://10.0.30.204:8086 2. **influxdb-k8s-token** (API Credential category, ID: ec7u4yb2zumae7ewq3yxciynhy) - TWO-STEP required: `op item create` then `op item edit credential=""` ### ExternalSecret Key Mapping ```yaml data: - secretKey: DOCKER_INFLUXDB_INIT_USERNAME remoteRef: key: influxdb-k8s-admin/username # Login category → "username" field - secretKey: DOCKER_INFLUXDB_INIT_PASSWORD remoteRef: key: influxdb-k8s-admin/password # Login category → "password" field - secretKey: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN remoteRef: key: influxdb-k8s-token/credential # API Credential → "credential" field ``` ## Deployment Details ### Resource Allocation - VIP: 10.0.30.204 (Cilium L2 LB IPAM, pool .200-.250) - PVC data: 30Gi on ceph-hdd-replica - PVC config: 1Gi on ceph-hdd-replica - Resources: 512Mi-2Gi RAM, 250m-2000m CPU ### Init Mode Behavior - `DOCKER_INFLUXDB_INIT_MODE=setup` creates Org `homelab` + Bucket `ha_hot_90d` (90d retention) - Admin token from 1Password is set as the initial all-access token - PostSync Job creates: `ha_archive_5y_5m` bucket (5y/1825d retention) + downsampling task ### K8s Org ID differs from CT109 - CT109 org `homelab`: ID `d2aeab314f4408ee` - K8s org `homelab`: ID `430395547892375d` - Replication `--remote-org-id` must use the DESTINATION (K8s) org ID ## Replication Setup ```bash # On CT109: create remote pointing to K8s influx remote create \ --name k8s-influxdb \ --remote-url http://10.0.30.204:8086 \ --remote-api-token \ --remote-org-id 430395547892375d \ --org homelab # Create replication for hot bucket # CRITICAL: flags are --local-bucket-id and --remote-bucket-id (NOT --local-bucket) influx replication create \ --name ha-hot-to-k8s \ --remote-id \ --local-bucket-id 3ca435503b6d5f59 \ --remote-bucket-id 7a2ec96a41099ee8 \ --org homelab ``` Verification: `influx replication list` shows `Latest Status Code: 204` (success). ## Backfill via Flux `to()` Chunked backfill (3 parallel ranges) from CT109 to K8s: ```flux from(bucket: "ha_hot_90d") |> range(start: 2026-04-15T00:00:00Z, stop: 2026-05-15T00:00:00Z) |> to(bucket: "ha_hot_90d", host: "http://10.0.30.204:8086", token: "", org: "homelab") ``` Result: 4,253,134 rows transferred. Disk footprint only 51 MB on K8s (vs 4.1 GB on CT109) — InfluxDB v2 columnar compression is extremely effective on fresh shards. ## Downsampling Task — Pitfall and Fix ### Error 1: Missing `option task` header ``` 400 Bad Request: invalid options: no task options defined ``` Fix: Add `option task = { name: "archive_5m_to_hdd_90d", every: 1h }` at the top. ### Error 2: String fields crash `mean()` HA writes `icon_str`, `state_class_str`, `attribution_str` alongside numeric `value`. Fix: `|> filter(fn: (r) => r._field == "value")` before aggregation. ### Final working Flux ```flux option task = { name: "archive_5m_to_hdd_90d", every: 1h } from(bucket: "ha_hot_90d") |> range(start: -91d, stop: -90d) |> filter(fn: (r) => r._field == "value") |> aggregateWindow(every: 5m, fn: mean, createEmpty: false) |> to(bucket: "ha_archive_5y_5m", org: "homelab") ``` ## ArgoCD ConfigMap Update Failure After fixing the Flux script in the ConfigMap and pushing to Git, ArgoCD showed `OutOfSync/Healthy` but the live ConfigMap retained the old script content. The PostSync Hook Job ran with the OLD ConfigMap and failed again. ### Workaround 1. `kubectl delete cm influxdb-init -n influxdb` 2. Hard refresh ArgoCD app 3. Created the task manually via `kubectl exec` (faster for one-time fixes) ## HA Cutover — Home Assistant OS Access ### Discovery HA was NOT on 10.0.30.100 (Docker host). Found by checking CT109 inbound connections: ```bash ssh root@10.0.30.109 'ss -tnp | grep ":8086"' # Found: 10.0.30.10 connecting to CT109:8086 ``` ### HA OS SSH Access - **Port 22222** (NOT 22 — port 22 is refused) - `ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10` - HA OS is a minimal Alpine-based system (no `hostname`, no `python3`) ### HA Configuration Path - Config at `/mnt/data/supervisor/homeassistant/configuration.yaml` - NOT at `/config/` (which is empty) - Supervisor metadata at `/mnt/data/supervisor/homeassistant.json` (contains `access_token` and `refresh_token` for HA Supervisor API) ### InfluxDB Config in HA ```yaml influxdb: api_version: 2 ssl: false host: 10.0.30.204 # Changed from 10.0.30.109 port: 8086 token: # Changed from CT109 token organization: homelab bucket: ha_hot_90d tags: source: HA tags_attributes: - friendly_name default_measurement: units ``` ### Applying Changes Used `sed -i` (HA OS has no python3): ```bash ssh -p 22222 root@10.0.30.10 \ "sed -i 's/host: 10.0.30.109/host: 10.0.30.204/' /mnt/data/supervisor/homeassistant/configuration.yaml && \ sed -i 's/token: 92ec.../token: /' /mnt/data/supervisor/homeassistant/configuration.yaml" ``` Restart HA: ```bash ssh -p 22222 root@10.0.30.10 'docker restart homeassistant' ``` ### Verification After 90s, queried K8s InfluxDB for recent data — confirmed HA writing directly to K8s with `source: HA` tag and timestamps within the last 3 minutes. ## Backup CronJob — Multi-Container Pattern ### Image Limitations Discovered - `influxdb:2.7` — has `tar` but NO `python3`, `aws`, `rclone`, working `apt-get` - `amazon/aws-cli:2` — tag DOES NOT EXIST, must use specific version (e.g. `2.35.22`) - `amazon/aws-cli:2.35.22` — has `aws` but NO `tar` ### Working Pattern: init-container + upload-container ```yaml initContainers: - name: backup image: influxdb:2.7 command: [/bin/sh, -c, | influx backup /shared/backup --host http://influxdb:8086 \ --token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab cd /shared && tar czf backup.tar.gz backup/] envFrom: - secretRef: { name: influxdb-admin } volumeMounts: - { name: shared, mountPath: /shared } containers: - name: upload image: amazon/aws-cli:2.35.22 command: [/bin/sh, -c, | aws s3 cp /shared/backup.tar.gz s3://homelab-influxdb-backup/$(date +%Y-%m-%d)/backup.tar.gz \ --endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg] env: # MUST be uppercase AWS_* — envFrom with lowercase fails - name: AWS_ACCESS_KEY_ID valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_access_key_id } } - name: AWS_SECRET_ACCESS_KEY valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_secret_access_key } } volumeMounts: - { name: shared, mountPath: /shared } volumes: - { name: shared, emptyDir: {} } ``` ### S3 Credentials Reused existing `postgres-s3-backup` 1Password item (same Norris S3 account). ExternalSecret references `postgres-s3-backup/access_key_id` and `postgres-s3-backup/secret_access_key`. ### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg` Bucket `homelab-influxdb-backup` must be created before first backup run. **CRITICAL**: `aws s3 mb` with `--region nsc-nbg` fails with `InvalidLocationConstraint: The nsc-nbg location constraint is not valid`. Must use `--region us-east-1` for bucket creation. `nsc-nbg` works for subsequent `aws s3 cp` operations but NOT for `s3 mb`. Use a one-off pod: ```bash kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \ --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \ --command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \ --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1" ``` ### Backup Verified Successful ✅ InfluxDB backup CronJob ran successfully on 2026-07-14: - 77 shards backed up via `influx backup` - `backup.tar.gz` = 25.4 MB (compressed) - Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz` - Total time: ~45 seconds - Both S3 buckets created: `homelab-influxdb-backup` + `homelab-gitea-backup` ## Commits - `c2840f4` — Design + Implementation Plan docs - `85be11f` — InfluxDB StatefulSet + all 6 manifests - `03808c1` — Fix: `option task` header in Flux script - `a4ef1fe` — Backup CronJob + S3 ExternalSecret - `f4208d3` — Fix: rclone/aws/boto3 fallback (attempted) - `2728f2e` — Fix: init-container + aws-cli container split - `39879ed` — Fix: correct aws-cli image tag 2.35.22 - `ebd1f5c` — Fix: move tar to backup container - `be26461` — Fix: uppercase AWS env vars ## Remaining Steps 1. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done 2. ~~Trigger backup test run~~ ✅ Verified (25.4 MB uploaded) 3. ~~Verify backup uploaded to Norris S3~~ ✅ Confirmed 4. After 1-2 weeks stable: decommission CT109 + CT134 - Stop replication on CT109 - Stop InfluxDB services on CT109 + CT134 - Stilllege CTs in Proxmox