hermes-skills/devops/rke2-cluster-administration/references/influxdb-k8s-migration-2026-07.md

# InfluxDB K8s Migration — Session Detail (2026-07-14)

## Context

Migrating InfluxDB v2 from CT109 (hot, 10.0.30.109) + CT134 (archive, 10.0.30.110)
to a single K8s StatefulSet. Design was developed through the Compound Engineering
brainstorming process (brainstorming → writing-plans → execution).

Design doc: `docs/plans/2026-07-14-influxdb-k8s-migration-design.md`
Implementation plan: `docs/plans/2026-07-14-influxdb-k8s-migration-plan.md`

## Key Decisions

- **Fresh Deploy + Replication** (Option B) over Backup/Restore (Option A)
- **Single instance** with both buckets + local downsampling task (not two instances)
- **Dual-Write Phase** cutover: CT109 stays as fallback for 1-2 weeks
- **Daily backup** to Norris S3 via CronJob (same pattern as CNPG)

## 1Password Item Creation

### Vault: "Kubernetes ESO" (ID: 334ykdtj5kar3jlpcrztjvx2fu)

Created two items using the K8s ESO service account token (extracted from
`onepassword-token` secret in `external-secrets` namespace):

1. **influxdb-k8s-admin** (Login category, ID: pc7olfgnbfc34ljjcnafrbf4si)
   - username: dominik
   - password: generated via `openssl rand -base64 24`
   - url: http://10.0.30.204:8086

2. **influxdb-k8s-token** (API Credential category, ID: ec7u4yb2zumae7ewq3yxciynhy)
   - TWO-STEP required: `op item create` then `op item edit <ID> credential="<token>"`

### ExternalSecret Key Mapping

```yaml
data:
  - secretKey: DOCKER_INFLUXDB_INIT_USERNAME
    remoteRef:
      key: influxdb-k8s-admin/username       # Login category → "username" field
  - secretKey: DOCKER_INFLUXDB_INIT_PASSWORD
    remoteRef:
      key: influxdb-k8s-admin/password       # Login category → "password" field
  - secretKey: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
    remoteRef:
      key: influxdb-k8s-token/credential     # API Credential → "credential" field
```

## Deployment Details

### Resource Allocation
- VIP: 10.0.30.204 (Cilium L2 LB IPAM, pool .200-.250)
- PVC data: 30Gi on ceph-hdd-replica
- PVC config: 1Gi on ceph-hdd-replica
- Resources: 512Mi-2Gi RAM, 250m-2000m CPU

### Init Mode Behavior
- `DOCKER_INFLUXDB_INIT_MODE=setup` creates Org `homelab` + Bucket `ha_hot_90d` (90d retention)
- Admin token from 1Password is set as the initial all-access token
- PostSync Job creates: `ha_archive_5y_5m` bucket (5y/1825d retention) + downsampling task

### K8s Org ID differs from CT109
- CT109 org `homelab`: ID `d2aeab314f4408ee`
- K8s org `homelab`: ID `430395547892375d`
- Replication `--remote-org-id` must use the DESTINATION (K8s) org ID

## Replication Setup

```bash
# On CT109: create remote pointing to K8s
influx remote create \
  --name k8s-influxdb \
  --remote-url http://10.0.30.204:8086 \
  --remote-api-token <K8S_TOKEN> \
  --remote-org-id 430395547892375d \
  --org homelab

# Create replication for hot bucket
# CRITICAL: flags are --local-bucket-id and --remote-bucket-id (NOT --local-bucket)
influx replication create \
  --name ha-hot-to-k8s \
  --remote-id <REMOTE_ID> \
  --local-bucket-id 3ca435503b6d5f59 \
  --remote-bucket-id 7a2ec96a41099ee8 \
  --org homelab
```

Verification: `influx replication list` shows `Latest Status Code: 204` (success).

## Backfill via Flux `to()`

Chunked backfill (3 parallel ranges) from CT109 to K8s:

```flux
from(bucket: "ha_hot_90d")
  |> range(start: 2026-04-15T00:00:00Z, stop: 2026-05-15T00:00:00Z)
  |> to(bucket: "ha_hot_90d", host: "http://10.0.30.204:8086",
        token: "<K8S_TOKEN>", org: "homelab")
```

Result: 4,253,134 rows transferred. Disk footprint only 51 MB on K8s (vs 4.1 GB
on CT109) — InfluxDB v2 columnar compression is extremely effective on fresh shards.

## Downsampling Task — Pitfall and Fix

### Error 1: Missing `option task` header
```
400 Bad Request: invalid options: no task options defined
```
Fix: Add `option task = { name: "archive_5m_to_hdd_90d", every: 1h }` at the top.

### Error 2: String fields crash `mean()`
HA writes `icon_str`, `state_class_str`, `attribution_str` alongside numeric `value`.
Fix: `|> filter(fn: (r) => r._field == "value")` before aggregation.

### Final working Flux
```flux
option task = { name: "archive_5m_to_hdd_90d", every: 1h }

from(bucket: "ha_hot_90d")
  |> range(start: -91d, stop: -90d)
  |> filter(fn: (r) => r._field == "value")
  |> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
  |> to(bucket: "ha_archive_5y_5m", org: "homelab")
```

## ArgoCD ConfigMap Update Failure

After fixing the Flux script in the ConfigMap and pushing to Git, ArgoCD showed
`OutOfSync/Healthy` but the live ConfigMap retained the old script content.
The PostSync Hook Job ran with the OLD ConfigMap and failed again.

### Workaround
1. `kubectl delete cm influxdb-init -n influxdb`
2. Hard refresh ArgoCD app
3. Created the task manually via `kubectl exec` (faster for one-time fixes)

## HA Cutover — Home Assistant OS Access

### Discovery
HA was NOT on 10.0.30.100 (Docker host). Found by checking CT109 inbound connections:
```bash
ssh root@10.0.30.109 'ss -tnp | grep ":8086"'
# Found: 10.0.30.10 connecting to CT109:8086
```

### HA OS SSH Access
- **Port 22222** (NOT 22 — port 22 is refused)
- `ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10`
- HA OS is a minimal Alpine-based system (no `hostname`, no `python3`)

### HA Configuration Path
- Config at `/mnt/data/supervisor/homeassistant/configuration.yaml`
- NOT at `/config/` (which is empty)
- Supervisor metadata at `/mnt/data/supervisor/homeassistant.json` (contains
  `access_token` and `refresh_token` for HA Supervisor API)

### InfluxDB Config in HA
```yaml
influxdb:
  api_version: 2
  ssl: false
  host: 10.0.30.204       # Changed from 10.0.30.109
  port: 8086
  token: <K8S_TOKEN>     # Changed from CT109 token
  organization: homelab
  bucket: ha_hot_90d
  tags:
    source: HA
  tags_attributes:
    - friendly_name
  default_measurement: units
```

### Applying Changes
Used `sed -i` (HA OS has no python3):
```bash
ssh -p 22222 root@10.0.30.10 \
  "sed -i 's/host: 10.0.30.109/host: 10.0.30.204/' /mnt/data/supervisor/homeassistant/configuration.yaml && \
   sed -i 's/token: 92ec.../token: <K8S_TOKEN>/' /mnt/data/supervisor/homeassistant/configuration.yaml"
```

Restart HA:
```bash
ssh -p 22222 root@10.0.30.10 'docker restart homeassistant'
```

### Verification
After 90s, queried K8s InfluxDB for recent data — confirmed HA writing directly
to K8s with `source: HA` tag and timestamps within the last 3 minutes.

## Backup CronJob — Multi-Container Pattern

### Image Limitations Discovered
- `influxdb:2.7` — has `tar` but NO `python3`, `aws`, `rclone`, working `apt-get`
- `amazon/aws-cli:2` — tag DOES NOT EXIST, must use specific version (e.g. `2.35.22`)
- `amazon/aws-cli:2.35.22` — has `aws` but NO `tar`

### Working Pattern: init-container + upload-container
```yaml
initContainers:
  - name: backup
    image: influxdb:2.7
    command: [/bin/sh, -c, |
      influx backup /shared/backup --host http://influxdb:8086 \
        --token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
      cd /shared && tar czf backup.tar.gz backup/]
    envFrom:
      - secretRef: { name: influxdb-admin }
    volumeMounts:
      - { name: shared, mountPath: /shared }
containers:
  - name: upload
    image: amazon/aws-cli:2.35.22
    command: [/bin/sh, -c, |
      aws s3 cp /shared/backup.tar.gz s3://homelab-influxdb-backup/$(date +%Y-%m-%d)/backup.tar.gz \
        --endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
    env:  # MUST be uppercase AWS_* — envFrom with lowercase fails
      - name: AWS_ACCESS_KEY_ID
        valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_access_key_id } }
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_secret_access_key } }
    volumeMounts:
      - { name: shared, mountPath: /shared }
volumes:
  - { name: shared, emptyDir: {} }
```

### S3 Credentials
Reused existing `postgres-s3-backup` 1Password item (same Norris S3 account).
ExternalSecret references `postgres-s3-backup/access_key_id` and
`postgres-s3-backup/secret_access_key`.

### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`
Bucket `homelab-influxdb-backup` must be created before first backup run.
**CRITICAL**: `aws s3 mb` with `--region nsc-nbg` fails with
`InvalidLocationConstraint: The nsc-nbg location constraint is not valid`.
Must use `--region us-east-1` for bucket creation. `nsc-nbg` works for
subsequent `aws s3 cp` operations but NOT for `s3 mb`.
Use a one-off pod:
```bash
kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
```

### Backup Verified Successful ✅
InfluxDB backup CronJob ran successfully on 2026-07-14:
- 77 shards backed up via `influx backup`
- `backup.tar.gz` = 25.4 MB (compressed)
- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
- Total time: ~45 seconds
- Both S3 buckets created: `homelab-influxdb-backup` + `homelab-gitea-backup`

## Commits

- `c2840f4` — Design + Implementation Plan docs
- `85be11f` — InfluxDB StatefulSet + all 6 manifests
- `03808c1` — Fix: `option task` header in Flux script
- `a4ef1fe` — Backup CronJob + S3 ExternalSecret
- `f4208d3` — Fix: rclone/aws/boto3 fallback (attempted)
- `2728f2e` — Fix: init-container + aws-cli container split
- `39879ed` — Fix: correct aws-cli image tag 2.35.22
- `ebd1f5c` — Fix: move tar to backup container
- `be26461` — Fix: uppercase AWS env vars

## Remaining Steps

1. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
2. ~~Trigger backup test run~~ ✅ Verified (25.4 MB uploaded)
3. ~~Verify backup uploaded to Norris S3~~ ✅ Confirmed
4. After 1-2 weeks stable: decommission CT109 + CT134
   - Stop replication on CT109
   - Stop InfluxDB services on CT109 + CT134
   - Stilllege CTs in Proxmox