feat: add home-assistant-dashboard-conventions skill + update multiple skills
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
This commit is contained in:
Debian
@@ -0,0 +1,276 @@
|
||||
# InfluxDB K8s Migration — Session Detail (2026-07-14)
|
||||
|
||||
## Context
|
||||
|
||||
Migrating InfluxDB v2 from CT109 (hot, 10.0.30.109) + CT134 (archive, 10.0.30.110)
|
||||
to a single K8s StatefulSet. Design was developed through the Compound Engineering
|
||||
brainstorming process (brainstorming → writing-plans → execution).
|
||||
|
||||
Design doc: `docs/plans/2026-07-14-influxdb-k8s-migration-design.md`
|
||||
Implementation plan: `docs/plans/2026-07-14-influxdb-k8s-migration-plan.md`
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Fresh Deploy + Replication** (Option B) over Backup/Restore (Option A)
|
||||
- **Single instance** with both buckets + local downsampling task (not two instances)
|
||||
- **Dual-Write Phase** cutover: CT109 stays as fallback for 1-2 weeks
|
||||
- **Daily backup** to Norris S3 via CronJob (same pattern as CNPG)
|
||||
|
||||
## 1Password Item Creation
|
||||
|
||||
### Vault: "Kubernetes ESO" (ID: 334ykdtj5kar3jlpcrztjvx2fu)
|
||||
|
||||
Created two items using the K8s ESO service account token (extracted from
|
||||
`onepassword-token` secret in `external-secrets` namespace):
|
||||
|
||||
1. **influxdb-k8s-admin** (Login category, ID: pc7olfgnbfc34ljjcnafrbf4si)
|
||||
- username: dominik
|
||||
- password: generated via `openssl rand -base64 24`
|
||||
- url: http://10.0.30.204:8086
|
||||
|
||||
2. **influxdb-k8s-token** (API Credential category, ID: ec7u4yb2zumae7ewq3yxciynhy)
|
||||
- TWO-STEP required: `op item create` then `op item edit <ID> credential="<token>"`
|
||||
|
||||
### ExternalSecret Key Mapping
|
||||
|
||||
```yaml
|
||||
data:
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_USERNAME
|
||||
remoteRef:
|
||||
key: influxdb-k8s-admin/username # Login category → "username" field
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_PASSWORD
|
||||
remoteRef:
|
||||
key: influxdb-k8s-admin/password # Login category → "password" field
|
||||
- secretKey: DOCKER_INFLUXDB_INIT_ADMIN_TOKEN
|
||||
remoteRef:
|
||||
key: influxdb-k8s-token/credential # API Credential → "credential" field
|
||||
```
|
||||
|
||||
## Deployment Details
|
||||
|
||||
### Resource Allocation
|
||||
- VIP: 10.0.30.204 (Cilium L2 LB IPAM, pool .200-.250)
|
||||
- PVC data: 30Gi on ceph-hdd-replica
|
||||
- PVC config: 1Gi on ceph-hdd-replica
|
||||
- Resources: 512Mi-2Gi RAM, 250m-2000m CPU
|
||||
|
||||
### Init Mode Behavior
|
||||
- `DOCKER_INFLUXDB_INIT_MODE=setup` creates Org `homelab` + Bucket `ha_hot_90d` (90d retention)
|
||||
- Admin token from 1Password is set as the initial all-access token
|
||||
- PostSync Job creates: `ha_archive_5y_5m` bucket (5y/1825d retention) + downsampling task
|
||||
|
||||
### K8s Org ID differs from CT109
|
||||
- CT109 org `homelab`: ID `d2aeab314f4408ee`
|
||||
- K8s org `homelab`: ID `430395547892375d`
|
||||
- Replication `--remote-org-id` must use the DESTINATION (K8s) org ID
|
||||
|
||||
## Replication Setup
|
||||
|
||||
```bash
|
||||
# On CT109: create remote pointing to K8s
|
||||
influx remote create \
|
||||
--name k8s-influxdb \
|
||||
--remote-url http://10.0.30.204:8086 \
|
||||
--remote-api-token <K8S_TOKEN> \
|
||||
--remote-org-id 430395547892375d \
|
||||
--org homelab
|
||||
|
||||
# Create replication for hot bucket
|
||||
# CRITICAL: flags are --local-bucket-id and --remote-bucket-id (NOT --local-bucket)
|
||||
influx replication create \
|
||||
--name ha-hot-to-k8s \
|
||||
--remote-id <REMOTE_ID> \
|
||||
--local-bucket-id 3ca435503b6d5f59 \
|
||||
--remote-bucket-id 7a2ec96a41099ee8 \
|
||||
--org homelab
|
||||
```
|
||||
|
||||
Verification: `influx replication list` shows `Latest Status Code: 204` (success).
|
||||
|
||||
## Backfill via Flux `to()`
|
||||
|
||||
Chunked backfill (3 parallel ranges) from CT109 to K8s:
|
||||
|
||||
```flux
|
||||
from(bucket: "ha_hot_90d")
|
||||
|> range(start: 2026-04-15T00:00:00Z, stop: 2026-05-15T00:00:00Z)
|
||||
|> to(bucket: "ha_hot_90d", host: "http://10.0.30.204:8086",
|
||||
token: "<K8S_TOKEN>", org: "homelab")
|
||||
```
|
||||
|
||||
Result: 4,253,134 rows transferred. Disk footprint only 51 MB on K8s (vs 4.1 GB
|
||||
on CT109) — InfluxDB v2 columnar compression is extremely effective on fresh shards.
|
||||
|
||||
## Downsampling Task — Pitfall and Fix
|
||||
|
||||
### Error 1: Missing `option task` header
|
||||
```
|
||||
400 Bad Request: invalid options: no task options defined
|
||||
```
|
||||
Fix: Add `option task = { name: "archive_5m_to_hdd_90d", every: 1h }` at the top.
|
||||
|
||||
### Error 2: String fields crash `mean()`
|
||||
HA writes `icon_str`, `state_class_str`, `attribution_str` alongside numeric `value`.
|
||||
Fix: `|> filter(fn: (r) => r._field == "value")` before aggregation.
|
||||
|
||||
### Final working Flux
|
||||
```flux
|
||||
option task = { name: "archive_5m_to_hdd_90d", every: 1h }
|
||||
|
||||
from(bucket: "ha_hot_90d")
|
||||
|> range(start: -91d, stop: -90d)
|
||||
|> filter(fn: (r) => r._field == "value")
|
||||
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|
||||
|> to(bucket: "ha_archive_5y_5m", org: "homelab")
|
||||
```
|
||||
|
||||
## ArgoCD ConfigMap Update Failure
|
||||
|
||||
After fixing the Flux script in the ConfigMap and pushing to Git, ArgoCD showed
|
||||
`OutOfSync/Healthy` but the live ConfigMap retained the old script content.
|
||||
The PostSync Hook Job ran with the OLD ConfigMap and failed again.
|
||||
|
||||
### Workaround
|
||||
1. `kubectl delete cm influxdb-init -n influxdb`
|
||||
2. Hard refresh ArgoCD app
|
||||
3. Created the task manually via `kubectl exec` (faster for one-time fixes)
|
||||
|
||||
## HA Cutover — Home Assistant OS Access
|
||||
|
||||
### Discovery
|
||||
HA was NOT on 10.0.30.100 (Docker host). Found by checking CT109 inbound connections:
|
||||
```bash
|
||||
ssh root@10.0.30.109 'ss -tnp | grep ":8086"'
|
||||
# Found: 10.0.30.10 connecting to CT109:8086
|
||||
```
|
||||
|
||||
### HA OS SSH Access
|
||||
- **Port 22222** (NOT 22 — port 22 is refused)
|
||||
- `ssh -i ~/.ssh/id_ed25519_proxmox -p 22222 root@10.0.30.10`
|
||||
- HA OS is a minimal Alpine-based system (no `hostname`, no `python3`)
|
||||
|
||||
### HA Configuration Path
|
||||
- Config at `/mnt/data/supervisor/homeassistant/configuration.yaml`
|
||||
- NOT at `/config/` (which is empty)
|
||||
- Supervisor metadata at `/mnt/data/supervisor/homeassistant.json` (contains
|
||||
`access_token` and `refresh_token` for HA Supervisor API)
|
||||
|
||||
### InfluxDB Config in HA
|
||||
```yaml
|
||||
influxdb:
|
||||
api_version: 2
|
||||
ssl: false
|
||||
host: 10.0.30.204 # Changed from 10.0.30.109
|
||||
port: 8086
|
||||
token: <K8S_TOKEN> # Changed from CT109 token
|
||||
organization: homelab
|
||||
bucket: ha_hot_90d
|
||||
tags:
|
||||
source: HA
|
||||
tags_attributes:
|
||||
- friendly_name
|
||||
default_measurement: units
|
||||
```
|
||||
|
||||
### Applying Changes
|
||||
Used `sed -i` (HA OS has no python3):
|
||||
```bash
|
||||
ssh -p 22222 root@10.0.30.10 \
|
||||
"sed -i 's/host: 10.0.30.109/host: 10.0.30.204/' /mnt/data/supervisor/homeassistant/configuration.yaml && \
|
||||
sed -i 's/token: 92ec.../token: <K8S_TOKEN>/' /mnt/data/supervisor/homeassistant/configuration.yaml"
|
||||
```
|
||||
|
||||
Restart HA:
|
||||
```bash
|
||||
ssh -p 22222 root@10.0.30.10 'docker restart homeassistant'
|
||||
```
|
||||
|
||||
### Verification
|
||||
After 90s, queried K8s InfluxDB for recent data — confirmed HA writing directly
|
||||
to K8s with `source: HA` tag and timestamps within the last 3 minutes.
|
||||
|
||||
## Backup CronJob — Multi-Container Pattern
|
||||
|
||||
### Image Limitations Discovered
|
||||
- `influxdb:2.7` — has `tar` but NO `python3`, `aws`, `rclone`, working `apt-get`
|
||||
- `amazon/aws-cli:2` — tag DOES NOT EXIST, must use specific version (e.g. `2.35.22`)
|
||||
- `amazon/aws-cli:2.35.22` — has `aws` but NO `tar`
|
||||
|
||||
### Working Pattern: init-container + upload-container
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: backup
|
||||
image: influxdb:2.7
|
||||
command: [/bin/sh, -c, |
|
||||
influx backup /shared/backup --host http://influxdb:8086 \
|
||||
--token "${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}" --org homelab
|
||||
cd /shared && tar czf backup.tar.gz backup/]
|
||||
envFrom:
|
||||
- secretRef: { name: influxdb-admin }
|
||||
volumeMounts:
|
||||
- { name: shared, mountPath: /shared }
|
||||
containers:
|
||||
- name: upload
|
||||
image: amazon/aws-cli:2.35.22
|
||||
command: [/bin/sh, -c, |
|
||||
aws s3 cp /shared/backup.tar.gz s3://homelab-influxdb-backup/$(date +%Y-%m-%d)/backup.tar.gz \
|
||||
--endpoint-url https://rgw.nbg.nsc.noris.cloud --region nsc-nbg]
|
||||
env: # MUST be uppercase AWS_* — envFrom with lowercase fails
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_access_key_id } }
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
valueFrom: { secretKeyRef: { name: influxdb-s3-backup, key: aws_secret_access_key } }
|
||||
volumeMounts:
|
||||
- { name: shared, mountPath: /shared }
|
||||
volumes:
|
||||
- { name: shared, emptyDir: {} }
|
||||
```
|
||||
|
||||
### S3 Credentials
|
||||
Reused existing `postgres-s3-backup` 1Password item (same Norris S3 account).
|
||||
ExternalSecret references `postgres-s3-backup/access_key_id` and
|
||||
`postgres-s3-backup/secret_access_key`.
|
||||
|
||||
### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`
|
||||
Bucket `homelab-influxdb-backup` must be created before first backup run.
|
||||
**CRITICAL**: `aws s3 mb` with `--region nsc-nbg` fails with
|
||||
`InvalidLocationConstraint: The nsc-nbg location constraint is not valid`.
|
||||
Must use `--region us-east-1` for bucket creation. `nsc-nbg` works for
|
||||
subsequent `aws s3 cp` operations but NOT for `s3 mb`.
|
||||
Use a one-off pod:
|
||||
```bash
|
||||
kubectl run aws-cli-create-bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
|
||||
--env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
|
||||
--command -- sh -c "aws s3 mb s3://homelab-influxdb-backup \
|
||||
--endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
|
||||
```
|
||||
|
||||
### Backup Verified Successful ✅
|
||||
InfluxDB backup CronJob ran successfully on 2026-07-14:
|
||||
- 77 shards backed up via `influx backup`
|
||||
- `backup.tar.gz` = 25.4 MB (compressed)
|
||||
- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
|
||||
- Total time: ~45 seconds
|
||||
- Both S3 buckets created: `homelab-influxdb-backup` + `homelab-gitea-backup`
|
||||
|
||||
## Commits
|
||||
|
||||
- `c2840f4` — Design + Implementation Plan docs
|
||||
- `85be11f` — InfluxDB StatefulSet + all 6 manifests
|
||||
- `03808c1` — Fix: `option task` header in Flux script
|
||||
- `a4ef1fe` — Backup CronJob + S3 ExternalSecret
|
||||
- `f4208d3` — Fix: rclone/aws/boto3 fallback (attempted)
|
||||
- `2728f2e` — Fix: init-container + aws-cli container split
|
||||
- `39879ed` — Fix: correct aws-cli image tag 2.35.22
|
||||
- `ebd1f5c` — Fix: move tar to backup container
|
||||
- `be26461` — Fix: uppercase AWS env vars
|
||||
|
||||
## Remaining Steps
|
||||
|
||||
1. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
|
||||
2. ~~Trigger backup test run~~ ✅ Verified (25.4 MB uploaded)
|
||||
3. ~~Verify backup uploaded to Norris S3~~ ✅ Confirmed
|
||||
4. After 1-2 weeks stable: decommission CT109 + CT134
|
||||
- Stop replication on CT109
|
||||
- Stop InfluxDB services on CT109 + CT134
|
||||
- Stilllege CTs in Proxmox
|
||||
Reference in New Issue
Block a user