hermes-skills/devops/rke2-cluster-administration/references/gitea-backup-cronjob-2026-07.md

# Gitea Backup CronJob — Session Detail (2026-07-14)

## Context

After deploying InfluxDB backup CronJob (§4.3i), user requested the
same backup pattern for Gitea. Unlike InfluxDB (single PVC, single
backup command), Gitea has two data stores:
1. **PVC** (`/data`) — repos, LFS, config, attachments, avatars
2. **External Galera DB** (10.0.30.70:3306) — users, issues, PRs, labels

## Design Decisions

### tar over `gitea dump`
- `gitea dump` produces a zip but can miss LFS objects and repos outside
  the default path
- `tar czf /data` captures everything reliably
- Trade-off: larger archive (143 MB), but complete

### kubectl exec + kubectl cp for PVC access
- PVC is RWO, mounted by the running `gitea-0` pod
- Can't mount it in the backup pod (Multi-Attach error)
- Solution: exec into the running pod to tar, then cp the archive out
- Requires RBAC: ServiceAccount with pods/exec permission

### Separate mysqldump initContainer
- `mysql:8.0` image has `mysqldump`
- Credentials from existing `gitea-db` K8s secret (already used by Gitea)
- `--single-transaction` for consistent dump without locking

### S3 credentials reuse
- Same Norris S3 account as Postgres/InfluxDB backups
- New ExternalSecret per namespace, same 1Password item keys
- New bucket: `homelab-gitea-backup`

## Manifest Files Created

| File | Purpose |
|------|---------|
| `clusters/main/gitea/backup-rbac.yaml` | ServiceAccount + Role (pods/exec, pods/get) + RoleBinding |
| `clusters/main/gitea/backup-s3-secret.yaml` | ExternalSecret → 1Password `postgres-s3-backup` creds |
| `clusters/main/gitea/backup-cronjob.yaml` | CronJob: 2 initContainers (files + db) + 1 upload container |

Commit: `1679786`

## CronJob Spec

- **Schedule:** `30 3 * * *` (03:30 UTC — 30 min after InfluxDB backup)
- **initContainer 1 (gitea-files):** `bitnami/kubectl:1.31` — kubectl exec tar + kubectl cp
- **initContainer 2 (gitea-db):** `mysql:8.0` — mysqldump against Galera
- **Container (upload):** `amazon/aws-cli:2.35.22` — aws s3 cp to Norris S3
- **Shared volume:** emptyDir (passed between all containers)
- **ServiceAccount:** `gitea-backup` (with RBAC for kubectl exec/cp)

## S3 Upload Paths

```
s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-data.tar.gz
s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-db.sql.gz
```

## Execution Results

### S3 Bucket Creation — CRITICAL: `--region us-east-1` not `nsc-nbg`

Both buckets (`homelab-influxdb-backup` + `homelab-gitea-backup`) were
created successfully via one-off pods. **CRITICAL PITFALL**: `aws s3 mb`
with `--region nsc-nbg` fails with `InvalidLocationConstraint: The
nsc-nbg location constraint is not valid`. Must use `--region us-east-1`
for bucket creation. The `nsc-nbg` region works for subsequent `aws s3 cp`
operations but NOT for `s3 mb`.

```bash
# CORRECT — us-east-1 for bucket creation
kubectl run aws-mb-$bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://$bucket \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"
```

### InfluxDB Backup — VERIFIED SUCCESSFUL ✅

InfluxDB backup CronJob ran successfully:
- 77 shards backed up
- `backup.tar.gz` = 25.4 MB
- Uploaded to `s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz`
- Total time: ~45 seconds

### Gitea Backup — BLOCKED by Image Pull Issues

The Gitea CronJob requires a kubectl image for the initContainer that
runs `kubectl exec` + `kubectl cp`. Multiple images failed:

1. **`bitnami/kubectl:1.31`** — tag does NOT exist on Docker Hub
   (`failed to resolve reference: docker.io/bitnami/kubectl:1.31: not found`)
2. **`bitnami/kubectl:1.31.4`** — same NotFound error
3. **`bitnami/kubectl:latest`** — pulling but extremely slow (Docker Hub rate-limit)
4. **`alpine/k8s:1.31.4`** — pulling but took 3+ minutes, still Pending

The `amazon/aws-cli:2.35.22` and `mysql:8.0` images pull fine (already
cached from the InfluxDB backup CronJob). Only the kubectl image is
problematic.

### kubectl cp Fails on Large Files

Attempting `kubectl cp` of the 143 MB gitea-data.tar.gz from the
mgmt-runner into an upload pod failed with:
```
write tcp 10.0.30.124:35976->10.0.30.51:6443: write: connection reset by peer
websocket: close 1006 (abnormal closure): unexpected EOF
```

`kubectl cp` uses the K8s API websocket to stream file data. Large files
(>~50 MB) are unreliable — the websocket connection resets before
transfer completes. Small files (<1 MB) work fine.

### ArgoCD Reports Success But Doesn't Create New Resources

When new manifest files (backup-rbac.yaml, backup-s3-secret.yaml,
backup-cronjob.yaml) were added to the existing `gitea-config` Application's
directory (`clusters/main/gitea/`), ArgoCD showed:
- Sync status: `Succeeded`
- Operation phase: `Succeeded`
- Message: `successfully synced (all tasks run)`
- BUT: `kubectl get cronjob -n gitea` returned `No resources found`

This is a known ArgoCD behavior with `ServerSideApply=true` — new files
in an existing directory aren't always detected on the first sync cycle.
The hard-refresh annotation was applied but still showed `OutOfSync`.

**Fix that worked:** Transfer the manifest files to VM200 via base64
encoding through SSH, then `kubectl apply -f` manually. The CronJob,
RBAC, and ExternalSecret were all created successfully this way.

```bash
# Transfer files from Hermes host to VM200 via base64
B64=$(base64 -w0 /path/to/manifest.yaml)
ssh root@10.0.30.124 "echo '$B64' | base64 -d > /root/iac-homelab/path/to/manifest.yaml"
ssh root@10.0.30.124 "KUBECONFIG=/root/.kube/config kubectl apply -f /root/iac-homelab/path/to/manifest.yaml"
```

### VM200 Repo Remote Not Updated After Gitea Migration

The mgmt-runner's `/root/iac-homelab` repo still pointed at the old
Gitea on CT108 (`http://...@10.0.30.105:3000/...`). After ArgoCD was
switched to K8s Gitea, the VM200 repo became stale. New commits pushed
to K8s Gitea were invisible to VM200.

**Fix**: Update the remote URL on VM200 to point at K8s Gitea, or
transfer files via base64 encoding through SSH.

### Manual Backup from mgmt-runner — Successful Workaround

When the CronJob pod approach is blocked by image issues, run the backup
directly from the mgmt-runner (VM200, 10.0.30.124) which has kubectl +
mysqldump already installed:

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
KUBECONFIG=/root/.kube/config
set -e

GITEA_POD=$(kubectl -n gitea get pod -l app.kubernetes.io/name=gitea \
  -o jsonpath="{.items[0].metadata.name}")
AKI=$(kubectl -n gitea get secret gitea-s3-backup \
  -o jsonpath="{.data.aws_access_key_id}" | base64 -d)
SAK=$(kubectl -n gitea get secret gitea-s3-backup \
  -o jsonpath="{.data.aws_secret_access_key}" | base64 -d)
DB_PASS=$(kubectl -n gitea get secret gitea-db \
  -o jsonpath="{.data.password}" | base64 -d)
DATE=$(date +%Y-%m-%d)

# 1. tar /data from gitea pod
kubectl -n gitea exec $GITEA_POD -- tar czf /tmp/gitea-data.tar.gz -C / data
kubectl -n gitea cp $GITEA_POD:/tmp/gitea-data.tar.gz /tmp/gitea-data.tar.gz
kubectl -n gitea exec $GITEA_POD -- rm -f /tmp/gitea-data.tar.gz

# 2. mysqldump against Galera
mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
  --password="$DB_PASS" --single-transaction \
  --routines --triggers gitea | gzip > /tmp/gitea-db.sql.gz

# 3. Upload via one-off pod with emptyDir (avoid hostPath — pods run on
#    different nodes!)
#    See pitfall below about hostPath
'
```

**Pitfall: hostPath volumes are node-local.** A pod scheduled on
worker-01 cannot read files from the mgmt-runner's `/tmp` via hostPath.
The pod lands on whichever node the scheduler chooses. Use `kubectl cp`
to transfer files into a pod with emptyDir, or use `nodeName:` to pin
the pod to a specific node (fragile — not recommended for production).

**Pitfall: `kubectl cp` to upload pod fails for large files.** The
143 MB gitea-data.tar.gz caused websocket connection resets. Workaround:
split the upload into smaller chunks, or use an initContainer that
generates the data directly (avoiding the cp step entirely).

## Recommended Fix for CronJob

The CronJob approach needs a kubectl image that can be reliably pulled.
Options:

1. **Pre-pull the image on all worker nodes** — `crictl pull alpine/k8s:1.31.4`
   on each worker, then the CronJob pulls instantly from local cache.
2. **Use a different kubectl image from a non-Docker-Hub registry** — e.g.
   `registry.k8s.io/kubectl-sidecar` or an internally hosted image.
3. **Restructure the CronJob to avoid kubectl entirely** — mount the PVC
   read-only (requires PVC to be RWX, or stop the pod first).
4. **Run backup as a script on the mgmt-runner** via systemd timer or
   crontab — pragmatic but not GitOps-managed.

Until the image issue is resolved, the manual mgmt-runner approach
(above) produces correct backups.

## Remaining Steps

1. ~~Create S3 bucket `homelab-gitea-backup`~~ ✅ Done
2. ~~Create S3 bucket `homelab-influxdb-backup`~~ ✅ Done
3. ~~InfluxDB backup test~~ ✅ Verified (25.4 MB uploaded)
4. **Fix Gitea CronJob kubectl image** — pre-pull or use alternative
5. **Gitea backup test run** — verify both files in S3
6. Monitor first scheduled runs (03:00 + 03:30 UTC)

## Pitfalls Summary

### `bitnami/kubectl:1.31` tag does NOT exist
The bitnami/kubectl image uses full semver tags (`1.31.4`, not `1.31`).
However, even with correct tags, Docker Hub rate-limiting causes
extremely slow pulls (3+ minutes to Pending). Pre-pull on all workers
or use an alternative registry.

### S3 bucket creation: `--region us-east-1` not `nsc-nbg`
`aws s3 mb` with `--region nsc-nbg` fails with `InvalidLocationConstraint`.
Use `--region us-east-1` for bucket creation. The `nsc-nbg` region works
for subsequent `aws s3 cp` operations.

### `kubectl cp` unreliable for large files (>50 MB)
WebSocket connection resets before transfer completes. Small files
(<1 MB) work fine. For large file transfers, consider splitting or
avoiding `kubectl cp` entirely.

### VM200 repo remote may be stale after Gitea migration
After switching ArgoCD from CT108 to K8s Gitea, the mgmt-runner's local
repo remote still points at the old CT108 URL. New commits are invisible.
Fix: `git remote set-url origin <new-url>` on VM200, or transfer files
via base64 through SSH.

### hostPath volumes are node-local
A pod using `hostPath: /tmp` reads from the NODE it's scheduled on, not
from the machine that created the files. If files are on the mgmt-runner
but the pod runs on a worker, the files are invisible. Use emptyDir +
`kubectl cp`, or pin the pod with `nodeName:` (fragile).

### RBAC required for kubectl exec/cp
Without explicit Role granting `pods/exec` + `pods/get`, the backup
pod's kubectl commands fail with `forbidden`. Must create ServiceAccount,
Role, and RoleBinding before the CronJob can function.

### kubectl cp namespace syntax
`kubectl cp` requires `namespace/pod:path` format. Missing the
namespace prefix causes failures when the backup pod and target pod
are in different namespaces (though typically same namespace here).

## Pattern Reuse

This pattern (PVC tar via kubectl exec + external DB dump + S3 upload)
applies to any K8s service with both PVC data and an external database:
- Gitea (PVC + Galera)
- Future services with similar dual-data-store architecture

For services with only PVC data (no external DB), skip the mysqldump
initContainer. For services with only an external DB (no PVC), skip
the kubectl exec initContainer.