hermes-skills/devops/rke2-cluster-administration/references/gitea-backup-cronjob-2026-07.md

Gitea Backup CronJob — Session Detail (2026-07-14)

Context

After deploying InfluxDB backup CronJob (§4.3i), user requested the same backup pattern for Gitea. Unlike InfluxDB (single PVC, single backup command), Gitea has two data stores:

1. PVC (/data) — repos, LFS, config, attachments, avatars
2. External Galera DB (10.0.30.70:3306) — users, issues, PRs, labels

Design Decisions

tar over gitea dump

• gitea dump produces a zip but can miss LFS objects and repos outside the default path
• tar czf /data captures everything reliably
• Trade-off: larger archive (143 MB), but complete

kubectl exec + kubectl cp for PVC access

• PVC is RWO, mounted by the running gitea-0 pod
• Can't mount it in the backup pod (Multi-Attach error)
• Solution: exec into the running pod to tar, then cp the archive out
• Requires RBAC: ServiceAccount with pods/exec permission

Separate mysqldump initContainer

• mysql:8.0 image has mysqldump
• Credentials from existing gitea-db K8s secret (already used by Gitea)
• --single-transaction for consistent dump without locking

S3 credentials reuse

• Same Norris S3 account as Postgres/InfluxDB backups
• New ExternalSecret per namespace, same 1Password item keys
• New bucket: homelab-gitea-backup

Manifest Files Created

File	Purpose
clusters/main/gitea/backup-rbac.yaml	ServiceAccount + Role (pods/exec, pods/get) + RoleBinding
clusters/main/gitea/backup-s3-secret.yaml	ExternalSecret → 1Password postgres-s3-backup creds
clusters/main/gitea/backup-cronjob.yaml	CronJob: 2 initContainers (files + db) + 1 upload container

Commit: 1679786

CronJob Spec

• Schedule: 30 3 * * * (03:30 UTC — 30 min after InfluxDB backup)
• initContainer 1 (gitea-files): bitnami/kubectl:1.31 — kubectl exec tar + kubectl cp
• initContainer 2 (gitea-db): mysql:8.0 — mysqldump against Galera
• Container (upload): amazon/aws-cli:2.35.22 — aws s3 cp to Norris S3
• Shared volume: emptyDir (passed between all containers)
• ServiceAccount: gitea-backup (with RBAC for kubectl exec/cp)

S3 Upload Paths

s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-data.tar.gz
s3://homelab-gitea-backup/<YYYY-MM-DD>/gitea-db.sql.gz

Execution Results

S3 Bucket Creation — CRITICAL: --region us-east-1 not nsc-nbg

Both buckets (homelab-influxdb-backup + homelab-gitea-backup) were created successfully via one-off pods. CRITICAL PITFALL: aws s3 mb with --region nsc-nbg fails with InvalidLocationConstraint: The nsc-nbg location constraint is not valid. Must use --region us-east-1 for bucket creation. The nsc-nbg region works for subsequent aws s3 cp operations but NOT for s3 mb.

# CORRECT — us-east-1 for bucket creation
kubectl run aws-mb-$bucket --image=amazon/aws-cli:2.35.22 --restart=Never \
  --env=AWS_ACCESS_KEY_ID=$AKI --env=AWS_SECRET_ACCESS_KEY=$SAK \
  --command -- sh -c "aws s3 mb s3://$bucket \
    --endpoint-url https://rgw.nbg.nsc.noris.cloud --region us-east-1"

InfluxDB Backup — VERIFIED SUCCESSFUL ✅

InfluxDB backup CronJob ran successfully:

• 77 shards backed up
• backup.tar.gz = 25.4 MB
• Uploaded to s3://homelab-influxdb-backup/2026-07-14/backup.tar.gz
• Total time: ~45 seconds

Gitea Backup — BLOCKED by Image Pull Issues

The Gitea CronJob requires a kubectl image for the initContainer that runs kubectl exec + kubectl cp. Multiple images failed:

1. bitnami/kubectl:1.31 — tag does NOT exist on Docker Hub (failed to resolve reference: docker.io/bitnami/kubectl:1.31: not found)
2. bitnami/kubectl:1.31.4 — same NotFound error
3. bitnami/kubectl:latest — pulling but extremely slow (Docker Hub rate-limit)
4. alpine/k8s:1.31.4 — pulling but took 3+ minutes, still Pending

The amazon/aws-cli:2.35.22 and mysql:8.0 images pull fine (already cached from the InfluxDB backup CronJob). Only the kubectl image is problematic.

kubectl cp Fails on Large Files

Attempting kubectl cp of the 143 MB gitea-data.tar.gz from the mgmt-runner into an upload pod failed with:

write tcp 10.0.30.124:35976->10.0.30.51:6443: write: connection reset by peer
websocket: close 1006 (abnormal closure): unexpected EOF

kubectl cp uses the K8s API websocket to stream file data. Large files (>~50 MB) are unreliable — the websocket connection resets before transfer completes. Small files (<1 MB) work fine.

ArgoCD Reports Success But Doesn't Create New Resources

When new manifest files (backup-rbac.yaml, backup-s3-secret.yaml, backup-cronjob.yaml) were added to the existing gitea-config Application's directory (clusters/main/gitea/), ArgoCD showed:

• Sync status: Succeeded
• Operation phase: Succeeded
• Message: successfully synced (all tasks run)
• BUT: kubectl get cronjob -n gitea returned No resources found

This is a known ArgoCD behavior with ServerSideApply=true — new files in an existing directory aren't always detected on the first sync cycle. The hard-refresh annotation was applied but still showed OutOfSync.

Fix that worked: Transfer the manifest files to VM200 via base64 encoding through SSH, then kubectl apply -f manually. The CronJob, RBAC, and ExternalSecret were all created successfully this way.

# Transfer files from Hermes host to VM200 via base64
B64=$(base64 -w0 /path/to/manifest.yaml)
ssh root@10.0.30.124 "echo '$B64' | base64 -d > /root/iac-homelab/path/to/manifest.yaml"
ssh root@10.0.30.124 "KUBECONFIG=/root/.kube/config kubectl apply -f /root/iac-homelab/path/to/manifest.yaml"

VM200 Repo Remote Not Updated After Gitea Migration

The mgmt-runner's /root/iac-homelab repo still pointed at the old Gitea on CT108 (http://...@10.0.30.105:3000/...). After ArgoCD was switched to K8s Gitea, the VM200 repo became stale. New commits pushed to K8s Gitea were invisible to VM200.

Fix: Update the remote URL on VM200 to point at K8s Gitea, or transfer files via base64 encoding through SSH.

Manual Backup from mgmt-runner — Successful Workaround

When the CronJob pod approach is blocked by image issues, run the backup directly from the mgmt-runner (VM200, 10.0.30.124) which has kubectl + mysqldump already installed:

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 '
KUBECONFIG=/root/.kube/config
set -e

GITEA_POD=$(kubectl -n gitea get pod -l app.kubernetes.io/name=gitea \
  -o jsonpath="{.items[0].metadata.name}")
AKI=$(kubectl -n gitea get secret gitea-s3-backup \
  -o jsonpath="{.data.aws_access_key_id}" | base64 -d)
SAK=$(kubectl -n gitea get secret gitea-s3-backup \
  -o jsonpath="{.data.aws_secret_access_key}" | base64 -d)
DB_PASS=$(kubectl -n gitea get secret gitea-db \
  -o jsonpath="{.data.password}" | base64 -d)
DATE=$(date +%Y-%m-%d)

# 1. tar /data from gitea pod
kubectl -n gitea exec $GITEA_POD -- tar czf /tmp/gitea-data.tar.gz -C / data
kubectl -n gitea cp $GITEA_POD:/tmp/gitea-data.tar.gz /tmp/gitea-data.tar.gz
kubectl -n gitea exec $GITEA_POD -- rm -f /tmp/gitea-data.tar.gz

# 2. mysqldump against Galera
mysqldump --host=10.0.30.70 --port=3306 --user=gitea \
  --password="$DB_PASS" --single-transaction \
  --routines --triggers gitea | gzip > /tmp/gitea-db.sql.gz

# 3. Upload via one-off pod with emptyDir (avoid hostPath — pods run on
#    different nodes!)
#    See pitfall below about hostPath
'

Pitfall: hostPath volumes are node-local. A pod scheduled on worker-01 cannot read files from the mgmt-runner's /tmp via hostPath. The pod lands on whichever node the scheduler chooses. Use kubectl cp to transfer files into a pod with emptyDir, or use nodeName: to pin the pod to a specific node (fragile — not recommended for production).

Pitfall: kubectl cp to upload pod fails for large files. The 143 MB gitea-data.tar.gz caused websocket connection resets. Workaround: split the upload into smaller chunks, or use an initContainer that generates the data directly (avoiding the cp step entirely).

Recommended Fix for CronJob

The CronJob approach needs a kubectl image that can be reliably pulled. Options:

1. Pre-pull the image on all worker nodes — crictl pull alpine/k8s:1.31.4 on each worker, then the CronJob pulls instantly from local cache.
2. Use a different kubectl image from a non-Docker-Hub registry — e.g. registry.k8s.io/kubectl-sidecar or an internally hosted image.
3. Restructure the CronJob to avoid kubectl entirely — mount the PVC read-only (requires PVC to be RWX, or stop the pod first).
4. Run backup as a script on the mgmt-runner via systemd timer or crontab — pragmatic but not GitOps-managed.

Until the image issue is resolved, the manual mgmt-runner approach (above) produces correct backups.

Remaining Steps

1. Create S3 bucket homelab-gitea-backup ✅ Done
2. Create S3 bucket homelab-influxdb-backup ✅ Done
3. InfluxDB backup test ✅ Verified (25.4 MB uploaded)
4. Fix Gitea CronJob kubectl image — pre-pull or use alternative
5. Gitea backup test run — verify both files in S3
6. Monitor first scheduled runs (03:00 + 03:30 UTC)

Pitfalls Summary

bitnami/kubectl:1.31 tag does NOT exist

The bitnami/kubectl image uses full semver tags (1.31.4, not 1.31). However, even with correct tags, Docker Hub rate-limiting causes extremely slow pulls (3+ minutes to Pending). Pre-pull on all workers or use an alternative registry.

S3 bucket creation: --region us-east-1 not nsc-nbg

aws s3 mb with --region nsc-nbg fails with InvalidLocationConstraint. Use --region us-east-1 for bucket creation. The nsc-nbg region works for subsequent aws s3 cp operations.

kubectl cp unreliable for large files (>50 MB)

WebSocket connection resets before transfer completes. Small files (<1 MB) work fine. For large file transfers, consider splitting or avoiding kubectl cp entirely.

VM200 repo remote may be stale after Gitea migration

After switching ArgoCD from CT108 to K8s Gitea, the mgmt-runner's local repo remote still points at the old CT108 URL. New commits are invisible. Fix: git remote set-url origin <new-url> on VM200, or transfer files via base64 through SSH.

hostPath volumes are node-local

A pod using hostPath: /tmp reads from the NODE it's scheduled on, not from the machine that created the files. If files are on the mgmt-runner but the pod runs on a worker, the files are invisible. Use emptyDir + kubectl cp, or pin the pod with nodeName: (fragile).

RBAC required for kubectl exec/cp

Without explicit Role granting pods/exec + pods/get, the backup pod's kubectl commands fail with forbidden. Must create ServiceAccount, Role, and RoleBinding before the CronJob can function.

kubectl cp namespace syntax

kubectl cp requires namespace/pod:path format. Missing the namespace prefix causes failures when the backup pod and target pod are in different namespaces (though typically same namespace here).

Pattern Reuse

This pattern (PVC tar via kubectl exec + external DB dump + S3 upload) applies to any K8s service with both PVC data and an external database:

• Gitea (PVC + Galera)
• Future services with similar dual-data-store architecture

For services with only PVC data (no external DB), skip the mysqldump initContainer. For services with only an external DB (no PVC), skip the kubectl exec initContainer.