hermes-skills/devops/rke2-cluster-administration/references/hermes-git-remote-cutover-2026-07.md

Hermes Host Git Remote Cutover to K8s Gitea — 2026-07-14

Context

After ArgoCD self-management cutover (Section 4.3e), the Hermes host's own iac-homelab git remote still pointed to CT108 (10.0.30.105:3000). User instructed: "nutze du intern in hermes ab jetzt auch das k8s gitea" — switch Hermes host's git remote to K8s Gitea.

Key Learnings

CoreDNS Is K8s-Internal Only

The CoreDNS hosts override (Section 4.3d) makes git.schoen.codes resolve to 10.0.30.203 only inside K8s pods. The Hermes host, being outside K8s, cannot use CoreDNS. Public DNS was explicitly rejected by the user ("Nutze für Internet Zwecke nicht den externen DNS Resolver, sondern einen internen").

Solution: /etc/hosts entry on the Hermes host:

echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts

Gitea API Tokens Do NOT Migrate

When repos are migrated from CT108 Gitea to K8s Gitea via the Gitea Migration API, API tokens are NOT copied. Tokens are stored hashed in the database and are instance-specific. The old token (07efa...) from CT108 returns 401 Unauthorized on K8s Gitea.

New token must be generated on K8s Gitea:

# Via mgmt-runner (VM200) — Hermes host has no kubectl:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
   gitea admin user generate-access-token \
   -u dominik -t hermes-gitops --scopes all --raw"

Note: The --config /data/gitea/conf/app.ini flag is optional — gitea admin user generate-access-token auto-discovers the config when running inside the pod. The command outputs the raw token to stdout.

JWT ≠ Gitea API Token (Critical Distinction)

The 1Password ExternalSecrets for Gitea (gitea-security) contain internal_token, jwt_secret, secret_key, lfs_jwt_secret. These are internal security tokens (JWTs), NOT API access tokens.

How to distinguish:

• Gitea API token: hex string, e.g. 07efa534ea1a147fce1bf5813c669cb0ebc7b522
• JWT / internal security token: three base64 parts separated by dots, e.g. eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3ODM5NjM1NDd9.Lyr6fqc...

A JWT passed as Authorization: token <JWT> or Authorization: Bearer <JWT> returns 401 Unauthorized on both old and new Gitea instances. The nbf (Not Before) claim in the JWT payload identifies it as a security token, not an access token.

If a user provides a token that looks like a JWT (dots separating base64 segments), it is NOT a Gitea API token. Direct them to Settings → Applications → Generate New Token in the Gitea web UI, or generate one via gitea admin user generate-access-token (see above).

K8s Gitea Reachability from Outside K8s

Path	Works?	Notes
http://10.0.30.203:3000	❌ Timeout	Port 3000 not exposed on VIP
http://10.0.30.203 (Host: git.schoen.codes)	✅ 200	Traefik routes by Host header
https://10.0.30.203 (Host: git.schoen.codes)	✅ 200	TLS via Traefik
10.0.30.202:2222 (SSH)	Separate LB	Gitea SSH LoadBalancer service
git.schoen.codes (via /etc/hosts)	✅	Resolves to 10.0.30.203

Hermes Host Has No kubectl

The Hermes host (/home/debian) does not have kubectl, helm, or kubeconfig. All K8s operations go through SSH to mgmt-runner (VM200) at 10.0.30.124. See SKILL.md Section 1.2.

ArgoCD Repo Secret Also Needs Token Update

After generating a new Gitea API token, the ArgoCD repository secret must also be patched with the new token — otherwise ArgoCD's repo connection breaks even though the Hermes host git remote works:

ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl patch secret \
   argocd-repo-k8s-gitea -n argocd --type merge \
   -p '{\"data\":{\"password\":\"$(echo -n TOKEN | base64)\"}}'"

Gitea Helm Chart Admin Password Extraction

The Gitea Helm Chart v12 auto-generates an admin password on first install (unless explicitly set in values). The password is stored as the GITEA_ADMIN_PASSWORD environment variable in the configure-gitea init container — NOT in any K8s Secret or Helm values file.

# Extract admin password from the init container env vars:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl get deploy gitea -n gitea \
   -o jsonpath='{.spec.template.spec.initContainers[?(@.name==\"configure-gitea\")].env}'" \
  | python3 -c "
import json,sys
for e in json.load(sys.stdin):
    if e.get('name')=='GITEA_ADMIN_PASSWORD':
        print(e['value'])
"

Additional env vars in the same container:

• GITEA_ADMIN_USERNAME: the admin username (e.g. dominik)
• GITEA_ADMIN_PASSWORD_MODE: keepUpdated (chart syncs password on every sync) or initialOnlyRequireReset

Pitfall: The password is NOT in the gitea-security K8s Secret (that contains JWT-based internal tokens only — see "JWT ≠ Gitea API Token" above). It is also NOT in the Helm values gitea.admin section (unless explicitly set — by default the chart generates a random one).

Storing Credentials in 1Password

After extracting the admin password and generating an API token, store both in 1Password vault "Hermes":

• API Token: Category API Credential, title K8s Gitea Token (hermes-gitops). Requires two-step create+edit (see 1password-cli skill — op item create does not populate the credential field for API Credential items).
• Admin Login: Category Login, title K8s Gitea Admin (dominik). Works in one step with op item create --category="Login".

Steps Performed (Completed)

1. Checked current remote: git remote -v → CT108 (10.0.30.105:3000)
2. Verified K8s Gitea reachable via Traefik VIP: curl -H "Host: git.schoen.codes" http://10.0.30.203/ → 200
3. Added /etc/hosts entry: 10.0.30.203 git.schoen.codes
4. Changed git remote URL to http://dominik:TOKEN@git.schoen.codes/dominik/iac-homelab.git
5. git fetch origin → Authentication failed (old CT108 token invalid on K8s Gitea)
6. User provided a JWT (internal security token from 1Password) — identified as NOT an API token (see pitfall above)
7. Generated new API token via mgmt-runner SSH: gitea admin user generate-access-token -u dominik -t hermes-gitops --scopes all --raw → 4a3fd...
8. Updated git remote with new token: git remote set-url origin "http://dominik:4a3fd...@git.schoen.codes/..."
9. git fetch origin → ✅ success (forced update from d9135ca → 68adfab)
10. git push --dry-run origin main → ✅ "Everything up-to-date"
11. Patched ArgoCD repo secret argocd-repo-k8s-gitea with new token

Result

Hermes host git operations now use K8s Gitea exclusively. ArgoCD also pulls from K8s Gitea. The entire GitOps loop is self-contained in K8s. CT108 can be decommissioned (Task 10 — pending user approval).