# Hermes Host Git Remote Cutover to K8s Gitea — 2026-07-14

## Context

After ArgoCD self-management cutover (Section 4.3e), the Hermes host's
own `iac-homelab` git remote still pointed to CT108
(`10.0.30.105:3000`). User instructed: "nutze du intern in hermes ab
jetzt auch das k8s gitea" — switch Hermes host's git remote to K8s Gitea.

## Key Learnings

### CoreDNS Is K8s-Internal Only

The CoreDNS `hosts` override (Section 4.3d) makes `git.schoen.codes`
resolve to `10.0.30.203` **only inside K8s pods**. The Hermes host,
being outside K8s, cannot use CoreDNS. Public DNS was explicitly
rejected by the user ("Nutze für Internet Zwecke nicht den externen DNS
Resolver, sondern einen internen").

Solution: `/etc/hosts` entry on the Hermes host:
```bash
echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts
```

### Gitea API Tokens Do NOT Migrate

When repos are migrated from CT108 Gitea to K8s Gitea via the Gitea
Migration API, **API tokens are NOT copied**. Tokens are stored hashed
in the database and are instance-specific. The old token
(`07efa...`) from CT108 returns `401 Unauthorized` on K8s Gitea.

New token must be generated on K8s Gitea:
```bash
# Via mgmt-runner (VM200) — Hermes host has no kubectl:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
   gitea admin user generate-access-token \
   -u dominik -t hermes-gitops --scopes all --raw"
```

Note: The `--config /data/gitea/conf/app.ini` flag is optional —
`gitea admin user generate-access-token` auto-discovers the config when
running inside the pod. The command outputs the raw token to stdout.

### JWT ≠ Gitea API Token (Critical Distinction)

The 1Password ExternalSecrets for Gitea (`gitea-security`) contain
`internal_token`, `jwt_secret`, `secret_key`, `lfs_jwt_secret`. These
are **internal security tokens** (JWTs), NOT API access tokens.

**How to distinguish:**
- **Gitea API token**: hex string, e.g. `07efa534ea1a147fce1bf5813c669cb0ebc7b522`
- **JWT / internal security token**: three base64 parts separated by dots,
  e.g. `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3ODM5NjM1NDd9.Lyr6fqc...`

A JWT passed as `Authorization: token <JWT>` or `Authorization: Bearer
<JWT>` returns `401 Unauthorized` on both old and new Gitea instances.
The `nbf` (Not Before) claim in the JWT payload identifies it as a
security token, not an access token.

**If a user provides a token that looks like a JWT (dots separating
base64 segments), it is NOT a Gitea API token.** Direct them to
Settings → Applications → Generate New Token in the Gitea web UI, or
generate one via `gitea admin user generate-access-token` (see above).

### K8s Gitea Reachability from Outside K8s

| Path | Works? | Notes |
|------|--------|-------|
| `http://10.0.30.203:3000` | ❌ Timeout | Port 3000 not exposed on VIP |
| `http://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | Traefik routes by Host header |
| `https://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | TLS via Traefik |
| `10.0.30.202:2222` (SSH) | Separate LB | Gitea SSH LoadBalancer service |
| `git.schoen.codes` (via /etc/hosts) | ✅ | Resolves to 10.0.30.203 |

### Hermes Host Has No kubectl

The Hermes host (`/home/debian`) does not have `kubectl`, `helm`, or
`kubeconfig`. All K8s operations go through SSH to mgmt-runner (VM200)
at `10.0.30.124`. See SKILL.md Section 1.2.

### ArgoCD Repo Secret Also Needs Token Update

After generating a new Gitea API token, the ArgoCD repository secret
must also be patched with the new token — otherwise ArgoCD's repo
connection breaks even though the Hermes host git remote works:

```bash
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl patch secret \
   argocd-repo-k8s-gitea -n argocd --type merge \
   -p '{\"data\":{\"password\":\"$(echo -n TOKEN | base64)\"}}'"
```

### Gitea Helm Chart Admin Password Extraction

The Gitea Helm Chart v12 auto-generates an admin password on first
install (unless explicitly set in values). The password is stored as
the `GITEA_ADMIN_PASSWORD` environment variable in the `configure-gitea`
init container — NOT in any K8s Secret or Helm values file.

```bash
# Extract admin password from the init container env vars:
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
  "KUBECONFIG=/root/.kube/config kubectl get deploy gitea -n gitea \
   -o jsonpath='{.spec.template.spec.initContainers[?(@.name==\"configure-gitea\")].env}'" \
  | python3 -c "
import json,sys
for e in json.load(sys.stdin):
    if e.get('name')=='GITEA_ADMIN_PASSWORD':
        print(e['value'])
"
```

Additional env vars in the same container:
- `GITEA_ADMIN_USERNAME`: the admin username (e.g. `dominik`)
- `GITEA_ADMIN_PASSWORD_MODE`: `keepUpdated` (chart syncs password on
  every sync) or `initialOnlyRequireReset`

**Pitfall:** The password is NOT in the `gitea-security` K8s Secret
(that contains JWT-based internal tokens only — see "JWT ≠ Gitea API
Token" above). It is also NOT in the Helm values `gitea.admin` section
(unless explicitly set — by default the chart generates a random one).

### Storing Credentials in 1Password

After extracting the admin password and generating an API token, store
both in 1Password vault "Hermes":

- **API Token**: Category `API Credential`, title `K8s Gitea Token
  (hermes-gitops)`. Requires two-step create+edit (see 1password-cli
  skill — `op item create` does not populate the `credential` field
  for API Credential items).
- **Admin Login**: Category `Login`, title `K8s Gitea Admin (dominik)`.
  Works in one step with `op item create --category="Login"`.

## Steps Performed (Completed)

1. Checked current remote: `git remote -v` → CT108 (`10.0.30.105:3000`)
2. Verified K8s Gitea reachable via Traefik VIP: `curl -H "Host: git.schoen.codes" http://10.0.30.203/` → 200
3. Added `/etc/hosts` entry: `10.0.30.203 git.schoen.codes`
4. Changed git remote URL to `http://dominik:TOKEN@git.schoen.codes/dominik/iac-homelab.git`
5. `git fetch origin` → `Authentication failed` (old CT108 token invalid on K8s Gitea)
6. User provided a JWT (internal security token from 1Password) — identified as NOT an API token (see pitfall above)
7. Generated new API token via mgmt-runner SSH: `gitea admin user generate-access-token -u dominik -t hermes-gitops --scopes all --raw` → `4a3fd...`
8. Updated git remote with new token: `git remote set-url origin "http://dominik:4a3fd...@git.schoen.codes/..."`
9. `git fetch origin` → ✅ success (forced update from `d9135ca` → `68adfab`)
10. `git push --dry-run origin main` → ✅ "Everything up-to-date"
11. Patched ArgoCD repo secret `argocd-repo-k8s-gitea` with new token

## Result

Hermes host git operations now use K8s Gitea exclusively. ArgoCD also
pulls from K8s Gitea. The entire GitOps loop is self-contained in K8s.
CT108 can be decommissioned (Task 10 — pending user approval).