Debian
01bd921ced
- New: smart-home/home-assistant-dashboard-conventions (Mushroom cards, view tabs, no Bubble Cards) - Updated: rke2, ceph, galera, proxmox, brainstorming, compound-learning, 1password-cli, smart-home-automation skills - New references: ceph-cluster-administration, docker-volume-forensics, ceph-crush-weight, ceph-ec-mixed-size
156 lines
6.8 KiB
Markdown
156 lines
6.8 KiB
Markdown
# Hermes Host Git Remote Cutover to K8s Gitea — 2026-07-14
|
|
|
|
## Context
|
|
|
|
After ArgoCD self-management cutover (Section 4.3e), the Hermes host's
|
|
own `iac-homelab` git remote still pointed to CT108
|
|
(`10.0.30.105:3000`). User instructed: "nutze du intern in hermes ab
|
|
jetzt auch das k8s gitea" — switch Hermes host's git remote to K8s Gitea.
|
|
|
|
## Key Learnings
|
|
|
|
### CoreDNS Is K8s-Internal Only
|
|
|
|
The CoreDNS `hosts` override (Section 4.3d) makes `git.schoen.codes`
|
|
resolve to `10.0.30.203` **only inside K8s pods**. The Hermes host,
|
|
being outside K8s, cannot use CoreDNS. Public DNS was explicitly
|
|
rejected by the user ("Nutze für Internet Zwecke nicht den externen DNS
|
|
Resolver, sondern einen internen").
|
|
|
|
Solution: `/etc/hosts` entry on the Hermes host:
|
|
```bash
|
|
echo "10.0.30.203 git.schoen.codes" | sudo tee -a /etc/hosts
|
|
```
|
|
|
|
### Gitea API Tokens Do NOT Migrate
|
|
|
|
When repos are migrated from CT108 Gitea to K8s Gitea via the Gitea
|
|
Migration API, **API tokens are NOT copied**. Tokens are stored hashed
|
|
in the database and are instance-specific. The old token
|
|
(`07efa...`) from CT108 returns `401 Unauthorized` on K8s Gitea.
|
|
|
|
New token must be generated on K8s Gitea:
|
|
```bash
|
|
# Via mgmt-runner (VM200) — Hermes host has no kubectl:
|
|
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
|
"KUBECONFIG=/root/.kube/config kubectl exec -n gitea deploy/gitea -- \
|
|
gitea admin user generate-access-token \
|
|
-u dominik -t hermes-gitops --scopes all --raw"
|
|
```
|
|
|
|
Note: The `--config /data/gitea/conf/app.ini` flag is optional —
|
|
`gitea admin user generate-access-token` auto-discovers the config when
|
|
running inside the pod. The command outputs the raw token to stdout.
|
|
|
|
### JWT ≠ Gitea API Token (Critical Distinction)
|
|
|
|
The 1Password ExternalSecrets for Gitea (`gitea-security`) contain
|
|
`internal_token`, `jwt_secret`, `secret_key`, `lfs_jwt_secret`. These
|
|
are **internal security tokens** (JWTs), NOT API access tokens.
|
|
|
|
**How to distinguish:**
|
|
- **Gitea API token**: hex string, e.g. `07efa534ea1a147fce1bf5813c669cb0ebc7b522`
|
|
- **JWT / internal security token**: three base64 parts separated by dots,
|
|
e.g. `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3ODM5NjM1NDd9.Lyr6fqc...`
|
|
|
|
A JWT passed as `Authorization: token <JWT>` or `Authorization: Bearer
|
|
<JWT>` returns `401 Unauthorized` on both old and new Gitea instances.
|
|
The `nbf` (Not Before) claim in the JWT payload identifies it as a
|
|
security token, not an access token.
|
|
|
|
**If a user provides a token that looks like a JWT (dots separating
|
|
base64 segments), it is NOT a Gitea API token.** Direct them to
|
|
Settings → Applications → Generate New Token in the Gitea web UI, or
|
|
generate one via `gitea admin user generate-access-token` (see above).
|
|
|
|
### K8s Gitea Reachability from Outside K8s
|
|
|
|
| Path | Works? | Notes |
|
|
|------|--------|-------|
|
|
| `http://10.0.30.203:3000` | ❌ Timeout | Port 3000 not exposed on VIP |
|
|
| `http://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | Traefik routes by Host header |
|
|
| `https://10.0.30.203` (Host: git.schoen.codes) | ✅ 200 | TLS via Traefik |
|
|
| `10.0.30.202:2222` (SSH) | Separate LB | Gitea SSH LoadBalancer service |
|
|
| `git.schoen.codes` (via /etc/hosts) | ✅ | Resolves to 10.0.30.203 |
|
|
|
|
### Hermes Host Has No kubectl
|
|
|
|
The Hermes host (`/home/debian`) does not have `kubectl`, `helm`, or
|
|
`kubeconfig`. All K8s operations go through SSH to mgmt-runner (VM200)
|
|
at `10.0.30.124`. See SKILL.md Section 1.2.
|
|
|
|
### ArgoCD Repo Secret Also Needs Token Update
|
|
|
|
After generating a new Gitea API token, the ArgoCD repository secret
|
|
must also be patched with the new token — otherwise ArgoCD's repo
|
|
connection breaks even though the Hermes host git remote works:
|
|
|
|
```bash
|
|
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
|
"KUBECONFIG=/root/.kube/config kubectl patch secret \
|
|
argocd-repo-k8s-gitea -n argocd --type merge \
|
|
-p '{\"data\":{\"password\":\"$(echo -n TOKEN | base64)\"}}'"
|
|
```
|
|
|
|
### Gitea Helm Chart Admin Password Extraction
|
|
|
|
The Gitea Helm Chart v12 auto-generates an admin password on first
|
|
install (unless explicitly set in values). The password is stored as
|
|
the `GITEA_ADMIN_PASSWORD` environment variable in the `configure-gitea`
|
|
init container — NOT in any K8s Secret or Helm values file.
|
|
|
|
```bash
|
|
# Extract admin password from the init container env vars:
|
|
ssh -i ~/.ssh/id_ed25519_proxmox root@10.0.30.124 \
|
|
"KUBECONFIG=/root/.kube/config kubectl get deploy gitea -n gitea \
|
|
-o jsonpath='{.spec.template.spec.initContainers[?(@.name==\"configure-gitea\")].env}'" \
|
|
| python3 -c "
|
|
import json,sys
|
|
for e in json.load(sys.stdin):
|
|
if e.get('name')=='GITEA_ADMIN_PASSWORD':
|
|
print(e['value'])
|
|
"
|
|
```
|
|
|
|
Additional env vars in the same container:
|
|
- `GITEA_ADMIN_USERNAME`: the admin username (e.g. `dominik`)
|
|
- `GITEA_ADMIN_PASSWORD_MODE`: `keepUpdated` (chart syncs password on
|
|
every sync) or `initialOnlyRequireReset`
|
|
|
|
**Pitfall:** The password is NOT in the `gitea-security` K8s Secret
|
|
(that contains JWT-based internal tokens only — see "JWT ≠ Gitea API
|
|
Token" above). It is also NOT in the Helm values `gitea.admin` section
|
|
(unless explicitly set — by default the chart generates a random one).
|
|
|
|
### Storing Credentials in 1Password
|
|
|
|
After extracting the admin password and generating an API token, store
|
|
both in 1Password vault "Hermes":
|
|
|
|
- **API Token**: Category `API Credential`, title `K8s Gitea Token
|
|
(hermes-gitops)`. Requires two-step create+edit (see 1password-cli
|
|
skill — `op item create` does not populate the `credential` field
|
|
for API Credential items).
|
|
- **Admin Login**: Category `Login`, title `K8s Gitea Admin (dominik)`.
|
|
Works in one step with `op item create --category="Login"`.
|
|
|
|
## Steps Performed (Completed)
|
|
|
|
1. Checked current remote: `git remote -v` → CT108 (`10.0.30.105:3000`)
|
|
2. Verified K8s Gitea reachable via Traefik VIP: `curl -H "Host: git.schoen.codes" http://10.0.30.203/` → 200
|
|
3. Added `/etc/hosts` entry: `10.0.30.203 git.schoen.codes`
|
|
4. Changed git remote URL to `http://dominik:TOKEN@git.schoen.codes/dominik/iac-homelab.git`
|
|
5. `git fetch origin` → `Authentication failed` (old CT108 token invalid on K8s Gitea)
|
|
6. User provided a JWT (internal security token from 1Password) — identified as NOT an API token (see pitfall above)
|
|
7. Generated new API token via mgmt-runner SSH: `gitea admin user generate-access-token -u dominik -t hermes-gitops --scopes all --raw` → `4a3fd...`
|
|
8. Updated git remote with new token: `git remote set-url origin "http://dominik:4a3fd...@git.schoen.codes/..."`
|
|
9. `git fetch origin` → ✅ success (forced update from `d9135ca` → `68adfab`)
|
|
10. `git push --dry-run origin main` → ✅ "Everything up-to-date"
|
|
11. Patched ArgoCD repo secret `argocd-repo-k8s-gitea` with new token
|
|
|
|
## Result
|
|
|
|
Hermes host git operations now use K8s Gitea exclusively. ArgoCD also
|
|
pulls from K8s Gitea. The entire GitOps loop is self-contained in K8s.
|
|
CT108 can be decommissioned (Task 10 — pending user approval).
|