hermes-skills/devops/proxmox-ve-administration/references/avahi-reflector-setup.md

# Avahi mDNS Reflector Setup — Inter-VLAN Bonjour Forwarding

## When to Use

When Bonjour/mDNS service discovery (Time Machine, AirPlay, printers) needs to
cross VLAN boundaries. Omada does not provide mDNS relay; a dedicated CT with
interfaces in both VLANs runs `avahi-daemon` in reflector mode.

## Architecture

```
VLAN 30 (Servers)  ← eth0 (10.0.30.5) ─┐
                                         │ avahi-daemon (reflector)
VLAN 40 (Clients)  ← eth1 (10.0.40.5) ─┘
```

CT 127 (`avahi-reflector`) on proxmox6 (10.0.20.60), unprivileged, 1 core,
512 MB, 4 GB rootfs on `hdd_disk`.

### PVE Config

```
hostname: avahi-reflector
net0: name=eth0,bridge=vmbr0,gw=10.0.30.1,hwaddr=BC:24:11:1B:13:59,ip=10.0.30.5/24,tag=30,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=BC:24:11:37:40:10,ip=10.0.40.5/24,tag=40,type=veth
onboot: 1
features: nesting=1
```

Both interfaces on `vmbr0` with VLAN tags — the PVE bridge trunks to the
switch, and VLAN tagging separates the networks at L2.

## Avahi Daemon Configuration

`/etc/avahi/avahi-daemon.conf` key settings:

```ini
[server]
use-ipv4=yes
use-ipv6=yes
allow-interfaces=eth0,eth1

[reflector]
enable-reflector=yes
```

- `allow-interfaces` must list BOTH interfaces explicitly.
- `enable-reflector=yes` is the critical flag — default is `no`.
- No service files in `/etc/avahi/services/` — this CT only reflects,
  does not publish its own services.

## Unidirectional Reflection (30→40 only, block 40→30)

By default avahi-reflector is bidirectional. To make it unidirectional
(reflect server VLAN → client VLAN only, prevent client VLAN mDNS from
leaking into server VLAN), use iptables to drop incoming mDNS on the
client-facing interface:

```bash
# IPv4: drop mDNS multicast arriving on eth1 (VLAN 40 / client)
iptables -A INPUT -i eth1 -p udp --dport 5353 -d 224.0.0.251 -j DROP

# IPv6: same
ip6tables -A INPUT -i eth1 -p udp --dport 5353 -d ff02::fb -j DROP
```

### Why This Works

- avahi-daemon listens on both interfaces for incoming mDNS.
- The DROP rule on eth1 INPUT prevents avahi from ever SEEING packets
  from VLAN 40, so it never reflects them to VLAN 30.
- Outgoing reflected packets on eth1 (30→40 direction) are OUTPUT chain,
  unaffected by the INPUT rule.
- Result: VLAN 40 clients discover VLAN 30 services (Time Machine, SMB),
  but VLAN 40 mDNS noise (iPhone, iPad advertisements) does NOT leak
  into VLAN 30.

### Persistence

Install `iptables-persistent` and save rules:

```bash
apt-get install -y iptables-persistent
mkdir -p /etc/iptables
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6
```

### Verification

```bash
# Restart to flush cache
systemctl restart avahi-daemon

# Services on eth0 (VLAN 30) should show ONLY server-side services
# (no iPhone/iPad/client devices from VLAN 40)
timeout 10 avahi-browse -art | grep eth0 | awk '{print $4, $5}' | sort -u

# Services on eth1 (VLAN 40) should show reflected server services
# (smb-tm-sarah, smb-media, etc.)
timeout 10 avahi-browse -art | grep eth1 | awk '{print $4, $5}' | sort -u
```

### Pitfall: iptables Binary Missing in Minimal CT

Debian minimal CTs may have `iptables` package installed but no `iptables`
symlink in PATH — only `iptables-nft` and `iptables-legacy` exist. Fix:

```bash
ln -sf /usr/sbin/iptables-nft /usr/sbin/iptables
ln -sf /usr/sbin/ip6tables-nft /usr/sbin/ip6tables
```

### Pitfall: Interfaces DOWN After CT Start

If both eth0 and eth1 show `state DOWN` after `pct start`, bring them up
manually:

```bash
ip link set eth0 up
ip link set eth1 up
```

This can happen when the CT was previously stopped with interfaces
configured but networkd hasn't fully initialized. avahi-daemon will fail
silently if interfaces are down — check `ip addr show` before debugging
avahi.

## Avahi Service File for Time Machine (CT 138)

CT 138 (`smb-tm-sarah`) publishes its own Time Machine share via a local
avahi-daemon. The service file at `/etc/avahi/services/timemachine.service`:

```xml
<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="Yes">smb-tm-sarah</name>
  <service>
    <type>_smb._tcp</type>
    <port>445</port>
  </service>
  <service>
    <type>_adisk._tcp</type>
    <txt-record>sys=waMa=0,adVF=0x82</txt-record>
    <txt-record>dk0=adVN=sarah-backup,adVk=1</txt-record>
  </service>
</service-group>
```

### Pitfall: `%h` Wildcard Shows Literally

Using `%h` as the service name with `replace-wildcards="Yes"` should
resolve to the hostname, but in unprivileged CTs it sometimes appears
literally as `%h` in browse results. Replace with a literal hostname
(e.g. `smb-tm-sarah`) for clean display on macOS.

## Unprivileged CT Media Mount Permissions

When a media volume is formatted from outside the CT (e.g. via `pct exec`
running `mke2fs` on a mapped RBD device), the filesystem is owned by
`root:root` (uid 0) in the host namespace. Inside an unprivileged CT,
host uid 0 maps to uid 100000 — but the filesystem shows `nobody:nogroup`
with no write permissions for the CT's internal root.

### Fix: nsenter from Host to Chown

```bash
# Get the CT's init PID
INIT_PID=$(pct config 138 | grep "^rootfs" | ...)
# Or simply:
INIT_PID=$(cat /var/lib/lxc/138/lxc.conf 2>/dev/null | ...)  # unreliable

# Better: use the host-visible mount point
# The RBD device is mapped on the host as /dev/rbdN
# Find it:
RBD_DEV=$(rbd showmapped | grep "vm-138-disk-1" | awk '{print $NF}')

# Mount temporarily on host, chown to 100000:100000 (maps to root inside CT)
mount $RBD_DEV /mnt/tmp
chown -R 100000:100000 /mnt/tmp
umount /mnt/tmp
```

### Alternative: nsenter into CT namespace

```bash
# Get init PID
PID=$(pgrep -f "lxc.*138" | head -1)
# Or from pct status output
nsenter -t $PID -m -- chown -R root:root /mnt/tm
```

The `nsenter` approach enters the CT's mount namespace where uid 0 is
the CT's internal root, so `chown root:root` sets the correct ownership.

## Multi-VLAN Directional Matrix (3+ VLANs)

When more than two VLANs need selective reflection, the direction matrix
becomes non-trivial. Avahi's reflector is all-or-nothing per interface
pair — iptables controls which directions are allowed.

### Example: 30→40, 50→30, 50→40 (block all reverse)

Desired flows:

| From → To | Allowed? |
|-----------|----------|
| 30 → 40   | ✅ |
| 50 → 30   | ✅ |
| 50 → 40   | ✅ |
| 40 → 30   | ❌ |
| 40 → 50   | ❌ |
| 30 → 50   | ❌ |

Architecture (CT 127 with 3 interfaces):

```
VLAN 30 (Servers)     ← eth0 (10.0.30.5)  ─┐
VLAN 40 (Clients)     ← eth1 (10.0.40.5)  ─┤  avahi-daemon (reflector)
VLAN 50 (IoT)         ← eth2 (10.0.50.5)  ─┘
```

### PVE Config (3 interfaces)

```
net0: name=eth0,bridge=vmbr0,gw=10.0.30.1,hwaddr=BC:24:11:1B:13:59,ip=10.0.30.5/24,tag=30,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=BC:24:11:37:40:10,ip=10.0.40.5/24,tag=40,type=veth
net2: name=eth2,bridge=vmbr0,hwaddr=BC:24:11:50:50:05,ip=10.0.50.5/24,tag=50,type=veth
onboot: 1
```

### iptables Rules for Asymmetric 3-VLAN Matrix

```bash
# INPUT: block mDNS arriving on eth1 (VLAN 40 / clients)
# Prevents 40→30 and 40→50 reflection
iptables -A INPUT -i eth1 -p udp --dport 5353 -d 224.0.0.251 -j DROP
ip6tables -A INPUT -i eth1 -p udp --dport 5353 -d ff02::fb -j DROP

# OUTPUT: block mDNS leaving on eth2 (VLAN 50 / IoT)
# Prevents 30→50 and 40→50 reflection (nothing should leak INTO IoT)
iptables -A OUTPUT -o eth2 -p udp --dport 5353 -d 224.0.0.251 -j DROP
ip6tables -A OUTPUT -o eth2 -p udp --dport 5353 -d ff02::fb -j DROP
```

Logic:
- eth0 (VLAN 30): INPUT open, OUTPUT open → 30 services get reflected OUT
- eth1 (VLAN 40): INPUT DROP, OUTPUT open → receives 30+50, sends nothing back
- eth2 (VLAN 50): INPUT open, OUTPUT DROP → 50 services get reflected OUT, nothing comes IN

### Updating allow-interfaces for 3+ NICs

```bash
sed -i 's/allow-interfaces=eth0,eth1/allow-interfaces=eth0,eth1,eth2/' /etc/avahi/avahi-daemon.conf
systemctl restart avahi-daemon
```

### Verification: iptables Counter Method

`avahi-browse` cannot reliably distinguish reflected vs local packets
when the socket binds to 0.0.0.0. Use iptables packet counters instead:

```bash
# Add ACCEPT counting rules for outgoing mDNS per interface
iptables -I OUTPUT -o eth0 -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
iptables -I OUTPUT -o eth1 -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
# (eth2 already has DROP rule)

# Reset counters
iptables -Z OUTPUT

# Wait 10 seconds for traffic
sleep 10

# Check counters — non-zero = reflection active
iptables -L OUTPUT -n -v | grep 5353
```

Expected output (10s capture):
```
 6  2730 ACCEPT  udp -- * eth1  0.0.0.0/0  224.0.0.251  udp dpt:5353
 3   513 ACCEPT  udp -- * eth0  0.0.0.0/0  224.0.0.251  udp dpt:5353
 7  2579 DROP    udp -- * eth2  0.0.0.0/0  224.0.0.251  udp dpt:5353
```

- eth0 counter > 0 → 50→30 reflection working ✅
- eth1 counter > 0 → 30→40 + 50→40 reflection working ✅
- eth2 DROP counter > 0 → 30→50 blocked ✅

### Pitfall: apt Lock Contention in Minimal CT

Installing `iptables-persistent` right after another `apt-get install`
fails with "Could not get lock /var/lib/dpkg/lock-frontend". Wait for
the previous apt process to finish:

```bash
while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 2; done
```

Then retry the install. The iptables rules added before the install
are still in the running kernel — only persistence (save to
`/etc/iptables/rules.v4`) needs the package.

### Pitfall: iptables Rules Lost on CT Restart

`iptables-persistent` loads rules from `/etc/iptables/rules.v{4,6}` at
boot via systemd `netfilter-persistent.service`. If rules vanish after
`pct stop/start`, verify:
1. `iptables-persistent` is installed: `dpkg -l | grep iptables-persistent`
2. Rules file exists: `ls -la /etc/iptables/rules.v4`
3. Service is enabled: `systemctl is-enabled netfilter-persistent`
4. Re-save after any manual rule change: `iptables-save > /etc/iptables/rules.v4`

## Terminology: "HA" in Proxmox Context Means ha-manager, NOT Home Assistant

When the user says "Ist X im HA?" or refers to "HA" in a Proxmox/infrastructure
context, they mean **PVE's `ha-manager`** (High Availability cluster resource
manager), NOT Home Assistant. Do NOT query the Home Assistant API or look for
HA entity states.

```bash
# Correct: check if a CT/VM is managed by PVE HA
ha-manager status | grep ct:NNN
# If listed → CT is HA-managed (auto-restart on node failure)
# If absent → CT is NOT HA-managed (manual restart only)

# To add a CT to HA manager
ha-manager add ct:127 --group 1 --max_restart 1 --max_relocate 1

# To list all HA-managed resources
ha-manager status
```

As of 2026-07-05, CT 127 (avahi-reflector) is NOT in ha-manager.
HA-managed CTs: 104, 108, 116, 136, 137, 99999.
HA-managed VMs: 106, 118, 128-132, 230, 300-302, 310-311.

## Session Log — 2026-07-05

### Phase 1: CT 127 Discovery & Repair
- User indicated existing avahi-reflector CT — found CT 127 on proxmox6 (stopped).
- CT 127 had `enable-reflector=no` and both interfaces DOWN.
- Fixed: enabled reflector, brought interfaces up, started avahi-daemon.
- Added unidirectional filtering (iptables DROP on eth1 INPUT for mDNS).
- Installed iptables-persistent for rule survival across reboots.
- Set `onboot: 1` on CT 127 so it auto-starts after PVE reboots.
- Verified: VLAN 30 services (smb-tm-sarah, smb-media) visible in VLAN 40;
  VLAN 40 client mDNS (iPhone) NOT leaking into VLAN 30.
- Fixed `%h` → `smb-tm-sarah` in CT 138 avahi service file.

### Phase 2: Three-VLAN Extension (50→30, 50→40)
- User requested adding VLAN 50 (IoT) reflection to 30 and 40.
- Added eth2 (VLAN 50, 10.0.50.5/24) to CT 127 via `pct set 127 --net2`.
- Updated `allow-interfaces=eth0,eth1,eth2` in avahi-daemon.conf.
- Added OUTPUT DROP on eth2 to block any→50 reflection.
- Verified via iptables counters: eth0=3 pkts, eth1=6 pkts (reflection active),
  eth2=7 pkts DROP (blocked). Direction matrix confirmed correct.