dominik/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/devops/proxmox-ve-administration/references/avahi-reflector-setup.md
T

12 KiB
Raw Permalink Blame History

Avahi mDNS Reflector Setup — Inter-VLAN Bonjour Forwarding

When to Use

When Bonjour/mDNS service discovery (Time Machine, AirPlay, printers) needs to cross VLAN boundaries. Omada does not provide mDNS relay; a dedicated CT with interfaces in both VLANs runs avahi-daemon in reflector mode.

Architecture

VLAN 30 (Servers)  ← eth0 (10.0.30.5) ─┐
                                         │ avahi-daemon (reflector)
VLAN 40 (Clients)  ← eth1 (10.0.40.5) ─┘

CT 127 (avahi-reflector) on proxmox6 (10.0.20.60), unprivileged, 1 core, 512 MB, 4 GB rootfs on hdd_disk.

PVE Config

hostname: avahi-reflector
net0: name=eth0,bridge=vmbr0,gw=10.0.30.1,hwaddr=BC:24:11:1B:13:59,ip=10.0.30.5/24,tag=30,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=BC:24:11:37:40:10,ip=10.0.40.5/24,tag=40,type=veth
onboot: 1
features: nesting=1

Both interfaces on vmbr0 with VLAN tags — the PVE bridge trunks to the switch, and VLAN tagging separates the networks at L2.

Avahi Daemon Configuration

/etc/avahi/avahi-daemon.conf key settings:

[server]
use-ipv4=yes
use-ipv6=yes
allow-interfaces=eth0,eth1

[reflector]
enable-reflector=yes
  • allow-interfaces must list BOTH interfaces explicitly.
  • enable-reflector=yes is the critical flag — default is no.
  • No service files in /etc/avahi/services/ — this CT only reflects, does not publish its own services.

Unidirectional Reflection (30→40 only, block 40→30)

By default avahi-reflector is bidirectional. To make it unidirectional (reflect server VLAN → client VLAN only, prevent client VLAN mDNS from leaking into server VLAN), use iptables to drop incoming mDNS on the client-facing interface:

# IPv4: drop mDNS multicast arriving on eth1 (VLAN 40 / client)
iptables -A INPUT -i eth1 -p udp --dport 5353 -d 224.0.0.251 -j DROP

# IPv6: same
ip6tables -A INPUT -i eth1 -p udp --dport 5353 -d ff02::fb -j DROP

Why This Works

  • avahi-daemon listens on both interfaces for incoming mDNS.
  • The DROP rule on eth1 INPUT prevents avahi from ever SEEING packets from VLAN 40, so it never reflects them to VLAN 30.
  • Outgoing reflected packets on eth1 (30→40 direction) are OUTPUT chain, unaffected by the INPUT rule.
  • Result: VLAN 40 clients discover VLAN 30 services (Time Machine, SMB), but VLAN 40 mDNS noise (iPhone, iPad advertisements) does NOT leak into VLAN 30.

Persistence

Install iptables-persistent and save rules:

apt-get install -y iptables-persistent
mkdir -p /etc/iptables
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

Verification

# Restart to flush cache
systemctl restart avahi-daemon

# Services on eth0 (VLAN 30) should show ONLY server-side services
# (no iPhone/iPad/client devices from VLAN 40)
timeout 10 avahi-browse -art | grep eth0 | awk '{print $4, $5}' | sort -u

# Services on eth1 (VLAN 40) should show reflected server services
# (smb-tm-sarah, smb-media, etc.)
timeout 10 avahi-browse -art | grep eth1 | awk '{print $4, $5}' | sort -u

Pitfall: iptables Binary Missing in Minimal CT

Debian minimal CTs may have iptables package installed but no iptables symlink in PATH — only iptables-nft and iptables-legacy exist. Fix:

ln -sf /usr/sbin/iptables-nft /usr/sbin/iptables
ln -sf /usr/sbin/ip6tables-nft /usr/sbin/ip6tables

Pitfall: Interfaces DOWN After CT Start

If both eth0 and eth1 show state DOWN after pct start, bring them up manually:

ip link set eth0 up
ip link set eth1 up

This can happen when the CT was previously stopped with interfaces configured but networkd hasn't fully initialized. avahi-daemon will fail silently if interfaces are down — check ip addr show before debugging avahi.

Avahi Service File for Time Machine (CT 138)

CT 138 (smb-tm-sarah) publishes its own Time Machine share via a local avahi-daemon. The service file at /etc/avahi/services/timemachine.service:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="Yes">smb-tm-sarah</name>
  <service>
    <type>_smb._tcp</type>
    <port>445</port>
  </service>
  <service>
    <type>_adisk._tcp</type>
    <txt-record>sys=waMa=0,adVF=0x82</txt-record>
    <txt-record>dk0=adVN=sarah-backup,adVk=1</txt-record>
  </service>
</service-group>

Pitfall: %h Wildcard Shows Literally

Using %h as the service name with replace-wildcards="Yes" should resolve to the hostname, but in unprivileged CTs it sometimes appears literally as %h in browse results. Replace with a literal hostname (e.g. smb-tm-sarah) for clean display on macOS.

Unprivileged CT Media Mount Permissions

When a media volume is formatted from outside the CT (e.g. via pct exec running mke2fs on a mapped RBD device), the filesystem is owned by root:root (uid 0) in the host namespace. Inside an unprivileged CT, host uid 0 maps to uid 100000 — but the filesystem shows nobody:nogroup with no write permissions for the CT's internal root.

Fix: nsenter from Host to Chown

# Get the CT's init PID
INIT_PID=$(pct config 138 | grep "^rootfs" | ...)
# Or simply:
INIT_PID=$(cat /var/lib/lxc/138/lxc.conf 2>/dev/null | ...)  # unreliable

# Better: use the host-visible mount point
# The RBD device is mapped on the host as /dev/rbdN
# Find it:
RBD_DEV=$(rbd showmapped | grep "vm-138-disk-1" | awk '{print $NF}')

# Mount temporarily on host, chown to 100000:100000 (maps to root inside CT)
mount $RBD_DEV /mnt/tmp
chown -R 100000:100000 /mnt/tmp
umount /mnt/tmp

Alternative: nsenter into CT namespace

# Get init PID
PID=$(pgrep -f "lxc.*138" | head -1)
# Or from pct status output
nsenter -t $PID -m -- chown -R root:root /mnt/tm

The nsenter approach enters the CT's mount namespace where uid 0 is the CT's internal root, so chown root:root sets the correct ownership.

Multi-VLAN Directional Matrix (3+ VLANs)

When more than two VLANs need selective reflection, the direction matrix becomes non-trivial. Avahi's reflector is all-or-nothing per interface pair — iptables controls which directions are allowed.

Example: 30→40, 50→30, 50→40 (block all reverse)

Desired flows:

From → To Allowed?
30 → 40 ✅
50 → 30 ✅
50 → 40 ✅
40 → 30 ❌
40 → 50 ❌
30 → 50 ❌

Architecture (CT 127 with 3 interfaces):

VLAN 30 (Servers)     ← eth0 (10.0.30.5)  ─┐
VLAN 40 (Clients)     ← eth1 (10.0.40.5)  ─┤  avahi-daemon (reflector)
VLAN 50 (IoT)         ← eth2 (10.0.50.5)  ─┘

PVE Config (3 interfaces)

net0: name=eth0,bridge=vmbr0,gw=10.0.30.1,hwaddr=BC:24:11:1B:13:59,ip=10.0.30.5/24,tag=30,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=BC:24:11:37:40:10,ip=10.0.40.5/24,tag=40,type=veth
net2: name=eth2,bridge=vmbr0,hwaddr=BC:24:11:50:50:05,ip=10.0.50.5/24,tag=50,type=veth
onboot: 1

iptables Rules for Asymmetric 3-VLAN Matrix

# INPUT: block mDNS arriving on eth1 (VLAN 40 / clients)
# Prevents 40→30 and 40→50 reflection
iptables -A INPUT -i eth1 -p udp --dport 5353 -d 224.0.0.251 -j DROP
ip6tables -A INPUT -i eth1 -p udp --dport 5353 -d ff02::fb -j DROP

# OUTPUT: block mDNS leaving on eth2 (VLAN 50 / IoT)
# Prevents 30→50 and 40→50 reflection (nothing should leak INTO IoT)
iptables -A OUTPUT -o eth2 -p udp --dport 5353 -d 224.0.0.251 -j DROP
ip6tables -A OUTPUT -o eth2 -p udp --dport 5353 -d ff02::fb -j DROP

Logic:

  • eth0 (VLAN 30): INPUT open, OUTPUT open → 30 services get reflected OUT
  • eth1 (VLAN 40): INPUT DROP, OUTPUT open → receives 30+50, sends nothing back
  • eth2 (VLAN 50): INPUT open, OUTPUT DROP → 50 services get reflected OUT, nothing comes IN

Updating allow-interfaces for 3+ NICs

sed -i 's/allow-interfaces=eth0,eth1/allow-interfaces=eth0,eth1,eth2/' /etc/avahi/avahi-daemon.conf
systemctl restart avahi-daemon

Verification: iptables Counter Method

avahi-browse cannot reliably distinguish reflected vs local packets when the socket binds to 0.0.0.0. Use iptables packet counters instead:

# Add ACCEPT counting rules for outgoing mDNS per interface
iptables -I OUTPUT -o eth0 -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
iptables -I OUTPUT -o eth1 -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
# (eth2 already has DROP rule)

# Reset counters
iptables -Z OUTPUT

# Wait 10 seconds for traffic
sleep 10

# Check counters — non-zero = reflection active
iptables -L OUTPUT -n -v | grep 5353

Expected output (10s capture):

 6  2730 ACCEPT  udp -- * eth1  0.0.0.0/0  224.0.0.251  udp dpt:5353
 3   513 ACCEPT  udp -- * eth0  0.0.0.0/0  224.0.0.251  udp dpt:5353
 7  2579 DROP    udp -- * eth2  0.0.0.0/0  224.0.0.251  udp dpt:5353
  • eth0 counter > 0 → 50→30 reflection working ✅
  • eth1 counter > 0 → 30→40 + 50→40 reflection working ✅
  • eth2 DROP counter > 0 → 30→50 blocked ✅

Pitfall: apt Lock Contention in Minimal CT

Installing iptables-persistent right after another apt-get install fails with "Could not get lock /var/lib/dpkg/lock-frontend". Wait for the previous apt process to finish:

while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 2; done

Then retry the install. The iptables rules added before the install are still in the running kernel — only persistence (save to /etc/iptables/rules.v4) needs the package.

Pitfall: iptables Rules Lost on CT Restart

iptables-persistent loads rules from /etc/iptables/rules.v{4,6} at boot via systemd netfilter-persistent.service. If rules vanish after pct stop/start, verify:

  1. iptables-persistent is installed: dpkg -l | grep iptables-persistent
  2. Rules file exists: ls -la /etc/iptables/rules.v4
  3. Service is enabled: systemctl is-enabled netfilter-persistent
  4. Re-save after any manual rule change: iptables-save > /etc/iptables/rules.v4

Terminology: "HA" in Proxmox Context Means ha-manager, NOT Home Assistant

When the user says "Ist X im HA?" or refers to "HA" in a Proxmox/infrastructure context, they mean PVE's ha-manager (High Availability cluster resource manager), NOT Home Assistant. Do NOT query the Home Assistant API or look for HA entity states.

# Correct: check if a CT/VM is managed by PVE HA
ha-manager status | grep ct:NNN
# If listed → CT is HA-managed (auto-restart on node failure)
# If absent → CT is NOT HA-managed (manual restart only)

# To add a CT to HA manager
ha-manager add ct:127 --group 1 --max_restart 1 --max_relocate 1

# To list all HA-managed resources
ha-manager status

As of 2026-07-05, CT 127 (avahi-reflector) is NOT in ha-manager. HA-managed CTs: 104, 108, 116, 136, 137, 99999. HA-managed VMs: 106, 118, 128-132, 230, 300-302, 310-311.

Session Log — 2026-07-05

Phase 1: CT 127 Discovery & Repair

  • User indicated existing avahi-reflector CT — found CT 127 on proxmox6 (stopped).
  • CT 127 had enable-reflector=no and both interfaces DOWN.
  • Fixed: enabled reflector, brought interfaces up, started avahi-daemon.
  • Added unidirectional filtering (iptables DROP on eth1 INPUT for mDNS).
  • Installed iptables-persistent for rule survival across reboots.
  • Set onboot: 1 on CT 127 so it auto-starts after PVE reboots.
  • Verified: VLAN 30 services (smb-tm-sarah, smb-media) visible in VLAN 40; VLAN 40 client mDNS (iPhone) NOT leaking into VLAN 30.
  • Fixed %h → smb-tm-sarah in CT 138 avahi service file.

Phase 2: Three-VLAN Extension (50→30, 50→40)

  • User requested adding VLAN 50 (IoT) reflection to 30 and 40.
  • Added eth2 (VLAN 50, 10.0.50.5/24) to CT 127 via pct set 127 --net2.
  • Updated allow-interfaces=eth0,eth1,eth2 in avahi-daemon.conf.
  • Added OUTPUT DROP on eth2 to block any→50 reflection.
  • Verified via iptables counters: eth0=3 pkts, eth1=6 pkts (reflection active), eth2=7 pkts DROP (blocked). Direction matrix confirmed correct.
Reference in New Issue View Git Blame Copy Permalink