228 lines
8.2 KiB
Markdown
228 lines
8.2 KiB
Markdown
---
|
||
name: network-reconnaissance
|
||
description: "Class-level skill for network reconnaissance from containerized and general environments. Covers internal subnet scanning (TCP sweep, port scan, service detection, TLS/SSH analysis), external domain/service reconnaissance (DNS, SSL, HTTP headers, API discovery), and automated recurring scan setups."
|
||
version: 1.0.0
|
||
tags: [network, reconnaissance, scanning, security, dns, ssl, ports, containers]
|
||
---
|
||
|
||
## Overview
|
||
|
||
This umbrella skill consolidates all network scanning and reconnaissance capabilities:
|
||
- **Internal Subnet Scanning** — TCP sweep, port scanning, banner grabbing, SSH/TLS analysis from containers or Python
|
||
- **External Domain Reconnaissance** — DNS, SSL certificates, HTTP headers, API/service discovery
|
||
- **Automated Recurring Scans** — Weekly cronjob-based scanning with change detection
|
||
|
||
No nmap, no root privileges required. Pure Python stdlib (`socket`, `ssl`, `concurrent.futures`) for container environments.
|
||
|
||
---
|
||
|
||
## Section 1: Internal Subnet Scanning
|
||
|
||
### 1.1 Network Layout Discovery
|
||
**Always discover the actual IP format first.** Users may describe segments as "50.x" but the real network may be `10.0.X.Y`.
|
||
|
||
```bash
|
||
ip addr show eth0 | grep inet
|
||
ip route show
|
||
ping -c 1 -W 2 10.0.50.1
|
||
ping -c 1 -W 2 50.0.50.1
|
||
```
|
||
|
||
Common formats:
|
||
- `10.0.X.Y` where X = segment index (e.g., 10.0.20.10 = Hypervisor)
|
||
- Container usually sits in one segment (e.g., 10.0.30.230)
|
||
|
||
### 1.2 Two-Phase Scan Workflow
|
||
|
||
**Phase 1 — Host Discovery (fast TCP sweep)**
|
||
```python
|
||
import socket, concurrent.futures
|
||
PROBE_PORTS = [443, 80, 22, 2222, 3306, 8080, 8443]
|
||
|
||
def host_alive(ip, ports=PROBE_PORTS, timeout=0.3):
|
||
for p in ports:
|
||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||
sock.settimeout(timeout)
|
||
if sock.connect_ex((ip, p)) == 0:
|
||
sock.close(); return True
|
||
sock.close()
|
||
return False
|
||
|
||
with concurrent.futures.ThreadPoolExecutor(100) as pool:
|
||
alive = [ip for ip, ok in zip(segment_ips, pool.map(host_alive, segment_ips)) if ok]
|
||
```
|
||
|
||
**Phase 2 — Deep Scan (per host)**
|
||
```python
|
||
IMPORTANT_PORTS = {
|
||
22: "SSH", 23: "Telnet", 25: "SMTP", 53: "DNS", 80: "HTTP",
|
||
443: "HTTPS/TLS", 1883: "MQTT", 2222: "SSH-alt",
|
||
2375: "Docker", 2376: "Docker-TLS", 3306: "MySQL",
|
||
5432: "PostgreSQL", 6379: "Redis", 8080: "HTTP-alt",
|
||
8443: "HTTPS-alt", 8883: "MQTT-TLS", 9090: "Prometheus/Grafana",
|
||
9200: "Elasticsearch", 11211: "Memcached", 27017: "MongoDB",
|
||
}
|
||
|
||
def deep_scan(ip):
|
||
hostname = socket.gethostbyaddr(ip)[0] # may be None
|
||
ports = scan_host_ports(ip, IMPORTANT_PORTS, workers=30)
|
||
tls = analyze_tls(ip) if any(p[0]==443 for p in ports) else None
|
||
ssh = get_ssh_banner(ip, 22) or get_ssh_banner(ip, 2222)
|
||
web = get_web_header(ip, 80)
|
||
return {"ip": ip, "hostname": hostname, "ports": ports, "tls": tls, "ssh": ssh, "web_server": web}
|
||
```
|
||
|
||
### 1.3 TLS Analysis
|
||
```python
|
||
def analyze_tls(ip, port=443, timeout=5.0):
|
||
ctx = ssl.create_default_context()
|
||
ctx.check_hostname = False
|
||
ctx.verify_mode = ssl.CERT_NONE
|
||
try:
|
||
with socket.create_connection((ip, port), timeout=timeout) as sock:
|
||
with ctx.wrap_socket(sock, server_hostname=ip) as s:
|
||
return {
|
||
"protocol": s.version(), "cipher": s.cipher()[0],
|
||
"cert_cn": next((f[0][1] for f in s.getpeercert().get("subject",[]) if f[0][0]=="commonName"), None),
|
||
"cert_issuer": next((f[0][1] for f in s.getpeercert().get("issuer",[]) if f[0][0]=="commonName"), None),
|
||
"valid_from": s.getpeercert().get("notBefore"),
|
||
"valid_until": s.getpeercert().get("notAfter"),
|
||
}
|
||
except Exception as e:
|
||
return {"error": str(e)[:80]}
|
||
```
|
||
|
||
### 1.4 SSH Banner Grab
|
||
```python
|
||
def get_ssh_banner(ip, port=22, timeout=5.0):
|
||
sock = socket.create_connection((ip, port), timeout=timeout)
|
||
banner = sock.recv(256).decode("utf-8", errors="ignore").strip()
|
||
sock.close(); return banner
|
||
```
|
||
|
||
### 1.5 Output Format
|
||
```json
|
||
{
|
||
"scan_date": "2026-05-17T08:03:11.184435+00:00",
|
||
"total_hosts": 60,
|
||
"scan_duration_seconds": 94.4,
|
||
"segments": {
|
||
"Hypervisor": {"subnet": "10.0.20.0/24", "alive_count": 10, "hosts": [...]},
|
||
"Container": {"subnet": "10.0.30.0/24", "alive_count": 26, "hosts": [...]},
|
||
"Smart Home": {"subnet": "10.0.50.0/24", "alive_count": 24, "hosts": [...]}
|
||
}
|
||
}
|
||
```
|
||
|
||
- `hostname` is usually `null` (no reverse-DNS in internal networks)
|
||
- `vendor` and `mac` are NOT included (no MAC access from container)
|
||
|
||
---
|
||
|
||
## Section 2: External Domain & Service Reconnaissance
|
||
|
||
### 2.1 DNS Resolution
|
||
```bash
|
||
getent hosts cloud.familie-schoen.com
|
||
host cloud.familie-schoen.com
|
||
```
|
||
|
||
### 2.2 Port Scanning (Bash /dev/tcp)
|
||
```bash
|
||
bash -c 'echo >/dev/tcp/HOST/PORT 2>&1 && echo open || echo closed'
|
||
# Common ports: 22, 25, 53, 80, 443, 587, 993, 995, 3306, 5432, 8080, 8443, 9090
|
||
```
|
||
|
||
### 2.3 HTTP/HTTPS Analysis
|
||
```bash
|
||
curl -sI http://DOMAIN
|
||
curl -ksI https://DOMAIN
|
||
curl -kI -L https/DOMAIN
|
||
curl -ks https://DOMAIN/path
|
||
# Security headers:
|
||
curl -ksI https://DOMAIN | grep -iE 'strict-transport|content-security|x-frame|x-xss|cache-control'
|
||
```
|
||
|
||
### 2.4 SSL Certificate Extraction
|
||
```bash
|
||
openssl s_client -connect HOST:443 -servername HOST 2>&1 </dev/null
|
||
openssl s_client -connect HOST:443 -servername HOST 2>&1 </dev/null | \
|
||
openssl x509 -noout -subject -issuer -dates -ext subjectAltName
|
||
```
|
||
|
||
### 2.5 API/Service Discovery
|
||
```bash
|
||
for endpoint in /api/v1/version /api/health /healthz /version /status /robots.txt; do
|
||
r=$(curl -ks https://HOST$endpoint)
|
||
[ -n "$r" ] && echo "$endpoint: $r"
|
||
done
|
||
```
|
||
|
||
Common services to fingerprint:
|
||
- Seafile: `/api2/server-info/`, `/seafhttp/`
|
||
- GitLab: `/api/v4/version`
|
||
- Nextcloud: `/.well-known/caldav`
|
||
- Grafana: `/api/health`
|
||
- Prometheus: `/metrics`
|
||
|
||
### 2.6 HTML Content Analysis
|
||
```bash
|
||
curl -ks https://DOMAIN | grep -ioE '<title>[^<]+'
|
||
curl -ks https://DOMAIN | grep -ioE 'wordpress|django|flask|react|angular|nextcloud|seafile|gitea|jenkins|grafana|prometheus'
|
||
```
|
||
|
||
---
|
||
|
||
## Section 3: Automated Recurring Scanning
|
||
|
||
For SRE-style weekly network scans with change detection, set up a `no_agent` cronjob:
|
||
|
||
```bash
|
||
# 1. Write the scan script to ~/.hermes/scripts/network-scan.py
|
||
# 2. Create the cronjob:
|
||
cronjob action=create \
|
||
name="weekly-network-scan" \
|
||
script="network-scan.py" \
|
||
schedule="0 8 * * 0" \
|
||
no_agent=true \
|
||
deliver="origin"
|
||
```
|
||
|
||
**Critical rules:**
|
||
- `script` parameter accepts **filename only** (relative to `~/.hermes/scripts/`), NOT an absolute path.
|
||
- Prefer `--no-agent` for pure script execution — faster, avoids model format issues.
|
||
- After any Hermes update, verify cron job model format (see `references/cron-model-fix.md`).
|
||
|
||
### Change Detection
|
||
Load the previous scan file and diff:
|
||
- New IPs → new hosts
|
||
- Missing IPs → gone hosts
|
||
- New ports on existing hosts → new services
|
||
- Different SSH banners → version changes
|
||
|
||
---
|
||
|
||
## Pitfalls
|
||
|
||
| Pitfall | Symptom | Solution |
|
||
|---------|---------|----------|
|
||
| Using nmap | Permission denied | Use Python stdlib only |
|
||
| Using arping | Hangs forever | NEVER use arping in containers |
|
||
| Using `ip neigh` for MAC | No MACs returned | OUI vendor detection is useless from container |
|
||
| Sequential TCP scan | 14+ min for /24 | Use ThreadPoolExecutor (100 workers sweep, 30 deep scan) |
|
||
| Wrong IP format | Can't reach targets | Verify with `ip route` and `ping` first |
|
||
| `script` path absolute | Cronjob fails | Use filename only in `script` param |
|
||
| Cron model null after update | `Error code: 400` | Update with `provider/model` format |
|
||
| ShareGPT first-run delay | 2–5 min wait | Pre-download once before benchmark suite |
|
||
| NumPy X86_V2 crash | `RuntimeError: NumPy...X86_V2` | `pip install "numpy<2.0" --force-reinstall` |
|
||
| Prefill throughput bias | First request inflates latency | Pre-warm with throwaway run, or filter `turn_index=0` from JSONL |
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- `references/container-network-recon-session-2026-04.md` — Session notes from initial network discovery
|
||
- `references/domain-service-recon-familie-schoen.md` — External domain scan example
|
||
- `references/cron-model-fix.md` — Post-Hermes-update cron job model fix
|
||
- `scripts/network-scan.py` — Full internal scan script (copy to `~/.hermes/scripts/`)
|