PlanetOnyx Edge Platform

• Queue-based incident processing with one-ticket-at-a-time worker execution.
• Control API for standardized plan submission and host-aware ticket routing.
• Cloud-model orchestration with fallback strategy and structured outputs.
• Restore-only execution contract to keep runtime stable and reproducible.

• Included: runtime operations code, worker orchestration, API, queue lifecycle.
• Excluded: knowledgebase modules, security/compliance automation, firewall redesign.
• All feature updates are published through the URC Git history and changelog.

github.com/fine77/ultimate-repair-center

• Reads active ZFS pools/datasets and exposes pool health, allocation and filesystem controls.
• Provides NFS export visibility and management in one operational view.
• Provides SMB share visibility and management aligned with storage datasets.
• Runs as Cockpit module and keeps command execution on the managed host side.

• Built for production-like storage operations with clear, repeatable actions.
• Theme-compatible with Cockpit for consistent operator UX.
• Designed for iterative hardening; current release line is alpha.

github.com/fine77/cockpit-zfs-manager

• Change guardrails with backup-before-apply and explicit rollback path.
• Least-privilege ACL model with dedicated tags and policy checks.
• Central log pipeline plus targeted alerts (no alert noise flood).

• Every critical change is validated by runtime checks (route, DNS, HTTP, logs).
• Security events are correlated across firewalls, VPS, and app ingress.
• Backup and restore outcomes are documented as operational evidence.

• K3s baseline active on ct210..ct214; storage backend runs on kilox-stor001 with ZFS.
• Ingress remains fixed on HAProxy VIP 10.10.10.6:443 (no direct node-IP cutover).
• Tailnet + LAN routes validated after failover events and recovery runs.

• Syslog pipeline remains active: rsyslog -> promtail -> loki -> grafana.
• OPNsense and CrowdSec sources are mapped into Grafana dashboards.
• Monitoring focus is on runtime operations and incident debug retention.

• All five workers are online and idle-ready in restore-only mode.
• Canary plan flow validated: sre_diagnoser -> executor handoff works.
• Executor dry-run completed via ops-repair without drift changes.

• Large incremental backup set completed on PBS on 2026-03-06.
• Included: host/vps-planetonyx, ct/210..216, and host/kilox-stor001-nfs-apps.
• Current restore posture: full VPS + Proxmox CT + NFS app-data recovery path verified.

• Where TLS terminates (VPS ingress vs internal VIP).
• Where policy is enforced (ACLs, firewalls, allowlists, auth gateways).
• Which domains are public vs private (split DNS boundaries).
• Which paths are control-plane vs data-plane.

• Publish exact details only on an internal/private page.
• For the public page, keep ranges generic and use aggregated metrics.
• Never publish tokens, key material, or admin endpoints.

• Document power redundancy (A/B feeds) and critical dependencies (firewalls, switches, storage).
• Keep control-plane reachable (OOB) even when the main network is degraded.
• Reduce mean-time-to-repair by mapping “what is where” to ports and roles.

• Exact port maps (switch ports, patch panel numbers, optics types).
• Device serials, iDRAC/IPMI addresses, inventory and warranty dates.
• Thermal and power budget calculations per rack.

api:
  dashboard: true

entryPoints:
  http:
    address: \":80\"
    http:
      redirections:
        entrypoint:
          to: https
          scheme: https
  https:
    address: \":443\"

providers:
  docker:
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: false
  file:
    filename: /config.yaml

certificatesResolvers:
  cloudflare:
    acme:
      email: you@example.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53

http:
  middlewares:
    authelia:
      forwardauth:
        address: http://authelia:9091/api/authz/forward-auth
        trustForwardHeader: true
    crowdsec-bouncer:
      forwardauth:
        address: http://bouncer-traefik:8080/api/v1/forwardAuth
        trustForwardHeader: true

  routers:
    headscale-edge:
      entryPoints: [https]
      rule: HostRegexp(`{subdomain:[a-z0-9-]+}.headscale.planetonyx.net`)
      middlewares: [authelia, crowdsec-bouncer]
      tls: { certResolver: cloudflare }
      service: home-edge

  services:
    home-edge:
      loadBalancer:
        passHostHeader: true
        servers:
          - url: http://<FIREWALL_A_TS_IP>:80
          - url: http://<FIREWALL_B_TS_IP>:80

services:
  about-site:
    image: your/about-site:latest
    networks: [proxy]
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.www.rule=Host(`www.planetonyx.net`)
      - traefik.http.routers.www.entrypoints=https
      - traefik.http.routers.www.tls=true
      - traefik.http.routers.www.tls.certresolver=cloudflare
      - traefik.http.services.www.loadbalancer.server.port=80

networks:
  proxy:
    external: true

// policy.hujson (template)
{
  \"groups\": { \"group:admins\": [\"marcus@example.com\"] },
  \"tagOwners\": {
    \"tag:vps\": [\"group:admins\"],
    \"tag:secure-app\": [\"group:admins\"]
  },
  \"acls\": [
    { \"action\": \"accept\", \"src\": [\"group:admins\"], \"dst\": [\"tag:secure-app:*\" ] },
    { \"action\": \"accept\", \"src\": [\"tag:vps\"], \"dst\": [\"tag:secure-app:2375\" ] }
  ]
}

# Frontend: terminate TLS on VIP
frontend fe_https
  bind :443 ssl crt /path/to/wildcard.pem
  mode tcp
  default_backend be_nas_https

backend be_nas_https
  mode tcp
  server nas_ingress <NAS_INGRESS_IP>:443 ssl verify none

; db.lan.planetonyx.net (template)
$ORIGIN lan.planetonyx.net.
@   IN SOA  ns1.lan.planetonyx.net. hostmaster.lan.planetonyx.net. (
        2026021801 3600 600 1209600 300 )
    IN NS   ns1.lan.planetonyx.net.

; Internal ingress targets
prometheus IN A  <NAS_INGRESS_IP>
paperless  IN A  <NAS_INGRESS_IP>

# /etc/systemd/system/atlas-data.service
[Service]
Type=oneshot
ExecStart=/root/about-site/tools/generate_public_atlas_data.sh

# /etc/systemd/system/atlas-data.timer
[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

# Example checklist
1) firewall config export (config.xml)
2) docker compose/stack definitions
3) critical volumes (app data)
4) control-plane state (headscale DB)
5) verify restore path (smoke restore)

services:
  app:
    image: nginxdemos/hello
    labels:
      - traefik.enable=true
      - traefik.http.routers.app.rule=Host(`app.example.com`)
      - traefik.http.routers.app.entrypoints=https
      - traefik.http.services.app.loadbalancer.server.port=80

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "tcp://log-gateway.example.com:1514",
    "syslog-format": "rfc5424micro",
    "tag": "{{.Name}}"
  }
}

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      listen_protocol: tcp
      labels:
        job: "opnsense-syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: host

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["node-exporter.example.com:9100"]
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor.example.com:8080"]

{
  "acls": [
    {
      "action": "accept",
      "src": ["tag:ops"],
      "dst": ["tag:secure-app:443,9090,1514"]
    }
  ]
}

services:
  web:
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:18
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]

// adapted from prometheus/config package shape
cfg, err := config.LoadFile("/etc/prometheus/prometheus.yml", false, logger)
if err != nil {
  return err
}
for _, sc := range cfg.ScrapeConfigs {
  _ = sc.JobName
}

// adapted from Loki Promtail syslog target config model
type SyslogTargetConfig struct {
  ListenAddress string            `yaml:"listen_address"`
  Labels        map[string]string `yaml:"labels"`
}
// relabeling maps __syslog_message_hostname to stable host label

• Responsibility: public ingress, certificate automation, optional auth/abuse controls.
• Risk posture: hardened, minimal exposed ports, strict routing rules.
• Operations: logs and access metrics provide proof of behavior.

• Internal apps are reached via tailnet DNS names and internal ingress.
• Only aggregated metrics and sanitized templates are published here.

entryPoints:
  http:  { address: \":80\" }
  https: { address: \":443\" }

providers:
  docker:
    exposedByDefault: false

certificatesResolvers:
  cloudflare:
    acme:
      dnsChallenge:
        provider: cloudflare

traefik.enable=true
traefik.docker.network=proxy

traefik.http.routers.www.rule=Host(`www.planetonyx.net`)
traefik.http.routers.www.entrypoints=https
traefik.http.routers.www.tls.certresolver=cloudflare

• Public + private app entry stays on HAProxy VIP 10.10.10.6:443.
• K3s service IPs are backend targets, not user-facing entrypoints.
• Storage traffic uses NFS exports from kilox-stor001 (ZFS-backed).

# K3s ingress objects
kubectl -n apps get ingress

# Core services
kubectl -n apps get svc

# Verify endpoint resolution
kubectl -n apps get endpoints

# Ingress + backend
kubectl -n apps describe ingress <name>
kubectl -n apps get svc,pods -o wide
kubectl -n apps logs deploy/traefik --tail=200

# Edge path
curl -kI https://<app>.srv.planetonyx.net
curl -kI https://<app>.headscale.planetonyx.net

• Ingress is only via routed domains; no direct node-port publishing for app UIs.
• Admin paths stay behind allowlists + SSO/2FA gates.
• Tailnet/LAN/public DNS views are intentionally separated by split-DNS policy.

# Tailnet-only naming
*.headscale.planetonyx.net

# Client gets:
DNS server: 100.100.100.100
Search domain: headscale.planetonyx.net

// HUJSON-like pattern
{ \"action\": \"accept\",
  \"src\": [\"group:admins\"],
  \"dst\": [\"tag:secure-app:*\"]
}

• Node inventory: who/what is connected, last seen, tags.
• Routes and exit nodes: approve explicitly, then monitor reachability.
• ACL policy: validate changes before applying (policy check).

# Concept (Headscale config)
policy:
  mode: database   # UI-editable
  # mode: file     # UI read-only; edit policy.hujson on disk

• Prefer tags over IP rules (intent-based policy).
• Keep “route approval” as a deliberate action, not an automatic default.
• Regularly review “no rules matched” drops to catch missing ACLs early.

# Examples (sanitized)
headscale nodes list
headscale routes list
headscale policy check -f /etc/headscale/policy.hujson

# Frontend (TLS termination)
bind :443 ssl crt /path/to/wildcard.pem
mode tcp

# Backend (re-encrypt to internal ingress)
server nas_ingress 192.168.x.x:443 ssl verify none

# VIP concept
VIP: internal clients connect to one address
HA: active firewall serves; passive takes over on fail

# Typical pattern
www.planetonyx.net   CNAME  vps.planetonyx.net
vps.planetonyx.net   A      <VPS_PUBLIC_IP>

# Tailnet (MagicDNS / split DNS)
app.headscale.planetonyx.net  A  <TAILNET_INGRESS_IP>

# LAN
app.lan.planetonyx.net        A  <LAN_INGRESS_IP>

*.planetonyx.net
*.lan.planetonyx.net
*.headscale.planetonyx.net
*.srv.planetonyx.net

• Identity and access controls (Authelia/2FA, role boundaries).
• Network segmentation and policy routing (tailnet + firewall zones).
• Logging, monitoring, evidence retention and incident traceability.
• Backup/recovery controls and documented change management.

• `docs/BSI-200-2-GAP-ANALYSIS.md`
• `docs/BSI-HARDENING-MAPPING.md`
• `docs/CONTROL-EVIDENCE-MATRIX.md`
• `docs/CMDB-CHANGE-MANAGEMENT-MODEL.md`

# Traefik middleware concept
middlewares:
  authelia:
    forwardauth:
      address: http://authelia:9091/api/authz/forward-auth

# ForwardAuth concept
	middlewares:
	  crowdsec-bouncer:
	    forwardauth:
	      address: http://bouncer-traefik:8080/api/v1/forwardAuth

• MFA enforcement for sensitive apps (TOTP or WebAuthn).
• Session security: short inactivity timeout, long-term token hygiene.
• Central incident response: one place to revoke, lock, and review.

# authelia configuration.yml (concept)
access_control:
  default_policy: deny
  rules:
    - domain: "*.headscale.planetonyx.net"
      policy: two_factor
    - domain: "*.lan.planetonyx.net"
      policy: two_factor

session:
  expiration: 8h
  inactivity: 30m
  cookies:
    - domain: planetonyx.net
      secure: true
      same_site: lax

# authelia OIDC (concept)
identity_providers:
  oidc:
    clients:
      - client_id: grafana
        client_name: "Grafana"
        redirect_uris:
          - https://grafana.example.internal/login/generic_oauth
        scopes: [openid, profile, email]

• Prefer WebAuthn for admins (phishing-resistant).
• Enforce NTP time sync across nodes (TOTP stability).
• Never expose Authelia without strict allowlists; keep it tailnet/LAN-only.
• Rotate secrets regularly (session, JWT, storage key) and keep them out of docs.

• Credential stuffing and brute-force on exposed logins.
• Misconfigured DNS/routing exposing internal apps publicly.
• Reverse proxy bypass via direct IP/port exposure.
• Supply-chain risk via container images and dependencies.

# config.yaml (example)
tls:
  options:
    modern:
      minVersion: VersionTLS12
      sniStrict: true

# router uses:
tls:
  options: modern
  certResolver: cloudflare

# config.yaml (example)
http:
  middlewares:
    sec-headers:
      headers:
        stsSeconds: 31536000
        stsIncludeSubdomains: true
        stsPreload: true
        contentTypeNosniff: true
        browserXssFilter: true
        frameDeny: true
        referrerPolicy: no-referrer

# router:
middlewares: [sec-headers]

# config.yaml (example)
http:
  middlewares:
    ratelimit:
      rateLimit:
        average: 30
        burst: 60

# router:
middlewares: [ratelimit, crowdsec-bouncer]

# config.yaml (example)
http:
  middlewares:
    admin-allow:
      ipAllowList:
        sourceRange:
          - 100.64.0.0/10   # tailnet CGNAT range

# router:
middlewares: [admin-allow, authelia]

services:
  app:
    read_only: true
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    tmpfs:
      - /tmp

# docker compose pattern
env_file: [.env]

# or (Swarm-style):
secrets:
  cf_token:
    file: ./secrets/cf_token.txt

# Traefik access log (json) -> rotate
/var/log/traefik/access.log

# CrowdSec decisions -> visibility
cscli decisions list -o json

• Reverse proxy + auth configuration (Traefik, Authelia), without embedding secrets in docs.
• Firewall config exports (OPNsense config.xml) and HAProxy/ACME state.
• Docker stack definitions (compose/stack files) and critical app volumes.
• Tailnet control-plane state (Headscale DB/config) and policy files.

# Example pattern (concept)
# - snapshot configs
# - push to backup storage
# - verify presence / retention

# Is the router label correct?
docker inspect about-site-about-site-1 --format '{{json .Config.Labels}}' | jq .

# Does Traefik see the request?
docker exec traefik sh -lc 'tail -n 200 /var/log/traefik/access.log | grep -F "\"RequestHost\":\"www.planetonyx.net\""' 

# Quick confirm SNI path
curl -kI https://127.0.0.1/ -H 'Host: www.planetonyx.net'

# Client: route should go via tailscale0
ip route get 1.1.1.1

# Confirm DNS still works (it often does)
nslookup heise.de

tailscale status
resolvectl status
resolvectl query <host>

# Client view (conceptual)
ip route get 1.1.1.1
  => dev tailscale0

# Pseudocode
if ingress == tailscale0:
  route-to VPN_GW
else:
  normal routing

• Asymmetric routing: packets leave via VPN, replies return via WAN (state breaks).
• NAT applied on the wrong interface (or missing on the VPN uplink).
• Over-broad rules: tailnet traffic accidentally NATs into LAN or bypasses security gates.
• Exceptions list: some destinations must never go to VPN (RFC1918, management networks, backups).

# OPNsense/pf concept (pseudocode)
# 1) Block tailnet -> RFC1918 by default (unless explicitly allowed)
block in quick on tailscale0 from 100.64.0.0/10 to <RFC1918>

# 2) Route tailnet traffic to VPN for non-internal destinations
pass in quick on tailscale0 route-to (wg1 <VPN_GW>)
  from 100.64.0.0/10 to ! <NO_WG_POLICY> keep state

# 3) NAT on VPN uplink
nat on wg1 from 100.64.0.0/10 to any -> (wg1)

# Client side
ip route get 1.1.1.1
curl -4sv --connect-timeout 5 https://ifconfig.me/ -o /dev/null

# Firewall side (concept)
- check states (are they created on tailscale0?)
- packet capture on tailscale0 and wg1
- verify NAT rules on the actual egress interface

• Keep a strict "tailnet ingress policy": allow by intent, not by convenience.
• Keep exception lists small and reviewed (aliases are your friend).
• Document the "must never go to VPN" destinations (backups, management, internal DNS).

• Wildcards require DNS-01; HTTP-01 can't issue *.example.com.
• DNS propagation and API permissions are the common failure points.
• Provider rate limits apply: use staging for tests, production only when stable.

# traefik.yaml (concept)
certificatesResolvers:
  cloudflare:
    acme:
      email: you@example.com
      storage: acme.json
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53

# secret handling (concept)
env_file: [.env]  # contains CF_DNS_API_TOKEN

• Public: VPS Traefik terminates TLS for public names.
• Private: HAProxy can terminate wildcard TLS for internal admin UIs.
• Apps: run without cert automation complexity; they sit behind a trusted ingress.

# Confirm correct SNI path (concept)
curl -vkI https://www.planetonyx.net/

# If wrong cert:
# - router rule mismatch
# - resolver name mismatch
# - ACME failed (check logs and storage)

# Limiters (example values)
WAN_DOWN: 475 Mbit/s  (fq_codel)
WAN_UP:    95 Mbit/s  (fq_codel)

# Apply on egress for WAN and on egress for VPN uplink (wg1),
# depending on which path your traffic takes.

• Shaping only on download but not upload (upload is where latency often dies).
• Setting shaper above real throughput (no effect, buffers still fill).
• Applying limiters on the wrong interface (VPN vs WAN).

# Quick check idea
- run a sustained upload/download
- ping a stable host in parallel
- latency should remain bounded instead of climbing into seconds

• System health: CPU, memory, disk usage, IO latency (node exporter).
• Container health: restarts, memory pressure, throttling (cAdvisor).
• Edge health: status buckets, p95 latency, request rate (Traefik logs/metrics).
• Network health: DNS, ICMP loss/latency, route reachability (blackbox checks).
• Security signals: auth failures and bouncer decisions (Authelia/CrowdSec).

• Page only on user-impacting conditions (ingress down, auth outage, route broken).
• Ticket on capacity trends (disk growth, sustained error budget burn).
• Keep alerts few, specific, and actionable.

# Example: error budget burn (concept)
burn = rate(http_requests_total{code=~"5.."}[5m]) /
       rate(http_requests_total[5m])

alert if burn > 0.01 for 10m

# Examples (conceptual)
- Ingress down (no 2xx/3xx for 5m)
- DNS failure for internal domains
- Route advertised but target unreachable
- Disk > 85% for 30m
- Container crashloop (restarts > N in 10m)

# Examples (concept)
authelia_login_failures_total      # brute force / mistyped creds detection
crowdsec_decisions_active          # active blocks / pressure indicator
traefik_router_requests_total      # traffic and status buckets

# Uptime (probe_success)
avg_over_time(probe_success[24h])

# HTTP availability (%)
100 * (1 - (sum(rate(http_requests_total{code=~\"5..\"}[5m])) /
           sum(rate(http_requests_total[5m]))))

# Latency p95
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# ICMP packet loss p95 (concept)
histogram_quantile(0.95, rate(ping_loss_bucket[24h]))

• Security hardening baseline with verified controls.
• Hostname classification and ingress termination map.
• Monthly restore drill process and evidence template.
• Release status updated with implemented priorities.

• Telegram bot token rotation (deferred by owner).
• Cloudflare DNS API token rotation for Traefik ACME.
• Optional staged Influx token rotation in maintenance window.

• Reverse proxies and TLS automation (Traefik, HAProxy, DNS-01 wildcards).
• Network overlays, routing, and access control (Headscale/Tailscale, ACLs, split DNS).
• Operational tooling (monitoring, runbooks, reproducible Docker deployments).
• Security guardrails (CrowdSec, auth gateways, least privilege).

public:  DNS -> VPS Traefik -> container routing
private: tailnet + split DNS -> HAProxy VIP -> NAS ingress