pic/Personal Internet Cell – Project Wiki.md

Personal Internet Cell – Project Wiki

Overview

Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, DHCP, NTP, WireGuard VPN, email, calendar/contacts, file storage, HTTPS reverse proxy, a certificate authority, and optional services — all managed from a single REST API and React web UI.

The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers.

---

Table of Contents

1. Architecture
2. Service Managers
3. First-Run Wizard
4. Authentication
5. API Reference
6. DDNS
7. Services UI
8. Service Store (Add-ons)
9. Cell-to-Cell Networking
10. Extended Connectivity
11. Security Model
12. Testing
13. Development

---

Architecture

Browser / WireGuard peer
  └── Caddy (:80/:443)             reverse proxy, TLS termination
        └── React SPA (:8081)      Vite + Tailwind (Nginx in container)
        └── Flask API (:3000)      REST API, bound to 127.0.0.1
              ├── NetworkManager       CoreDNS, dnsmasq, chrony
              ├── WireGuardManager     WireGuard VPN peer lifecycle
              ├── PeerRegistry         peer registration and trust
              ├── EmailManager         Postfix + Dovecot
              ├── CalendarManager      Radicale CalDAV/CardDAV
              ├── FileManager          WebDAV + Filegator
              ├── RoutingManager       iptables NAT and routing
              ├── FirewallManager      iptables firewall rules
              ├── VaultManager         internal CA, cert lifecycle, Age encryption
              ├── ContainerManager     Docker SDK
              ├── CellLinkManager      cell-to-cell WireGuard links
              ├── ConnectivityManager  exit routing (WG ext, OpenVPN, Tor)
              ├── DDNSManager          dynamic DNS heartbeat
              ├── ServiceStoreManager  optional service install/remove
              ├── CaddyManager         Caddyfile generation and reload
              ├── AuthManager          session auth, RBAC
              └── SetupManager         first-run wizard state

The 7 core containers run on a Docker bridge network (cell-network, 172.20.0.0/16 default). Static IPs per container are defined in docker-compose.yml. Installed optional services join the same network via their own compose projects, managed by ServiceComposer.

Runtime configuration lives in config/api/cell_config.json, managed by ConfigManager. All service managers read and write through ConfigManager, which validates and backs up automatically.

---

Service Managers

All managers inherit BaseServiceManager (api/base_service_manager.py), which provides:

• get_status() — current running state
• get_config() / update_config() — config read/write
• test_connectivity() — reachability check
• get_logs() — last N lines from the service log
• restart_service() — container restart via Docker SDK

The ServiceBus (api/service_bus.py) handles pub/sub events between managers (e.g., CONFIG_CHANGED, SERVICE_STARTED). Dependencies are declared in the bus (wireguard depends on network; email depends on network and vault).

Manager summary

Manager	Responsibilities
NetworkManager	CoreDNS zone files, dnsmasq DHCP config and lease monitoring, chrony NTP
WireGuardManager	Key generation, wg0.conf generation, peer add/remove, route sync
PeerRegistry	Peer registration, trust tracking, peer statistics
EmailManager	docker-mailserver accounts, mailbox config, alias management
CalendarManager	Radicale user/calendar/contacts lifecycle
FileManager	WebDAV user directories, Filegator access
RoutingManager	NAT rules, per-peer routing policy, fwmark-based exit routing
FirewallManager	iptables INPUT/FORWARD/OUTPUT rule management
VaultManager	Internal CA (self-signed root), TLS cert issue/revoke, Age public key
ContainerManager	Docker container/image/volume management via SDK
CellLinkManager	Site-to-site WireGuard links to other PIC cells, peer-sync protocol
ConnectivityManager	Per-peer exit routing via WireGuard external, OpenVPN, or Tor
DDNSManager	Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns)
ServiceStoreManager	Fetch manifest index, install/remove optional services
CaddyManager	Caddyfile generation, reload-on-change
AuthManager	bcrypt password store, session management, admin/peer RBAC
SetupManager	First-run wizard state, setup-complete flag

---

First-Run Wizard

On first start, SetupManager.is_setup_complete() returns False. The enforce_setup before-request hook returns HTTP 428 for all /api/* requests except /api/setup/* and /health, redirecting clients to /setup.

The wizard collects:

• Cell name — used for hostnames and DDNS subdomain (e.g. myhome → myhome.pic.ngo)
• Domain mode — determines TLS certificate source: lan (internal CA), pic_ngo, cloudflare, duckdns, http01
• Timezone
• Services to install — optional services (email, calendar, files) to install after setup; each starts a background install via ServiceStoreManager
• Admin password — minimum 12 characters

On completion:

1. Admin account is created in data/auth_users.json
2. Cell identity is written to config/api/cell_config.json
3. Caddy config is generated
4. If domain mode is pic_ngo, the cell registers <name>.pic.ngo with the DDNS service
5. Each selected service is installed in a background thread

Wizard endpoints: GET/POST /api/setup/step, GET /api/setup/status, POST /api/setup/complete.

---

Authentication

AuthManager stores bcrypt-hashed credentials in data/auth_users.json. Two roles:

Role	Access
admin	All /api/* endpoints except /api/peer/*
peer	/api/peer/* only (peer dashboard, key exchange)

Session auth flow:

• POST /api/auth/login — creates a Flask session
• GET /api/auth/me — current session info
• POST /api/auth/logout — clears session
• POST /api/auth/change-password — change own password
• POST /api/auth/admin/reset-password — admin resets another user's password

CSRF protection: all POST, PUT, DELETE, PATCH on /api/* (except /api/auth/* and /api/setup/*) require the X-CSRF-Token header matching the session token, obtained via GET /api/auth/csrf-token.

Cell-to-cell peer-sync endpoints (/api/cells/peer-sync/*) use source-IP + WireGuard public key auth, not session cookies.

Auth enforcement is active once any user exists in the store. If the store is empty (fresh install before wizard), all requests bypass auth — enforce_setup already blocks them with 428.

---

API Reference

Base URL: http://localhost:3000
Auth: session cookie (X-CSRF-Token header required for mutations)

Core

Method	Path	Description
GET	/health	Health check (always public)
GET	/api/status	All-service status summary
GET	/api/config	Full cell config
PUT	/api/config	Update cell config
GET	/api/health/history	Recent health check history

Auth (/api/auth/)

Method	Path	Description
POST	/api/auth/login	Create session
POST	/api/auth/logout	Destroy session
GET	/api/auth/me	Current user info
GET	/api/auth/csrf-token	Get CSRF token
POST	/api/auth/change-password	Change own password
POST	/api/auth/admin/reset-password	Admin: reset another user's password
GET	/api/auth/users	Admin: list users

Setup (/api/setup/)

Method	Path	Description
GET	/api/setup/status	Setup complete flag + current step
GET	/api/setup/step	Current wizard step data
POST	/api/setup/step	Submit current step
POST	/api/setup/complete	Finalize setup

Network Services (/api/dns/, /api/dhcp/, /api/ntp/, /api/network/)

DNS records, DHCP leases and reservations, NTP status, network connectivity test.

WireGuard (/api/wireguard/, /api/peers/)

Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.

Email (/api/email/) (available when email service is installed)

User account management, mailbox config, alias management, connectivity test. Returns HTTP 404 when the email service is not installed (except /api/email/status).

Calendar (/api/calendar/) (available when calendar service is installed)

User, calendar, and contacts (CardDAV) management. Returns HTTP 404 when the calendar service is not installed (except /api/calendar/status).

Files (/api/files/) (available when files service is installed)

WebDAV user management, file upload/download/delete, folder management. Returns HTTP 404 when the files service is not installed (except /api/files/status).

Routing (/api/routing/)

NAT rules, peer routes, exit node configuration.

Vault (/api/vault/)

Certificate issue/revoke, CA certificate, trust key management, Age public key.

Containers (/api/containers/)

List, start, stop, inspect containers; manage images and volumes.

Cell Network (/api/cells/)

List connected cells, add/remove cell links, peer-sync.

Connectivity (/api/connectivity/)

List exit nodes, configure WireGuard external / OpenVPN / Tor exits, assign per-peer exit policy.

Service Store (/api/store/)

List available services, install, remove.

Logs (/api/logs/)

Per-service log retrieval, log search, log statistics.

---

DDNS

DDNSManager maintains a <cell-name>.pic.ngo DNS A record pointing at the cell's public IP. A background thread runs every 5 minutes and calls provider.update(token, ip) only when the IP changes.

Registration happens during the setup wizard (if domain mode is pic_ngo) via provider.register(name, ip), which returns a bearer token stored in data/api/.ddns_token.

DDNS config lives in cell_config.json under the top-level ddns key:

{
  "ddns": {
    "provider": "pic_ngo",
    "api_base_url": "https://ddns.pic.ngo",
    "totp_secret": "<base32 secret>"
  }
}

Registration requires a time-based OTP (X-Register-OTP header) derived from the shared REGISTER_TOTP_SECRET on the DDNS server. This prevents unauthorized subdomain registration.

Supported providers: pic_ngo, cloudflare, duckdns, noip, freedns.

---

Services UI

Navigation

The left-hand navigation contains a Services group. Both admin and peer users see it. Sub-items for installed services (Email, Calendar, Files, etc.) are added dynamically: the UI fetches GET /api/services/active on load and after each install/uninstall. Services not yet installed do not appear in the nav.

Legacy paths redirect to their new canonical locations:

Old path	New path
/email	/services/email
/calendar	/services/calendar
/files	/services/files
/store	/services

Services page (/services)

A single unified catalog of all available services from the store index. Each card shows:

• Service name, description, version
• Install button (not installed) or Uninstall button (installed)
• Open link for installed services (navigates to the service sub-page)
• Running/stopped status dot for installed services

The pic-services-changed custom DOM event is dispatched after install/uninstall, causing the nav to re-fetch active services immediately.

Service sub-pages — admin view

Each sub-page at /services/email, /services/calendar, and /services/files shows:

1. Connection info — hostnames, ports, and protocol details (e.g. IMAP/SMTP/Webmail, CalDAV/CardDAV, WebDAV/Filegator).
2. Service status — current running state fetched from the API.
3. Users list — accounts registered with that service.
4. Inline config form — editable fields for that service's settings.

If the service is not installed, the page shows a ServiceNotInstalledBanner with a link to the catalog for admins, or a "contact your admin" message for peer users. All non-status API routes for uninstalled services return HTTP 404.

Config forms save automatically with an 800 ms debounce after the last change.

Service sub-pages — peer view

Peers access the same URLs. The peer view shows only:

• Connection info (hostnames, ports, copy buttons).
• Personal credentials for that service, fetched from /api/peer/*.

The config form and users list are not shown to peers.

Settings page

The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS/DHCP/NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.

Relevant API endpoints

Method	Path	Description
GET	/api/services/active	List installed services with id, name, subdomain, capabilities
GET	/api/config	Full cell config, includes installed_services dict

---

Service Store (Add-ons)

Email, calendar, and file storage are store services — not part of the core stack. All optional functionality ships through this mechanism.

ServiceStoreManager fetches a manifest index from https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json. Each manifest (schema_version: 3) declares:

• Container image and compose template
• Caddy subdomain routes
• Capabilities: has_subdomain, has_accounts, has_admin_config, has_storage, has_egress
• Account provisioning interface (accounts.manager)
• Backup declarations (backup.volumes, backup.config_paths)
• Egress routing policy (egress.allowed)
• Per-peer connection info template (peer_config_template)

POST /api/store/install fetches the manifest and compose template, validates them, renders the template with PIC-specific variables (${PIC_DOMAIN}, ${PIC_DATA_DIR}, etc.), writes a per-service compose file, and brings the containers up via ServiceComposer. Caddy routes and DNS entries are applied automatically.

POST /api/store/remove checks for dependent services, stops and removes containers, and regenerates Caddy.

ServiceComposer (api/service_composer.py) manages the per-service compose lifecycle independently of the main stack. Each service gets its own compose project at data/services/<id>/docker-compose.yml. On startup, reapply_active_services() brings up containers for all recorded installs.

See docs/service-developer-guide.md for the full manifest schema reference and submission process.

---

Cell-to-Cell Networking

CellLinkManager manages WireGuard site-to-site tunnels between PIC cells. Each link is a WireGuard peer configured with a dedicated /32 address and allowed-IPs covering the remote cell's subnet.

The peer-sync protocol (/api/cells/peer-sync/) exchanges public keys and allowed networks between cells using source-IP + WireGuard public key authentication (no session required).

Access control is per-service (calendar, files, mail, WebDAV) and enforced at the iptables level.

---

Extended Connectivity

ConnectivityManager provides per-peer exit routing: traffic from a specific WireGuard peer can be routed through an alternate exit instead of the cell's default gateway.

Supported exits:

• WireGuard external — another WireGuard endpoint (e.g. a VPS)
• OpenVPN — OpenVPN client running in a container
• Tor — Tor SOCKS proxy with transparent redirection

Routing uses fwmark and ip rule / ip route in separate routing tables. Configuration is via PUT /api/connectivity/peers/<peer_name>/exit.

---

Security Model

• No open ports for the API — Flask API binds to 127.0.0.1:3000 only; Caddy proxies HTTPS requests to it.
• Session auth — bcrypt passwords, Flask server-side sessions, CSRF double-submit.
• Setup wizard gate — all /api/* requests return 428 until setup is complete.
• Role separation — admin cannot access peer endpoints; peer cannot access admin endpoints.
• HTTPS everywhere — Caddy handles TLS; internal services are reached via reverse proxy paths.
• Internal CA — VaultManager issues certificates for services that don't use Let's Encrypt.
• Docker socket isolation — the Docker socket is mounted only into cell-api; other containers have no Docker access.
• iptables firewall — FirewallManager manages INPUT/FORWARD rules; WireGuard peer isolation is enforced at the packet level.

---

Testing

make test            # unit tests (pytest, ~1900+ functions)
make test-coverage   # coverage report in htmlcov/

Test layout:

• tests/ — unit and endpoint tests; no running services required
• tests/integration/ — require a running PIC stack
• tests/e2e/ — Playwright UI tests and WireGuard integration tests

CI: Gitea Actions runs pytest tests/ --ignore=tests/e2e --ignore=tests/integration on every push.

---

Development

# Full stack in Docker
make start
make stop
make logs

# Flask API without Docker (port 3000)
pip install -r api/requirements.txt
python api/app.py

# React UI dev server (port 5173, proxies /api → :3000)
cd webui && npm install && npm run dev

# Rebuild containers after code change
make build-api
make build-webui

Key files:

• api/app.py — Flask app, blueprint registration, before-request hooks, health monitor thread
• api/managers.py — singleton instantiation of all service managers
• api/base_service_manager.py — abstract base class all managers implement
• api/config_manager.py — cell_config.json read/write/validate/backup
• api/service_bus.py — pub/sub event system
• webui/src/services/api.js — Axios API client used by all UI pages
• docker-compose.yml — container definitions and network topology
• Makefile — all operational commands