From 4b3d695805aecd8d84691a53404e1f72dbe7ff10 Mon Sep 17 00:00:00 2001 From: Dmitrii Iurco Date: Thu, 11 Jun 2026 14:26:48 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20consolidate=20all=20manuals=20into=20th?= =?UTF-8?q?e=20Gitea=20wiki=20=E2=80=94=20repo=20keeps=20README=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit QUICKSTART, the monolithic project-wiki file, the API documentation, the service developer guide, and the webui README had drifted badly out of date (localhost-only auth, DHCP, v1 connectivity fwmarks, unsupported DDNS providers, "HTTP dispatch not implemented") while the four-persona Gitea wiki is current and maintained. Their still-accurate content now lives in the wiki (incl. the new Dev-Service-Manifest-Reference page), so the repo keeps a single README pointing there. README refreshed: Connectivity v2 named instances, signed store images, audit log, backup encryption, real provider list, current UI pages, dead LICENSE link removed. Co-Authored-By: Claude Fable 5 --- Personal Internet Cell – Project Wiki.md | 521 --------- QUICKSTART.md | 255 ----- README.md | 20 +- api/API_DOCUMENTATION.md | 1286 ---------------------- docs/service-developer-guide.md | 743 ------------- webui/README.md | 138 --- 6 files changed, 14 insertions(+), 2949 deletions(-) delete mode 100644 Personal Internet Cell – Project Wiki.md delete mode 100644 QUICKSTART.md delete mode 100644 api/API_DOCUMENTATION.md delete mode 100644 docs/service-developer-guide.md delete mode 100644 webui/README.md diff --git a/Personal Internet Cell – Project Wiki.md b/Personal Internet Cell – Project Wiki.md deleted file mode 100644 index 8988382..0000000 --- a/Personal Internet Cell – Project Wiki.md +++ /dev/null @@ -1,521 +0,0 @@ -# Personal Internet Cell – Project Wiki - -## Overview - -Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, NTP, WireGuard VPN, HTTPS reverse proxy, a certificate authority, and optional services (email, calendar/contacts, file storage, connectivity exits, and more) — all managed from a single REST API and React web UI. - -The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers. - ---- - -## Table of Contents - -1. [Architecture](#architecture) -2. [Service Managers](#service-managers) -3. [First-Run Wizard](#first-run-wizard) -4. [Authentication](#authentication) -5. [API Reference](#api-reference) -6. [DDNS](#ddns) -7. [Network Services and DNS](#network-services-and-dns) -8. [Services UI](#services-ui) -9. [Service Store (Add-ons)](#service-store-add-ons) -10. [Backup and Restore](#backup-and-restore) -11. [Cell-to-Cell Networking](#cell-to-cell-networking) -12. [Extended Connectivity](#extended-connectivity) -13. [Security Model](#security-model) -14. [Testing](#testing) -15. [Development](#development) - ---- - -## Architecture - -``` -Browser / WireGuard peer - └── Caddy (:80/:443) reverse proxy, TLS termination - └── React SPA (:8081→8080) Vite + Tailwind (Nginx in container) - └── Flask API (:3000) REST API, bound to 127.0.0.1 - ├── NetworkManager CoreDNS, chrony (split-horizon DNS) - ├── WireGuardManager WireGuard VPN peer lifecycle - ├── PeerRegistry peer registration and trust - ├── EmailManager Postfix + Dovecot (optional service) - ├── CalendarManager Radicale CalDAV/CardDAV (optional service) - ├── FileManager WebDAV + Filegator (optional service) - ├── RoutingManager iptables NAT and routing - ├── FirewallManager iptables firewall rules - ├── VaultManager internal CA, cert lifecycle, Fernet encryption - ├── ContainerManager Docker SDK - ├── CellLinkManager cell-to-cell WireGuard links - ├── ConnectivityManager exit routing (WG ext, OpenVPN, Tor, sshuttle, proxy) - ├── DDNSManager dynamic DNS heartbeat - ├── ServiceStoreManager optional service install/remove - ├── CaddyManager Caddyfile generation and reload - ├── AuthManager session auth, RBAC - └── SetupManager first-run wizard state -``` - -Six core containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default): `cell-caddy`, `cell-dns`, `cell-ntp`, `cell-wireguard`, `cell-api`, `cell-webui`. Static IPs per container are defined in `docker-compose.yml`. Installed optional services join the same network via their own compose projects, managed by `ServiceComposer`. - -The WireGuard container runs unprivileged — it uses `NET_ADMIN` only and requires the WireGuard module on the host kernel (Linux 5.6+ or a loadable module). The `cell-api` and `cell-webui` images are slim builds. The CoreDNS image is pinned to a specific digest. - -Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically. - ---- - -## Service Managers - -All managers inherit `BaseServiceManager` (`api/base_service_manager.py`), which provides: -- `get_status()` — current running state -- `get_config()` / `update_config()` — config read/write -- `test_connectivity()` — reachability check -- `get_logs()` — last N lines from the service log -- `restart_service()` — container restart via Docker SDK - -The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers (e.g., `CONFIG_CHANGED`, `SERVICE_STARTED`). Dependencies are declared in the bus (wireguard depends on network; email depends on network and vault). - -### Manager summary - -| Manager | Responsibilities | -|---|---| -| `NetworkManager` | CoreDNS zone files and split-horizon DNS, chrony NTP | -| `WireGuardManager` | Key generation, `wg0.conf` generation, peer add/remove, route sync | -| `PeerRegistry` | Peer registration, trust tracking, peer statistics | -| `EmailManager` | docker-mailserver accounts, mailbox config, alias management | -| `CalendarManager` | Radicale user/calendar/contacts lifecycle | -| `FileManager` | WebDAV user directories, Filegator access | -| `RoutingManager` | NAT rules, per-peer routing policy, fwmark-based exit routing | -| `FirewallManager` | iptables INPUT/FORWARD/OUTPUT rule management | -| `VaultManager` | Internal CA (self-signed root), TLS cert issue/revoke, Age public key | -| `ContainerManager` | Docker container/image/volume management via SDK | -| `CellLinkManager` | Site-to-site WireGuard links to other PIC cells, peer-sync protocol | -| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, Tor, sshuttle, or proxy (redsocks) | -| `DDNSManager` | Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns) | -| `ServiceStoreManager` | Fetch manifest index, install/remove optional services | -| `CaddyManager` | Caddyfile generation, reload-on-change | -| `AuthManager` | bcrypt password store, session management, admin/peer RBAC | -| `SetupManager` | First-run wizard state, setup-complete flag | - ---- - -## First-Run Wizard - -On first start, `SetupManager.is_setup_complete()` returns `False`. The `enforce_setup` before-request hook returns HTTP 428 for all `/api/*` requests except `/api/setup/*` and `/health`, redirecting clients to `/setup`. - -The wizard collects: -- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`) -- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01` -- **Timezone** -- **Services to install** — optional services (email, calendar, files) to install after setup; each starts a background install via `ServiceStoreManager` -- **Admin password** — minimum 12 characters - -Domain modes and their TLS behavior: - -| Mode | Certificate | -|---|---| -| `pic_ngo` | Wildcard Let's Encrypt cert via DNS-01 (requires accurate host clock for ACME + DDNS token) | -| `cloudflare` | Wildcard Let's Encrypt cert via Cloudflare DNS-01 | -| `duckdns` | Let's Encrypt via DuckDNS DNS-01 | -| `http01` | Let's Encrypt per-subdomain cert via HTTP-01 (no wildcard; port 80 must be reachable) | -| `lan` | Internal CA only; no internet required | - -On completion: -1. Admin account is created in `data/auth_users.json` -2. Cell identity is written to `config/api/cell_config.json` -3. Caddy config is generated -4. If domain mode is `pic_ngo`, the cell registers `.pic.ngo` with the DDNS service -5. Each selected service is installed in a background thread - -Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`. - ---- - -## Authentication - -`AuthManager` stores bcrypt-hashed credentials in `data/auth_users.json`. Two roles: - -| Role | Access | -|---|---| -| `admin` | All `/api/*` endpoints except `/api/peer/*` | -| `peer` | `/api/peer/*` only (peer dashboard, key exchange) | - -Session auth flow: -- `POST /api/auth/login` — creates a Flask session -- `GET /api/auth/me` — current session info -- `POST /api/auth/logout` — clears session -- `POST /api/auth/change-password` — change own password -- `POST /api/auth/admin/reset-password` — admin resets another user's password - -CSRF protection: all `POST`, `PUT`, `DELETE`, `PATCH` on `/api/*` (except `/api/auth/*` and `/api/setup/*`) require the `X-CSRF-Token` header matching the session token, obtained via `GET /api/auth/csrf-token`. - -Cell-to-cell peer-sync endpoints (`/api/cells/peer-sync/*`) use source-IP + WireGuard public key auth, not session cookies. - -Auth enforcement is active once any user exists in the store. If the store is empty (fresh install before wizard), all requests bypass auth — `enforce_setup` already blocks them with 428. - ---- - -## API Reference - -**Base URL:** `http://localhost:3000` -**Auth:** session cookie (`X-CSRF-Token` header required for mutations) - -### Core - -| Method | Path | Description | -|---|---|---| -| GET | `/health` | Health check (always public) | -| GET | `/api/status` | All-service status summary | -| GET | `/api/config` | Full cell config | -| PUT | `/api/config` | Update cell config | -| GET | `/api/health/history` | Recent health check history | - -### Auth (`/api/auth/`) - -| Method | Path | Description | -|---|---|---| -| POST | `/api/auth/login` | Create session | -| POST | `/api/auth/logout` | Destroy session | -| GET | `/api/auth/me` | Current user info | -| GET | `/api/auth/csrf-token` | Get CSRF token | -| POST | `/api/auth/change-password` | Change own password | -| POST | `/api/auth/admin/reset-password` | Admin: reset another user's password | -| GET | `/api/auth/users` | Admin: list users | - -### Setup (`/api/setup/`) - -| Method | Path | Description | -|---|---|---| -| GET | `/api/setup/status` | Setup complete flag + current step | -| GET | `/api/setup/step` | Current wizard step data | -| POST | `/api/setup/step` | Submit current step | -| POST | `/api/setup/complete` | Finalize setup | - -### Network Services (`/api/dns/`, `/api/ntp/`, `/api/network/`) - -DNS overview (effective domain, public records, internal records, per-mode actions), NTP status, network connectivity test. - -### WireGuard (`/api/wireguard/`, `/api/peers/`) - -Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status. - -### Email (`/api/email/`) _(available when email service is installed)_ - -User account management, mailbox config, alias management, connectivity test. Returns HTTP 404 when the email service is not installed (except `/api/email/status`). - -### Calendar (`/api/calendar/`) _(available when calendar service is installed)_ - -User, calendar, and contacts (CardDAV) management. Returns HTTP 404 when the calendar service is not installed (except `/api/calendar/status`). - -### Files (`/api/files/`) _(available when files service is installed)_ - -WebDAV user management, file upload/download/delete, folder management. Returns HTTP 404 when the files service is not installed (except `/api/files/status`). - -### Routing (`/api/routing/`) - -NAT rules, peer routes, exit node configuration. - -### Vault (`/api/vault/`) - -Certificate issue/revoke, CA certificate, trust key management, Age public key. - -### Containers (`/api/containers/`) - -List, start, stop, inspect containers; manage images and volumes. - -### Cell Network (`/api/cells/`) - -List connected cells, add/remove cell links, peer-sync. - -### Connectivity (`/api/connectivity/`) - -List exit nodes, configure WireGuard external / OpenVPN / Tor / sshuttle / proxy exits, assign per-peer exit policy, assign per-service egress. - -### Service Store (`/api/store/`) - -List available services, install, remove. - -### Logs (`/api/logs/`) - -Per-service log retrieval, log search, log statistics. - ---- - -## DDNS - -`DDNSManager` maintains a `.pic.ngo` DNS A record pointing at the cell's public IP. A background thread runs every 5 minutes and calls `provider.update(token, ip)` only when the IP changes. - -Registration happens during the setup wizard (if domain mode is `pic_ngo`) via `provider.register(name, ip)`, which returns a bearer token stored in `data/api/.ddns_token`. - -DDNS config lives in `cell_config.json` under the top-level `ddns` key: - -```json -{ - "ddns": { - "provider": "pic_ngo", - "api_base_url": "https://ddns.pic.ngo", - "totp_secret": "" - } -} -``` - -Registration requires a time-based OTP (`X-Register-OTP` header) derived from the shared `REGISTER_TOTP_SECRET` on the DDNS server. This prevents unauthorized subdomain registration. - -Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`. - ---- - -## Network Services and DNS - -### Split-horizon DNS - -PIC operates a split-horizon DNS configuration for the cell domain. - -- **Outside the VPN** — the cell domain (e.g. `myhome.pic.ngo`) resolves to the public IP via the DDNS provider. -- **Inside the VPN** — CoreDNS answers the same cell domain with the WireGuard internal IP of `cell-caddy`. Traffic therefore flows through the WireGuard tunnel and Caddy serves it on both the public and WireGuard interface. - -`NetworkManager.update_split_horizon_zone()` writes a zone file for the effective domain and reloads CoreDNS via SIGUSR1 whenever the cell name or domain mode changes. - -### Network Services page - -The **Network Services** page (`/network`) shows a provider-aware DNS overview: -- Current domain mode label and effective domain -- Public DNS records (A records registered with the DDNS provider) -- Service subdomains (shown for `pic_ngo` and `duckdns` modes) -- Internal records served by CoreDNS on the WireGuard network -- Per-mode action buttons (e.g. force-refresh DDNS, reload CoreDNS) -- NTP status (running/stopped, current time source) - -DHCP has been removed from PIC. There is no `cell-dhcp` container and no DHCP configuration in the UI. The `cell-dns` container runs CoreDNS only. - -### NTP - -The `cell-ntp` container runs chrony in a pinned Alpine image. It provides NTP to WireGuard peers that configure the cell as their NTP server. The installer also enables host-level chrony to keep the host clock accurate for ACME certificate issuance and DDNS TOTP tokens. - ---- - -## Backup and Restore - -### What `make backup` captures - -`make backup` creates `backups/cell-backup-.tar.gz` containing: -- `config/` — all service configuration including `cell_config.json` -- `data/` excluding logs (`data/logs/`) and internal config-backup snapshots (`data/api/config_backups/`) -- `docker-compose.yml` and `Makefile` - -The archive is written mode `0600`. It contains key material (WireGuard keys, internal CA, vault fernet.key, admin credentials, DDNS token, cell links, Caddy ACME certs). Store it securely. - -Data volumes of installed store services (email mailboxes, calendar collections, file trees) are **not** included in `make backup`. They are captured by the API-driven backup described below. - -### API-driven backup (`POST /api/config/backup`) - -The API backup captures everything `make backup` does, plus: -- `.env` -- The Caddyfile and Corefile (runtime-generated) -- DNS zone files -- WireGuard key material and live peer configs -- Vault directory (CA, certificates, fernet.key, trust store) -- Per-service connectivity configs (sshuttle keys, redsocks config, OpenVPN configs, WireGuard external config) -- Auth users, Flask secret key, DDNS token, cell links, peer service credentials -- Caddy issued ACME certificates and ACME state -- Live service data volumes (streamed via `docker exec tar`) - -**Passphrase encryption**: pass `{"passphrase": "..."}` in the request body to encrypt the archive. The encrypted file is named `.tar.gz.age` and uses Fernet with an scrypt-derived key. The plaintext staging directory is removed immediately after encryption. Supply the same passphrase when calling `POST /api/config/restore/`. - -Both backup and encrypted archive files are written mode `0600`. - -### Restore - -**From `make backup`:** - -```bash -tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz -make restart -``` - -**From API backup:** - -Call `POST /api/config/restore/` (with `{"passphrase": "..."}` for encrypted archives). The restore process: -1. Restores the vault first (fernet.key must be present before any encrypted secrets are read) -2. Restores identity, `.env`, WireGuard key material, cell links -3. Restores Caddy ACME certs, Caddyfile, Corefile, DNS zones -4. Restores connectivity configs, auth users, DDNS token -5. Restores service user account files -6. Reloads `cell_config.json` into memory -7. Restores live service data volumes (if service containers are running) -8. Calls `_reapply_runtime_state()` — regenerates Caddyfile and Corefile from the restored config and re-applies routing rules - -After an API restore, run `make restart` to ensure all containers pick up the restored configuration. - ---- - -## Services UI - -### Navigation - -The left-hand navigation contains a **Services** group. Both admin and peer users see it. Sub-items for installed services (Email, Calendar, Files, etc.) are added dynamically: the UI fetches `GET /api/services/active` on load and after each install/uninstall. Services not yet installed do not appear in the nav. - -Legacy paths redirect to their new canonical locations: - -| Old path | New path | -|---|---| -| `/email` | `/services/email` | -| `/calendar` | `/services/calendar` | -| `/files` | `/services/files` | -| `/store` | `/services` | - -### Services page (`/services`) - -A single unified catalog of all available services from the store index. Each card shows: -- Service name, description, version -- **Install** button (not installed) or **Uninstall** button (installed) -- **Open** link for installed services (navigates to the service sub-page) -- Running/stopped status dot for installed services - -The `pic-services-changed` custom DOM event is dispatched after install/uninstall, causing the nav to re-fetch active services immediately. - -### Service sub-pages — admin view - -Each sub-page at `/services/email`, `/services/calendar`, and `/services/files` shows: - -1. **Connection info** — hostnames, ports, and protocol details (e.g. IMAP/SMTP/Webmail, CalDAV/CardDAV, WebDAV/Filegator). -2. **Service status** — current running state fetched from the API. -3. **Users list** — accounts registered with that service. -4. **Inline config form** — editable fields for that service's settings. - -If the service is not installed, the page shows a `ServiceNotInstalledBanner` with a link to the catalog for admins, or a "contact your admin" message for peer users. All non-status API routes for uninstalled services return HTTP 404. - -Config forms save automatically with an 800 ms debounce after the last change. - -### Service sub-pages — peer view - -Peers access the same URLs. The peer view shows only: - -- Connection info (hostnames, ports, copy buttons). -- Personal credentials for that service, fetched from `/api/peer/*`. - -The config form and users list are not shown to peers. - -### Settings page - -The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS, NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore. - -### Relevant API endpoints - -| Method | Path | Description | -|---|---|---| -| GET | `/api/services/active` | List installed services with id, name, subdomain, capabilities | -| GET | `/api/config` | Full cell config, includes `installed_services` dict | - ---- - -## Service Store (Add-ons) - -Email, calendar, and file storage are store services — not part of the core stack. All optional functionality ships through this mechanism. - -`ServiceStoreManager` fetches a manifest index from `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest (`schema_version: 3`) declares: - -- Container image and compose template -- Caddy subdomain routes -- Capabilities: `has_subdomain`, `has_accounts`, `has_admin_config`, `has_storage`, `has_egress` -- Account provisioning interface (`accounts.manager`) -- Backup declarations (`backup.volumes`, `backup.config_paths`) -- Egress routing policy (`egress.allowed`) -- Per-peer connection info template (`peer_config_template`) - -`POST /api/store/install` fetches the manifest and compose template, validates them, renders the template with PIC-specific variables (`${PIC_DOMAIN}`, `${PIC_DATA_DIR}`, etc.), writes a per-service compose file, and brings the containers up via `ServiceComposer`. Caddy routes and DNS entries are applied automatically. - -`POST /api/store/remove` checks for dependent services, stops and removes containers, and regenerates Caddy. - -**`ServiceComposer`** (`api/service_composer.py`) manages the per-service compose lifecycle independently of the main stack. Each service gets its own compose project at `data/services//docker-compose.yml`. On startup, `reapply_active_services()` brings up containers for all recorded installs. - -See `docs/service-developer-guide.md` for the full manifest schema reference and submission process. - ---- - -## Cell-to-Cell Networking - -`CellLinkManager` manages WireGuard site-to-site tunnels between PIC cells. Each link is a WireGuard peer configured with a dedicated `/32` address and allowed-IPs covering the remote cell's subnet. - -The peer-sync protocol (`/api/cells/peer-sync/`) exchanges public keys and allowed networks between cells using source-IP + WireGuard public key authentication (no session required). - -Access control is per-service (calendar, files, mail, WebDAV) and enforced at the iptables level. - ---- - -## Extended Connectivity - -`ConnectivityManager` provides per-peer and per-service exit routing: traffic from a specific WireGuard peer (or a specific installed service) can be routed through an alternate exit instead of the cell's default gateway. - -All exit types are optional store services installed from the Services catalog. Each exit type corresponds to a store service ID: - -| Exit type | Store service | Mechanism | -|---|---|---| -| `wireguard_ext` | `wireguard-ext` | WireGuard client tunnel to an external server; iface `wg_ext0`; fwmark `0x10`, table 110 | -| `openvpn` | `openvpn-client` | OpenVPN client tunnel; iface `tun0`; fwmark `0x20`, table 120 | -| `tor` | `tor` | Transparent proxy → Tor SOCKS on port 9040; fwmark `0x30`, table 130 | -| `sshuttle` | `sshuttle` | SSH tunnel via sshuttle to any SSH server; transparent proxy on port 12300; fwmark `0x40`, table 140 | -| `proxy` | `proxy` | HTTP or SOCKS5 upstream proxy via redsocks transparent redirection on port 12345; fwmark `0x50`, table 150 | - -Routing uses fwmark and `ip rule` / `ip route` in separate routing tables inside the `cell-wireguard` container. A kill-switch FORWARD rule drops traffic for configured exits if the exit container is not active. - -Configuration is via `PUT /api/connectivity/peers//exit`. Service-level egress is declared in the service manifest's `egress` block and configured via the Connectivity page in the UI. - ---- - -## Security Model - -- **No open ports for the API** — Flask API binds to `127.0.0.1:3000` only; Caddy proxies HTTPS requests to it. -- **Session auth** — bcrypt passwords, Flask server-side sessions, CSRF double-submit. -- **Setup wizard gate** — all `/api/*` requests return 428 until setup is complete. -- **Role separation** — admin cannot access peer endpoints; peer cannot access admin endpoints. -- **HTTPS everywhere** — Caddy handles TLS; internal services are reached via reverse proxy paths. -- **Internal CA** — VaultManager issues certificates for services that don't use Let's Encrypt. -- **Docker socket isolation** — the Docker socket is mounted only into `cell-api`; other containers have no Docker access. -- **iptables firewall** — FirewallManager manages INPUT/FORWARD rules; WireGuard peer isolation is enforced at the packet level. - ---- - -## Testing - -```bash -make test # unit tests (pytest, ~1900+ functions) -make test-coverage # coverage report in htmlcov/ -``` - -Test layout: -- `tests/` — unit and endpoint tests; no running services required -- `tests/integration/` — require a running PIC stack -- `tests/e2e/` — Playwright UI tests and WireGuard integration tests - -CI: Gitea Actions runs `pytest tests/ --ignore=tests/e2e --ignore=tests/integration` on every push. - ---- - -## Development - -```bash -# Full stack in Docker -make start -make stop -make logs - -# Flask API without Docker (port 3000) -pip install -r api/requirements.txt -python api/app.py - -# React UI dev server (port 5173, proxies /api → :3000) -cd webui && npm install && npm run dev - -# Rebuild containers after code change -make build-api -make build-webui -``` - -Key files: -- `api/app.py` — Flask app, blueprint registration, before-request hooks, health monitor thread -- `api/managers.py` — singleton instantiation of all service managers -- `api/base_service_manager.py` — abstract base class all managers implement -- `api/config_manager.py` — `cell_config.json` read/write/validate/backup -- `api/service_bus.py` — pub/sub event system -- `webui/src/services/api.js` — Axios API client used by all UI pages -- `docker-compose.yml` — container definitions and network topology -- `Makefile` — all operational commands diff --git a/QUICKSTART.md b/QUICKSTART.md deleted file mode 100644 index 7d43067..0000000 --- a/QUICKSTART.md +++ /dev/null @@ -1,255 +0,0 @@ -# Quick Start - -This guide walks through a first-time PIC installation on a clean Linux host. - ---- - -## Prerequisites - -- Linux x86-64 host — Debian, Ubuntu, Fedora, RHEL, or Alpine -- 2 GB+ RAM, 10 GB+ disk -- Always-required ports: 53, 80, 443, 51820/udp -- Email service only (when installed): 25, 587, 993 -- WireGuard kernel module available on the host (`modprobe wireguard`); required — userspace WireGuard is not supported - -The installer handles all software dependencies (git, docker, make, etc.) automatically. - ---- - -## Option A — One-line installer (recommended) - -```bash -curl -fsSL https://install.pic.ngo | sudo bash -``` - -Always review the script before running it: - -```bash -curl -fsSL https://install.pic.ngo | less -``` - -The installer runs 7 steps and prints clean one-line progress for each. Run with `PIC_DEBUG=1` or `--debug` for full verbose output. A complete log is always written to `/var/log/pic-install.log`. - -The installer: -1. Detects your OS and installs Docker, git, make via the system package manager -2. Installs and starts host NTP (chrony) — required for ACME certificate issuance and DDNS token registration -3. Creates a `pic` system user and adds it to the `docker` group -4. Clones the repository to `/opt/pic` -5. Runs `make install` — generates keys and config, writes a systemd unit. The admin password is printed once here; it does not appear again. -6. Runs `make start-core` to bring up the six core containers -7. Enables the `pic` systemd unit so the stack starts on reboot, then waits for the API health check - -When it finishes, open the URL it prints: - -``` -http://:8081/setup -``` - ---- - -## Option B — Manual install - -Use this if you want to control where PIC is installed, or if you are installing on a machine that already has Docker. - -```bash -git clone https://git.pic.ngo/roof/pic.git pic -cd pic -sudo make install -make start-core -``` - -Then open `http://:8081/setup` in a browser. - -Note: install host NTP before running `make install` if you plan to use `pic_ngo` domain mode. The installer does this automatically in Option A. - -```bash -sudo apt-get install -y chrony && sudo systemctl enable --now chrony -``` - ---- - -## Complete the setup wizard - -The setup wizard appears automatically on first start. All API requests redirect to `/setup` until it is finished. - -The wizard asks for: - -- **Cell name** — used for hostnames and DDNS subdomain. Lowercase letters, digits, hyphens, 2–31 characters. Example: `myhome`. -- **Domain mode** — how HTTPS certificates are issued: - - `pic_ngo` — automatic `.pic.ngo` subdomain with a wildcard Let's Encrypt cert via DNS-01 (recommended for internet-facing cells; requires accurate host clock) - - `cloudflare` — wildcard Let's Encrypt cert via Cloudflare DNS-01 (bring your own domain) - - `duckdns` — Let's Encrypt via DuckDNS DNS-01 - - `http01` — Let's Encrypt via HTTP-01 (no wildcard; cell must be reachable on port 80) - - `lan` — internal CA only, no internet required (for LAN-only installs) -- **Timezone** -- **Services to install** — email, calendar, files (optional; installed in the background after setup completes; can be added later via the Services store page) -- **Admin password** — minimum 12 characters, must contain uppercase, lowercase, and a digit - -Click **Complete Setup**. The wizard creates the admin account, writes cell identity to `config/api/cell_config.json`, and redirects to the login page. Any services you selected begin installing in the background. - ---- - -## Log in - -After the wizard you are redirected to `/login`. - -- **Username:** `admin` -- **Password:** the password you set in the wizard - ---- - -## Add a WireGuard peer - -Go to **Peers** in the sidebar. - -1. Click **Add Peer**. -2. Enter a peer name (e.g. `laptop`). -3. The API generates a key pair and assigns the next available VPN IP automatically. -4. Click the QR code icon to display the peer configuration as a QR code. -5. Scan the QR code with a WireGuard client (Android, iOS, or the WireGuard desktop app). - -Once connected, `*.cell` names resolve through the cell's CoreDNS and traffic can be routed through the cell. - ---- - -## Installing and managing services - -Email, calendar, and file storage are optional services installed from the built-in service store. They are not running by default. - -**To install a service:** - -1. Go to **Services** in the sidebar. -2. Find the service card (Email, Calendar, Files, or any other listed service). -3. Click **Install**. PIC fetches the manifest, starts the container, and wires up DNS and Caddy routes automatically. -4. The service appears in the sidebar navigation once installation completes. - -**To check service status:** - -The Services page shows each installed service as "running" or "stopped". You can also check via the API: - -```bash -curl -s http://:3000/api/services/active -``` - -**To uninstall a service:** - -Click **Uninstall** on the service card. The container is stopped and removed. Data in `data/services//` is kept on disk unless you delete it manually. - ---- - -## Day-to-day operations - -```bash -# Check container status and API health -make status - -# Follow logs from all services -make logs - -# Follow logs from a single service -make logs-api -make logs-wireguard -make logs-caddy - -# Open a shell inside a container -make shell-api -make shell-dns -``` - ---- - -## Backup and restore - -```bash -make backup # archives config/ and data/ into backups/cell-backup-.tar.gz -make restore # list available backups -``` - -The backup archive is written mode `0600`. It contains secrets and key material — WireGuard keys, the internal CA, vault keys, admin credentials, DDNS token, cell links, and Caddy certificates. Store it securely. - -Data volumes for installed store services (email mailboxes, calendar data, file storage) are captured separately via the API-driven backup (`POST /api/config/backup`), which also supports an optional passphrase for encryption at rest. The encrypted file is named `.tar.gz.age`. - -To restore from a `make backup` archive: - -```bash -tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz -make restart -``` - -After restore, the API re-generates the Caddyfile and Corefile from the restored config and re-applies routing rules automatically. - ---- - -## Updating PIC - -```bash -make update # git pull + rebuild + restart -``` - ---- - -## Uninstalling - -```bash -make uninstall # stops containers; prompts to also delete config/ and data/ -``` - ---- - -## Troubleshooting - -### Containers not starting - -```bash -make logs -make logs-api -``` - -Look for errors about missing config files or port conflicts. - -### Port 53 already in use - -On Ubuntu and Debian, `systemd-resolved` listens on port 53. Disable it: - -```bash -sudo systemctl disable --now systemd-resolved -sudo rm /etc/resolv.conf -echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf -``` - -Then run `make start` again. - -### WireGuard container fails to start - -The WireGuard container runs unprivileged (NET_ADMIN only, no privileged mode). It requires the host kernel's WireGuard module — either compiled in (Linux 5.6+) or loadable. - -```bash -sudo modprobe wireguard -``` - -On minimal installs you may need `wireguard-tools` and the kernel headers for the running kernel. On kernels that lack a builtin WireGuard module, check your distro's `wireguard-dkms` or `wireguard-linux-compat` package. - -### API returns 428 and redirects to /setup - -The first-run wizard has not been completed. Open `http://:8081` and finish the wizard. - -### API returns 401 / UI shows "Not authenticated" - -Your session expired or you have not logged in. Go to `http://:8081/login`. - -### API returns 503 "Authentication not configured" - -The auth file exists but contains no accounts. To recover: - -```bash -make reset-admin-password -``` - -This generates a new admin password and prints it. - -### Forgot the admin password - -```bash -make show-admin-password # print current password -make reset-admin-password # generate a new random password -``` diff --git a/README.md b/README.md index 7321ea3..eeee0c5 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ The Flask API (`api/app.py`) contains REST endpoints and a background health-mon The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls go through `src/services/api.js` (Axios). -**Web UI pages:** Dashboard, Peers, Network Services, WireGuard, Email, Calendar, Files, Routing, Vault, Containers, Cell Network, Connectivity, Service Store, Logs, Settings. +**Web UI pages:** Dashboard, Peers, Network Services, WireGuard, Connectivity (tunnels, proxies, SSH, Tor, cells, assignments), Services (store catalog + per-service pages), Routing, Vault, Containers, Activity, Logs, Settings — plus peer-facing My Services and Account pages. --- @@ -35,9 +35,9 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g - **Session-based auth** — admin and peer roles. All `/api/*` endpoints require an authenticated session after setup. CSRF protection on all state-changing requests. - **WireGuard VPN** — peer lifecycle management, automatic key generation, QR code config export, per-peer routing policy. - **Caddy HTTPS** — automatic TLS via Let's Encrypt (DNS-01 or HTTP-01) or an internal CA, depending on domain mode. -- **DDNS (pic.ngo)** — registers a `.pic.ngo` subdomain. Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`. A background thread re-publishes the public IP every 5 minutes. -- **Service store** — install/remove optional third-party services from the `pic-services` index at `git.pic.ngo`. Manifests declare container images, Caddy routes, and iptables rules. -- **Extended connectivity** — per-peer egress routing through alternate exits: WireGuard external, OpenVPN, Tor, sshuttle (SSH tunnel), or proxy (HTTP/SOCKS5 via redsocks). Exit nodes are optional store services. Per-service egress policy is also supported. Routing uses fwmark and `ip rule` in the WireGuard container. +- **DDNS (pic.ngo)** — registers a `.pic.ngo` subdomain. Supported providers: `pic_ngo`, `cloudflare`, `duckdns`. A background thread re-publishes the public IP every 5 minutes. +- **Service store** — install/remove optional third-party services from the `pic-services` index at `git.pic.ngo`. Manifests declare container images, Caddy routes, and iptables rules. Store images are digest-pinned and cosign-signed by the build pipeline; the cell verifies signatures before starting a container (enforced by default). +- **Extended connectivity** — named connection instances per exit type: WireGuard external, OpenVPN, Tor, sshuttle (SSH tunnel), or proxy (HTTP/SOCKS5 via redsocks), plus cell-relay through another cell. Peers are assigned per-peer to a connection with configurable fail-open/fail-closed; per-connection health is tracked. Per-service egress policy is also supported. Routing uses per-instance fwmarks and `ip rule` in the WireGuard container. - **Cell-to-cell networking** — WireGuard-based site-to-site links between PIC cells with service-level access control (calendar, files, mail, WebDAV) and a peer-sync protocol. - **Certificate authority** — `vault_manager` issues and revokes TLS certificates for internal services. - **Network services** — CoreDNS (`.cell` TLD and split-horizon DNS for the cell domain), chrony NTP. @@ -48,6 +48,8 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g - **Container manager** — start/stop/inspect containers, pull images, manage volumes via the Docker SDK. - **Firewall manager** — iptables rule management (`firewall_manager.py`). - **Structured logging** — JSON logs with rotation (5 MB / 5 backups per service), log search, and per-service verbosity control. +- **Audit log** — append-only, hash-chained change log of all admin actions, with CSV export and an Activity page in the UI. +- **Backup / restore** — full backup of config, secrets, key material, and live service data volumes, with optional passphrase encryption; ordered restore with automatic runtime reapply. --- @@ -61,9 +63,15 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g --- +## Documentation + +Full documentation lives in the [project wiki](https://git.pic.ngo/roof/pic/wiki) — installation walkthrough, admin guide (setup, domains/TLS, services, connectivity, peers, backup, logging/audit, troubleshooting), user guide, and developer documentation (architecture, API reference, building store services, testing). + +--- + ## Quick Start -See [QUICKSTART.md](QUICKSTART.md) for step-by-step instructions. +See the wiki's [Setup and First Run](https://git.pic.ngo/roof/pic/wiki/Admin-Setup) for step-by-step instructions. The short version — one-line installer (recommended): @@ -212,4 +220,4 @@ make reset-admin-password # generate and set a new random admin password ## License -MIT — see [LICENSE](LICENSE). +MIT. diff --git a/api/API_DOCUMENTATION.md b/api/API_DOCUMENTATION.md deleted file mode 100644 index 26f5d2c..0000000 --- a/api/API_DOCUMENTATION.md +++ /dev/null @@ -1,1286 +0,0 @@ -# Personal Internet Cell API Documentation - -## Overview - -The Personal Internet Cell API provides a comprehensive REST interface for managing all aspects of a personal internet cell, including network services, VPN management, email/calendar/file services, routing, and security features. - -**Base URL**: `http://localhost:3000/api` -**Content-Type**: `application/json` -**Authentication**: Currently supports local access only (127.0.0.1, ::1, localhost) - -## Table of Contents - -1. [Authentication](#authentication) -2. [Error Handling](#error-handling) -3. [Health & Status](#health--status) -4. [Configuration Management](#configuration-management) -5. [Network Services](#network-services) -6. [WireGuard VPN](#wireguard-vpn) -7. [Peer Management](#peer-management) -8. [Email Services](#email-services) -9. [Calendar Services](#calendar-services) -10. [File Services](#file-services) -11. [Routing Services](#routing-services) -12. [Vault & Security](#vault--security) -13. [Container Management](#container-management) -14. [Logging & Monitoring](#logging--monitoring) - -## Authentication - -Currently, the API only accepts requests from localhost for security reasons. All endpoints require the client to be running on the same machine as the API server. - -```bash -# Valid client IPs -127.0.0.1 -::1 -localhost -``` - -## Error Handling - -All API endpoints return consistent error responses: - -```json -{ - "error": "Error description", - "timestamp": "2024-01-01T12:00:00Z", - "service": "service_name" -} -``` - -Common HTTP status codes: -- `200` - Success -- `400` - Bad Request (invalid parameters) -- `403` - Forbidden (access denied) -- `404` - Not Found -- `500` - Internal Server Error - -## Health & Status - -### Get Cell Status - -**GET** `/status` - -Returns overall cell status including all services. - -**Response:** -```json -{ - "cell_name": "personal-internet-cell", - "domain": "cell.local", - "uptime": 3600, - "peers_count": 5, - "services": { - "network": { - "running": true, - "status": "online" - }, - "wireguard": { - "running": true, - "status": "online" - }, - "email": { - "running": true, - "status": "online" - }, - "calendar": { - "running": true, - "status": "online" - }, - "files": { - "running": true, - "status": "online" - } - }, - "timestamp": "2024-01-01T12:00:00Z" -} -``` - -### Health Check - -**GET** `/health` - -Simple health check endpoint. - -**Response:** -```json -{ - "status": "healthy", - "timestamp": "2024-01-01T12:00:00Z", - "version": "1.0.0" -} -``` - -### Get All Services Status - -**GET** `/services/status` - -Returns detailed status of all services. - -**Response:** -```json -{ - "network": { - "running": true, - "status": "online", - "dns_running": true, - "dhcp_running": true, - "ntp_running": true - }, - "wireguard": { - "running": true, - "status": "online", - "peers_count": 5 - }, - "email": { - "running": true, - "status": "online", - "users_count": 3 - }, - "calendar": { - "running": true, - "status": "online", - "calendars_count": 2 - }, - "files": { - "running": true, - "status": "online", - "storage_used": "1.2GB" - }, - "routing": { - "running": true, - "status": "online", - "nat_rules_count": 3 - }, - "vault": { - "running": true, - "status": "online", - "ca_configured": true, - "certificates_count": 5 - }, - "timestamp": "2024-01-01T12:00:00Z" -} -``` - -## Configuration Management - -### Get Configuration - -**GET** `/config` - -Returns current cell configuration. - -**Response:** -```json -{ - "cell_name": "personal-internet-cell", - "domain": "cell.local", - "ip_range": "10.0.0.0/24", - "wireguard_port": 51820, - "dns_port": 53, - "dhcp_range": "10.0.0.100-10.0.0.200" -} -``` - -### Update Configuration - -**PUT** `/config` - -Update cell configuration. - -**Request:** -```json -{ - "cell_name": "my-cell", - "domain": "mycell.local", - "ip_range": "192.168.1.0/24" -} -``` - -**Response:** -```json -{ - "message": "Configuration updated successfully" -} -``` - -## Network Services - -### Get DNS Records - -**GET** `/dns/records` - -Returns all DNS records. - -**Response:** -```json -[ - { - "name": "www", - "type": "A", - "value": "10.0.0.10", - "ttl": 3600 - }, - { - "name": "mail", - "type": "CNAME", - "value": "www", - "ttl": 3600 - } -] -``` - -### Add DNS Record - -**POST** `/dns/records` - -Add a new DNS record. - -**Request:** -```json -{ - "zone": "cell.local", - "name": "api", - "type": "A", - "value": "10.0.0.5", - "ttl": 3600 -} -``` - -### Get DHCP Leases - -**GET** `/dhcp/leases` - -Returns current DHCP leases. - -**Response:** -```json -[ - { - "mac": "00:11:22:33:44:55", - "ip": "10.0.0.100", - "hostname": "laptop", - "timestamp": "2024-01-01T10:00:00Z" - } -] -``` - -### Add DHCP Reservation - -**POST** `/dhcp/reservations` - -Add a DHCP reservation. - -**Request:** -```json -{ - "mac": "00:11:22:33:44:55", - "ip": "10.0.0.50", - "hostname": "server" -} -``` - -### Get Network Info - -**GET** `/network/info` - -Returns detailed network information. - -**Response:** -```json -{ - "interfaces": [ - { - "name": "eth0", - "ip": "192.168.1.100", - "mac": "00:11:22:33:44:55", - "status": "up" - } - ], - "gateway": "192.168.1.1", - "dns_servers": ["8.8.8.8", "1.1.1.1"], - "routing_table": [ - { - "destination": "0.0.0.0/0", - "gateway": "192.168.1.1", - "interface": "eth0" - } - ] -} -``` - -### Get DNS Status - -**GET** `/dns/status` - -Returns DNS service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "zones_count": 2, - "records_count": 15, - "queries_per_second": 25.5, - "cache_hit_rate": 0.85 -} -``` - -### Get NTP Status - -**GET** `/ntp/status` - -Returns NTP service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "synchronized": true, - "stratum": 3, - "reference_id": "192.168.1.1", - "offset": 0.001234 -} -``` - -## WireGuard VPN - -### Get WireGuard Status - -**GET** `/wireguard/status` - -Returns WireGuard service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "interface": "wg0", - "peers_count": 5, - "total_traffic": { - "bytes_sent": 1048576, - "bytes_received": 2097152 - } -} -``` - -### Get WireGuard Peers - -**GET** `/wireguard/peers` - -Returns all WireGuard peers. - -**Response:** -```json -[ - { - "name": "alice", - "public_key": "abc123...", - "ip": "10.0.0.2", - "allowed_ips": "10.0.0.2/32", - "last_handshake": "2024-01-01T12:00:00Z", - "transfer_rx": 1048576, - "transfer_tx": 2097152 - } -] -``` - -### Add WireGuard Peer - -**POST** `/wireguard/peers` - -Add a new WireGuard peer. - -**Request:** -```json -{ - "name": "bob", - "public_key": "def456...", - "ip": "10.0.0.3", - "allowed_ips": "10.0.0.3/32" -} -``` - -### Generate Peer Keys - -**POST** `/wireguard/keys/peer` - -Generate new WireGuard keys for a peer. - -**Request:** -```json -{ - "peer_name": "charlie" -} -``` - -**Response:** -```json -{ - "private_key": "private_key_here", - "public_key": "public_key_here", - "peer_name": "charlie" -} -``` - -## Peer Management - -### Get All Peers - -**GET** `/peers` - -Returns all registered peers. - -**Response:** -```json -[ - { - "name": "alice", - "ip": "10.0.0.2", - "public_key": "abc123...", - "added_at": "2024-01-01T10:00:00Z" - } -] -``` - -### Add Peer - -**POST** `/peers` - -Add a new peer to the registry. - -**Request:** -```json -{ - "name": "bob", - "ip": "10.0.0.3", - "public_key": "def456..." -} -``` - -**Response:** -```json -{ - "message": "Peer bob added successfully" -} -``` - -### Remove Peer - -**DELETE** `/peers/{peer_name}` - -Remove a peer from the registry. - -**Response:** -```json -{ - "message": "Peer bob removed successfully" -} -``` - -### Update Peer IP - -**PUT** `/peers/{peer_name}/update-ip` - -Update a peer's IP address. - -**Request:** -```json -{ - "ip": "10.0.0.4" -} -``` - -## Email Services - -### Get Email Status - -**GET** `/email/status` - -Returns email service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "smtp_running": true, - "imap_running": true, - "users_count": 3, - "domain": "cell.local" -} -``` - -### Get Email Users - -**GET** `/email/users` - -Returns all email users. - -**Response:** -```json -[ - { - "username": "alice", - "domain": "cell.local", - "email": "alice@cell.local", - "created_at": "2024-01-01T10:00:00Z" - } -] -``` - -### Create Email User - -**POST** `/email/users` - -Create a new email user. - -**Request:** -```json -{ - "username": "bob", - "domain": "cell.local", - "password": "secure_password" -} -``` - -### Delete Email User - -**DELETE** `/email/users/{username}` - -Delete an email user. - -**Response:** -```json -{ - "message": "User bob deleted successfully" -} -``` - -### Send Email - -**POST** `/email/send` - -Send an email. - -**Request:** -```json -{ - "from_email": "alice@cell.local", - "to_email": "bob@cell.local", - "subject": "Hello", - "body": "This is a test email", - "html_body": "

This is a test email

" -} -``` - -## Calendar Services - -### Get Calendar Status - -**GET** `/calendar/status` - -Returns calendar service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "users_count": 2, - "calendars_count": 4, - "events_count": 25 -} -``` - -### Get Calendar Users - -**GET** `/calendar/users` - -Returns all calendar users. - -**Response:** -```json -[ - { - "username": "alice", - "calendars_count": 2, - "events_count": 15 - } -] -``` - -### Create Calendar User - -**POST** `/calendar/users` - -Create a new calendar user. - -**Request:** -```json -{ - "username": "bob", - "password": "secure_password" -} -``` - -### Create Calendar - -**POST** `/calendar/calendars` - -Create a new calendar. - -**Request:** -```json -{ - "username": "alice", - "calendar_name": "Work", - "description": "Work calendar" -} -``` - -### Get Calendar Events - -**GET** `/calendar/events/{username}/{calendar_name}` - -Returns calendar events. - -**Query Parameters:** -- `start_date`: Start date (YYYY-MM-DD) -- `end_date`: End date (YYYY-MM-DD) - -**Response:** -```json -[ - { - "id": "event_1", - "title": "Meeting", - "start": "2024-01-01T10:00:00Z", - "end": "2024-01-01T11:00:00Z", - "description": "Team meeting" - } -] -``` - -## File Services - -### Get File Status - -**GET** `/files/status` - -Returns file service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "users_count": 3, - "total_storage": "100GB", - "used_storage": "25GB" -} -``` - -### Get File Users - -**GET** `/files/users` - -Returns all file storage users. - -**Response:** -```json -[ - { - "username": "alice", - "storage_used": "5GB", - "files_count": 150 - } -] -``` - -### Create File User - -**POST** `/files/users` - -Create a new file storage user. - -**Request:** -```json -{ - "username": "bob", - "password": "secure_password", - "quota": "10GB" -} -``` - -### List Files - -**GET** `/files/list/{username}` - -List files for a user. - -**Query Parameters:** -- `folder`: Folder path (optional) - -**Response:** -```json -[ - { - "name": "document.pdf", - "path": "/documents/document.pdf", - "size": 1048576, - "modified": "2024-01-01T10:00:00Z", - "type": "file" - }, - { - "name": "documents", - "path": "/documents", - "type": "folder" - } -] -``` - -### Upload File - -**POST** `/files/upload/{username}` - -Upload a file. - -**Request:** Multipart form data -- `file`: File to upload -- `path`: Destination path (optional) - -### Download File - -**GET** `/files/download/{username}/{file_path}` - -Download a file. - -**Response:** File content - -## Routing Services - -### Get Routing Status - -**GET** `/routing/status` - -Returns routing service status. - -**Response:** -```json -{ - "running": true, - "status": "online", - "nat_enabled": true, - "firewall_enabled": true, - "nat_rules_count": 3, - "firewall_rules_count": 10, - "peer_routes_count": 5 -} -``` - -### Get NAT Rules - -**GET** `/routing/nat` - -Returns all NAT rules. - -**Response:** -```json -{ - "nat_rules": [ - { - "id": "rule_1", - "source_network": "10.0.0.0/24", - "target_interface": "eth0", - "masquerade": true, - "nat_type": "MASQUERADE", - "protocol": "ALL" - } - ] -} -``` - -### Add NAT Rule - -**POST** `/routing/nat` - -Add a new NAT rule. - -**Request:** -```json -{ - "source_network": "10.0.0.0/24", - "target_interface": "eth0", - "masquerade": true, - "nat_type": "MASQUERADE", - "protocol": "ALL" -} -``` - -### Get Firewall Rules - -**GET** `/routing/firewall` - -Returns all firewall rules. - -**Response:** -```json -{ - "firewall_rules": [ - { - "id": "rule_1", - "rule_type": "INPUT", - "source": "0.0.0.0/0", - "destination": "10.0.0.0/24", - "action": "ACCEPT", - "protocol": "TCP", - "port": "22" - } - ] -} -``` - -### Add Firewall Rule - -**POST** `/routing/firewall` - -Add a new firewall rule. - -**Request:** -```json -{ - "rule_type": "INPUT", - "source": "0.0.0.0/0", - "destination": "10.0.0.0/24", - "action": "ACCEPT", - "protocol": "TCP", - "port": "22" -} -``` - -### Get Peer Routes - -**GET** `/routing/peers` - -Returns all peer routes. - -**Response:** -```json -{ - "peer_routes": [ - { - "peer_name": "alice", - "peer_ip": "10.0.0.2", - "allowed_networks": ["192.168.1.0/24"], - "route_type": "split" - } - ] -} -``` - -### Add Peer Route - -**POST** `/routing/peers` - -Add a new peer route. - -**Request:** -```json -{ - "peer_name": "bob", - "peer_ip": "10.0.0.3", - "allowed_networks": ["192.168.2.0/24"], - "route_type": "bridge" -} -``` - -## Vault & Security - -### Get Vault Status - -**GET** `/vault/status` - -Returns vault service status. - -**Response:** -```json -{ - "ca_configured": true, - "age_configured": true, - "fernet_configured": true, - "certificates_count": 5, - "trusted_keys_count": 3, - "trust_chains_count": 2 -} -``` - -### Get Certificates - -**GET** `/vault/certificates` - -Returns all certificates. - -**Response:** -```json -[ - { - "common_name": "api.cell.local", - "domains": ["api.cell.local", "www.cell.local"], - "created": "2024-01-01T10:00:00Z", - "expires": "2025-01-01T10:00:00Z", - "status": "valid" - } -] -``` - -### Generate Certificate - -**POST** `/vault/certificates` - -Generate a new certificate. - -**Request:** -```json -{ - "common_name": "mail.cell.local", - "domains": ["mail.cell.local", "smtp.cell.local"], - "key_size": 2048, - "days": 365 -} -``` - -### Get CA Certificate - -**GET** `/vault/ca/certificate` - -Returns the CA certificate. - -**Response:** -```json -{ - "certificate": "-----BEGIN CERTIFICATE-----\n..." -} -``` - -### Get Trusted Keys - -**GET** `/vault/trust/keys` - -Returns all trusted keys. - -**Response:** -```json -[ - { - "name": "alice", - "public_key": "age1...", - "trust_level": "direct", - "added_at": "2024-01-01T10:00:00Z" - } -] -``` - -### Add Trusted Key - -**POST** `/vault/trust/keys` - -Add a new trusted key. - -**Request:** -```json -{ - "name": "bob", - "public_key": "age1...", - "trust_level": "direct" -} -``` - -## Container Management - -### List Containers - -**GET** `/containers` - -Returns all containers. - -**Response:** -```json -[ - { - "id": "abc123", - "name": "cell-api", - "status": "running", - "image": "personalinternetcell-api:latest", - "labels": {} - } -] -``` - -### Start Container - -**POST** `/containers/{name}/start` - -Start a container. - -**Response:** -```json -{ - "started": true -} -``` - -### Stop Container - -**POST** `/containers/{name}/stop` - -Stop a container. - -**Response:** -```json -{ - "stopped": true -} -``` - -### Restart Container - -**POST** `/containers/{name}/restart` - -Restart a container. - -**Response:** -```json -{ - "restarted": true -} -``` - -### Get Container Logs - -**GET** `/containers/{name}/logs` - -Returns container logs. - -**Query Parameters:** -- `tail`: Number of lines to return (default: 100) - -**Response:** -```json -{ - "logs": "Container log output..." -} -``` - -### Get Container Stats - -**GET** `/containers/{name}/stats` - -Returns container statistics. - -**Response:** -```json -{ - "cpu_usage": 2.5, - "memory_usage": "512MB", - "network_rx": 1048576, - "network_tx": 2097152, - "disk_usage": "1GB" -} -``` - -## Logging & Monitoring - -### Get Backend Logs - -**GET** `/logs` - -Returns backend log file contents. - -**Query Parameters:** -- `lines`: Number of lines to return (default: 100) - -**Response:** -```json -{ - "log": "Log file contents..." -} -``` - -### Get Health History - -**GET** `/health/history` - -Returns recent health check results. - -**Response:** -```json -[ - { - "timestamp": "2024-01-01T12:00:00Z", - "network": {"status": "online"}, - "wireguard": {"status": "online"}, - "email": {"status": "online"}, - "calendar": {"status": "online"}, - "files": {"status": "online"}, - "routing": {"status": "online"}, - "vault": {"status": "online"}, - "alerts": [] - } -] -``` - -## Usage Examples - -### Python Client Example - -```python -import requests - -API_BASE = "http://localhost:3000/api" - -def get_cell_status(): - response = requests.get(f"{API_BASE}/status") - return response.json() - -def add_peer(name, ip, public_key): - data = { - "name": name, - "ip": ip, - "public_key": public_key - } - response = requests.post(f"{API_BASE}/peers", json=data) - return response.json() - -def get_service_logs(service, lines=50): - response = requests.get(f"{API_BASE}/logs?lines={lines}") - return response.json() - -# Usage -status = get_cell_status() -print(f"Cell status: {status['cell_name']}") - -result = add_peer("alice", "10.0.0.2", "abc123...") -print(f"Add peer result: {result}") -``` - -### cURL Examples - -```bash -# Get cell status -curl -X GET http://localhost:3000/api/status - -# Add a peer -curl -X POST http://localhost:3000/api/peers \ - -H "Content-Type: application/json" \ - -d '{"name": "alice", "ip": "10.0.0.2", "public_key": "abc123..."}' - -# Get service logs -curl -X GET "http://localhost:3000/api/logs?lines=100" - -# Update configuration -curl -X PUT http://localhost:3000/api/config \ - -H "Content-Type: application/json" \ - -d '{"cell_name": "my-cell"}' -``` - -### JavaScript Client Example - -```javascript -const API_BASE = 'http://localhost:3000/api'; - -async function getCellStatus() { - const response = await fetch(`${API_BASE}/status`); - return await response.json(); -} - -async function addPeer(name, ip, publicKey) { - const response = await fetch(`${API_BASE}/peers`, { - method: 'POST', - headers: { - 'Content-Type': 'application/json' - }, - body: JSON.stringify({ - name: name, - ip: ip, - public_key: publicKey - }) - }); - return await response.json(); -} - -// Usage -getCellStatus().then(status => { - console.log('Cell status:', status); -}); - -addPeer('alice', '10.0.0.2', 'abc123...').then(result => { - console.log('Add peer result:', result); -}); -``` - -## Rate Limiting - -Currently, no rate limiting is implemented. However, it's recommended to: - -- Limit requests to reasonable frequencies -- Implement exponential backoff for retries -- Cache responses when appropriate - -## Best Practices - -1. **Error Handling**: Always check for errors in responses -2. **Logging**: Use appropriate log levels for debugging -3. **Configuration**: Validate configuration before applying -4. **Security**: Keep private keys and secrets secure -5. **Monitoring**: Regularly check service health -6. **Backup**: Create regular configuration backups - -## Troubleshooting - -### Common Issues - -1. **Connection Refused**: Ensure the API server is running -2. **403 Forbidden**: Check that you're accessing from localhost -3. **Service Not Found**: Verify the service is properly configured -4. **Configuration Errors**: Check configuration validation - -### Debug Commands - -```bash -# Check API server status -curl -X GET http://localhost:3000/api/health - -# Check all services status -curl -X GET http://localhost:3000/api/services/status - -# Get recent logs -curl -X GET "http://localhost:3000/api/logs?lines=50" - -# Check health history -curl -X GET http://localhost:3000/api/health/history -``` - -## Version History - -- **v1.0.0**: Initial API release with basic functionality -- **v1.1.0**: Added configuration management and backup features -- **v1.2.0**: Enhanced logging and monitoring capabilities -- **v1.3.0**: Added service bus and event-driven architecture -- **v1.4.0**: Improved error handling and validation - -## Support - -For issues and questions: - -1. Check the logs: `GET /api/logs` -2. Verify service status: `GET /api/services/status` -3. Review configuration: `GET /api/config` -4. Check health history: `GET /api/health/history` - -## License - -This API is part of the Personal Internet Cell project and is licensed under the MIT License. \ No newline at end of file diff --git a/docs/service-developer-guide.md b/docs/service-developer-guide.md deleted file mode 100644 index 0647c05..0000000 --- a/docs/service-developer-guide.md +++ /dev/null @@ -1,743 +0,0 @@ -# PIC Service Developer Guide - -This guide is for developers who want to build services that integrate with Personal Internet Cell (PIC). It covers the manifest format, how PIC wires up routing, DNS, backup, and account provisioning for your service, and how to package and submit your work. - -**Prerequisites:** you should be comfortable with Docker, Docker Compose, and basic Linux networking. You do not need to know Python to build a store service. - ---- - -## Table of Contents - -1. [What a PIC service is](#1-what-a-pic-service-is) -2. [Manifest reference](#2-manifest-reference) -3. [Compose template variables](#3-compose-template-variables) -4. [Account provisioning interface](#4-account-provisioning-interface) -5. [Backup integration](#5-backup-integration) -6. [Egress routing](#6-egress-routing) -7. [Quick-start example](#7-quick-start-example) -8. [Reference implementations](#8-reference-implementations) -9. [Submitting to the store](#9-submitting-to-the-store) - ---- - -## 1. What a PIC service is - -A PIC service is a Docker container (or a set of containers) that plugs into the PIC ecosystem through a single JSON file called the **manifest**. The manifest tells PIC everything it needs to know: - -- How to route HTTPS traffic to the service through Caddy -- What subdomains to expose -- Which users get accounts on the service and what credentials they receive -- Which paths to include in automated backups -- Which outbound network interfaces the service is allowed to use - -All PIC services are **store services** — optional packages installed by the cell admin from the `pic-services` catalog. PIC downloads the manifest, renders a per-service Docker Compose file, and starts the containers. The core PIC stack (DNS, NTP, WireGuard, Caddy, API, WebUI) runs independently of any installed services. - -The email, calendar, and files services (in `pic-services/services/`) are the reference implementations and show the full feature set. The `ServiceRegistry` in `api/service_registry.py` is the single source of truth for all installed services. `CaddyManager`, the backup system, and the peer services endpoint all read from it rather than from hardcoded lists. - ---- - -## 2. Manifest reference - -The manifest is a JSON file with `"schema_version": 3`. Every field is described below. The `email`, `calendar`, and `files` manifests in `pic-services/services/` are the canonical reference examples. - -### Top-level identity fields - -| Field | Type | Required | Description | -|---|---|---|---| -| `schema_version` | integer | yes | Must be `3`. | -| `id` | string | yes | Unique service identifier, lowercase, no spaces (e.g. `"notes"`). Must match the directory name for builtins, or the store index entry for store services. | -| `name` | string | yes | Human-readable display name (e.g. `"Notes"`). | -| `description` | string | yes | One-sentence description shown in the UI. | -| `version` | string | yes | Semver string for the service package itself (e.g. `"1.0.0"`). | -| `author` | string | yes | Your name or organisation. | -| `kind` | string | yes | Must be `"store"`. | -| `min_pic_version` | string | no | Minimum PIC version required (e.g. `"1.0"`). | - -```json -{ - "schema_version": 3, - "id": "notes", - "name": "Notes", - "description": "Self-hosted Markdown notes with full-text search", - "version": "1.0.0", - "author": "acme", - "kind": "store", - "min_pic_version": "1.0" -} -``` - -### `capabilities` - -A set of boolean flags that tell PIC which integrations to activate for your service. - -| Field | Type | Default | Description | -|---|---|---|---| -| `has_subdomain` | bool | `false` | The service gets a subdomain and a Caddy reverse-proxy route. Requires `subdomain` and `backend`. | -| `has_accounts` | bool | `false` | The service provisions per-peer accounts. Requires `accounts`. | -| `has_admin_config` | bool | `false` | The service has admin-configurable fields. Requires `config_schema`. | -| `has_storage` | bool | `false` | The service has data worth backing up. Requires `backup`. | -| `has_egress` | bool | `false` | The admin can choose which outbound interface this service uses. Requires `egress`. | -| `has_api_hooks` | bool | `false` | Reserved for future use; set `false`. | - -```json -"capabilities": { - "has_subdomain": true, - "has_accounts": true, - "has_admin_config": false, - "has_storage": true, - "has_egress": false, - "has_api_hooks": false -} -``` - -### `subdomain`, `extra_subdomains`, `backend`, `extra_backends` - -These fields are only read when `has_subdomain` is `true`. - -| Field | Type | Required | Description | -|---|---|---|---| -| `subdomain` | string | yes (if `has_subdomain`) | The primary subdomain (e.g. `"notes"`). Results in `notes.`. Must not collide with reserved names: `api`, `webui`, `admin`, `www`, `ns1`, `ns2`, `git`, `registry`, `install`. | -| `extra_subdomains` | array of strings | no | Additional subdomains that point to the same backend (e.g. `["webmail"]`). | -| `backend` | string | yes (if `has_subdomain`) | The container-name:port combination that Caddy proxies to (e.g. `"cell-notes:8080"`). Uses Docker DNS on the `cell-network`. | -| `extra_backends` | object | no | Maps extra subdomain names to separate backends. Key is the subdomain string; value is the backend string. The email service uses this to send `webdav.*` to a different container than `files.*`. | - -```json -"subdomain": "notes", -"extra_subdomains": [], -"backend": "cell-notes:8080" -``` - -**Validation at runtime:** `ServiceRegistry.get_caddy_routes()` validates all subdomain and backend values before passing them to CaddyManager or NetworkManager. Any entry whose `subdomain` does not match `^[a-z][a-z0-9-]{0,30}$`, whose `backend` does not match `^[A-Za-z0-9._-]+:\d{1,5}$`, or whose `subdomain` appears in the reserved list is silently skipped with a warning log. The same validation applies to `extra_subdomains` and `extra_backends` keys/values. For store services, this validation is also performed during installation by `ServiceStoreManager._validate_manifest()`. - -### `containers` - -Array of container names that belong to this service. Used by the UI and log viewer. For builtins this is informational; for store services PIC only manages the single container declared in the manifest. - -```json -"containers": ["cell-notes"] -``` - -### `config_schema` - -Defines admin-configurable fields for this service. When `has_admin_config` is `true`, the UI renders a settings form from this schema. PIC stores admin-saved values in `cell_config.json` and merges them with your `default` values at runtime. The merged result is available as the `config` key when `ServiceRegistry.get()` returns your service. - -Each field is an object: - -| Key | Type | Required | Description | -|---|---|---|---| -| `type` | string | yes | One of `"string"`, `"integer"`, `"boolean"`. | -| `label` | string | yes | Human-readable label for the settings form. | -| `required` | bool | no | Whether the field must have a value before the service starts. | -| `default` | any | no | Default value used when the admin has not set one. | -| `min` / `max` | integer | no (integer only) | Inclusive bounds for integer fields. | - -```json -"config_schema": { - "port": { - "type": "integer", - "label": "Internal HTTP port", - "default": 8080, - "min": 1, - "max": 65535 - }, - "storage_path": { - "type": "string", - "label": "Data directory inside container", - "default": "/data/notes" - } -} -``` - -### `peer_config_template` - -When a peer is provisioned on this service, PIC fills this template and returns the result to the peer as their connection info. Template substitution tokens: - -| Token | Replaced with | -|---|---| -| `{domain}` | The cell's public domain (e.g. `alice.pic.ngo`) | -| `{peer.username}` | The peer's username | -| `{peer.service_credentials..}` | A credential value; `` is the service `id`, `` matches a name in `accounts.credentials` | -| `{config.}` | A value from the merged `config_schema` result | - -```json -"peer_config_template": { - "url": "https://notes.{domain}/", - "username": "{peer.username}", - "password": "{peer.service_credentials.notes.password}" -} -``` - -### `accounts` - -Required when `has_accounts` is `true`. - -| Field | Type | Description | -|---|---|---| -| `manager` | string | Set to `"http"` for store services — PIC will call your container's HTTP API for account operations (see section 4). The reference services (`email`, `calendar`, `files`) use internal manager names (`"email_manager"`, `"calendar_manager"`, `"file_manager"`). | -| `credentials` | array of strings | Names of credential fields this service issues per peer. Most services use `["password"]`. The names appear in `peer_config_template` tokens. | - -```json -"accounts": { - "manager": "http", - "credentials": ["password"] -} -``` - -### `compose` - -Unused at the manifest level. Compose configuration is provided via `compose-template.yml` in the service package (see section 3). Set to `null` in the manifest. - -### `backup` - -Required when `has_storage` is `true`. Tells PIC's backup system what to snapshot. - -| Field | Type | Description | -|---|---|---| -| `volumes` | array of objects | Container paths to stream out via `docker exec tar`. Each entry has three string fields: `container` (container name), `path` (absolute path inside the container), and `name` (archive filename stem). | -| `config_paths` | array of strings | Paths relative to the PIC project root on the host that contain service configuration (not user data). Copied directly into the snapshot. | - -Each entry in `volumes` produces an archive at `.tar.gz` inside the snapshot. For example, `"name": "maildata"` produces `maildata.tar.gz`. - -```json -"backup": { - "volumes": [ - {"container": "cell-notes", "path": "/data/notes", "name": "notes_data"} - ], - "config_paths": ["config/notes"] -} -``` - -`ServiceRegistry.get_backup_plan()` aggregates these declarations across all installed services. The backup runner reads from that method rather than from any hardcoded list. - -### `egress` - -Required when `has_egress` is `true`. Declares which outbound network interfaces this service is permitted to use. - -| Field | Type | Description | -|---|---|---| -| `default` | string | The interface selected when the admin has not changed anything. | -| `allowed` | array of strings | The complete set of interfaces the admin can choose from. | - -Valid interface identifiers: `default`, `wireguard_ext`, `openvpn`, `tor`, `sshuttle`, `proxy`. - -```json -"egress": { - "default": "default", - "allowed": ["default", "wireguard_ext", "openvpn", "tor", "sshuttle", "proxy"] -} -``` - -How enforcement works is described in section 6. - -### `storage` - -Informational metadata used by the UI to show storage usage. - -| Field | Type | Description | -|---|---|---| -| `primary_path` | string | The path (relative to project root) that holds the bulk of user data. | -| `quota_mb` | integer or null | Storage quota in megabytes; `null` means no limit. | - -```json -"storage": { - "primary_path": "data/notes", - "quota_mb": null -} -``` - -### Store-only manifest fields - -Store services (where `kind` is `"store"`) have additional required fields that builtins do not use. These are validated by `ServiceStoreManager._validate_manifest()` before installation is permitted. - -| Field | Type | Required | Description | -|---|---|---|---| -| `image` | string | yes | Docker image to pull. Must match the pattern `git.pic.ngo/roof/*`. Images from other registries are rejected. | -| `container_name` | string | yes | The name Docker gives the running container. | -| `volumes` | array | no | Named volumes to mount. Each entry must have `name` (the volume name) and `mount` (the absolute path inside the container). Mounts to `/`, `/etc`, `/var`, `/proc`, `/sys`, `/dev`, `/app`, `/run`, `/boot`, and paths that are a prefix of the PIC project root are forbidden. | -| `env` | array | no | Environment variables to pass. Each entry has `key` and `value`. Values must match `^[A-Za-z0-9._@:/+\-= ]*$`. | -| `iptables_rules` | array | no | FORWARD ACCEPT rules PIC should install in `cell-wireguard`. Each rule must have `type: "ACCEPT"`, `dest_ip: "${SERVICE_IP}"`, an integer `dest_port` (1–65535), and an optional `proto` (`"tcp"` or `"udp"`, default `"tcp"`). The literal string `${SERVICE_IP}` is replaced with the allocated container IP at install time. | -| `caddy_route` | object | no | If the service exposes a web UI, provide `subdomain` (must not be reserved; must match `^[a-z][a-z0-9-]{0,30}$`). PIC inserts the corresponding `reverse_proxy` directive into the Caddyfile. | - ---- - -## 3. Compose template variables - -This section applies only to store services. Builtins define their containers directly in `docker-compose.yml`. - -When you ship a store service, you include a `compose-template.yml` alongside your `manifest.json`. `ServiceComposer.render_template()` substitutes the variables below before writing the per-service `docker-compose.yml`. - -| Variable | Syntax | Value | -|---|---|---| -| `${PIC_CFG_}` | uppercase `config_schema` key | The admin-saved value for that field, or the `default` from the schema if the admin has not set it. For example, `config_schema.port` → `${PIC_CFG_PORT}`. | -| `${PIC_SECRET_}` | any name you choose | An auto-generated random secret produced by `secrets.token_urlsafe(24)` (~32 URL-safe base64 characters). Generated once on first install, then reused unchanged on every reconfigure. Stored per service in `data/service_secrets.json`. | -| `${PIC_DOMAIN}` | literal | Effective domain from `ConfigManager` (e.g. `alice.pic.ngo`). | -| `${PIC_CELL_NAME}` | literal | Cell name from the identity config (e.g. `alice`). | -| `${PIC_SERVICE_ID}` | literal | The `id` field from the service manifest (e.g. `notes`). | - -**Volume mounts**: Because docker compose runs inside the API container but the Docker daemon runs on the host, relative volume paths in compose templates resolve relative to the compose file's directory as seen by the HOST filesystem. To avoid path resolution surprises, prefer **named volumes** for service data (Docker manages them independently). If bind mounts are required, use absolute host paths with `${PIC_PROJECT_DIR}` once that variable is implemented, or document the expected host layout clearly. - -Example `compose-template.yml` for a notes service: - -```yaml -services: - cell-notes: - image: git.pic.ngo/roof/pic-notes:latest - container_name: cell-notes - restart: unless-stopped - environment: - NOTES_PORT: "${PIC_CFG_PORT}" - NOTES_DOMAIN: "${PIC_DOMAIN}" - NOTES_DB_PASS: "${PIC_SECRET_DB_PASSWORD}" - volumes: - - notes-data:/data/notes - networks: - cell-network: - ipv4_address: "${SERVICE_IP}" - -volumes: - notes-data: - -networks: - cell-network: - external: true -``` - -The `SERVICE_IP` variable is the IP PIC allocated from the service pool. It is always set automatically. - ---- - -## 4. Account provisioning interface - -This section covers two related things: the `AccountManager` class that is PIC's central credential dispatcher, and the HTTP API that store services must implement to receive account operations. - -### How AccountManager works - -`AccountManager` (`api/account_manager.py`) is the single entry point for all account operations across every service type. It is instantiated once in `api/managers.py` and holds references to the service managers used by the reference services (`email_manager`, `calendar_manager`, `file_manager`). - -When a peer account is provisioned, `AccountManager`: - -1. Looks up the service in `ServiceRegistry` and reads `accounts.manager` from the manifest. -2. Dispatches to the appropriate internal manager method (for builtins) or to the service's HTTP API endpoint (for store services — not yet implemented; `"http"` manager support is planned). -3. Stores the returned credentials in `data/peer_service_credentials.json` with permissions `0o600`. - -Credentials are stored in plaintext. This is intentional: the peer credentials endpoint needs to return them verbatim for one-time client configuration. The `0o600` permission matches the pattern used for WireGuard keys and `data/service_secrets.json`. - -The credentials file structure is: - -```json -{ - "": { - "": { "password": "..." } - } -} -``` - -Writes use a write-then-rename pattern (`tmp` → final path) with `os.fsync` to avoid partial-write corruption. - -### Manifest `accounts` field - -The `accounts` block in the manifest wires a service into `AccountManager`. - -| Field | Type | Description | -|---|---|---| -| `manager` | string | Which underlying manager handles account operations. For builtins: `"email_manager"`, `"calendar_manager"`, or `"file_manager"`. | -| `credentials` | array of strings | Names of the credential fields this service issues per peer. Most services use `["password"]`. These names are used as token keys in `peer_config_template`. | - -```json -"accounts": { - "manager": "email_manager", - "credentials": ["password"] -} -``` - -The `manager` value must match a key that `AccountManager` was instantiated with. If the manager name has no registered dispatch entry, `provision()` raises `ValueError` immediately. - -### Provision flow - -``` -POST /api/services/catalog//accounts -Content-Type: application/json - -{ "username": "alice", "password": "optional" } -``` - -If `password` is omitted, `AccountManager` generates one with `secrets.token_urlsafe(16)`. The response on HTTP 201 is: - -```json -{ "service_id": "email", "username": "alice", "provisioned": true } -``` - -The password is not echoed in the response. To retrieve stored credentials for a provisioned peer, call `GET /api/services/catalog//accounts//credentials`. - -Internally, `AccountManager.provision(service_id, peer_username, password)`: - -1. Resolves the service and its manager via `_resolve_service()`. -2. Calls the appropriate `_provision_*` method, which delegates to the concrete manager: - - `email_manager` → `create_email_user(username, domain, password)` - - `calendar_manager` → `create_calendar_user(username, password)` - - `file_manager` → `create_user(username, password)` -3. Stores `{"password": ""}` under `[service_id][peer_username]` in the credentials file. -4. Returns the credential dict to the caller. - -If the underlying manager call returns `False`, `provision()` raises `RuntimeError`. The route handler maps this to HTTP 500. - -For email, the domain is read from the service's merged config (`svc['config']['domain']`). If that key is absent, provisioning raises `ValueError` before calling the manager. - -### Deprovision flow - -``` -DELETE /api/services/catalog//accounts/ -``` - -`AccountManager.deprovision(service_id, peer_username)`: - -1. Calls the appropriate `_deprovision_*` method on the underlying manager. -2. Removes the peer's entry from the credentials file. If that leaves the service block empty, the service block itself is removed. -3. Returns `True` if the underlying call succeeded. - -The route returns HTTP 200 with `{"message": "..."}` on success, or HTTP 400 if the service does not exist or does not support accounts. - -**Peer deletion** calls `AccountManager.deprovision_peer(peer_username)`, which iterates over every service the peer is provisioned on and calls `deprovision()` for each. Failures on individual services are logged and skipped rather than aborting the deletion — the method returns `{service_id: bool}` for every service attempted. - -### PIC admin API endpoints for account management - -These endpoints are in `api/routes/services.py` and `api/routes/peers.py`. - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/services/catalog//accounts` | Return `{"service_id": "...", "accounts": ["alice", "bob"]}` — reads directly from the credentials file. | -| `POST` | `/api/services/catalog//accounts` | Provision a peer account. Body: `{"username": "...", "password": "..."}` (password optional). Returns HTTP 201 with `{"service_id", "username", "provisioned": true}`. | -| `DELETE` | `/api/services/catalog//accounts/` | Deprovision the peer's account. Returns HTTP 200 on success, HTTP 400 if the service or username is unknown. | -| `GET` | `/api/services/catalog//accounts//credentials` | Return stored credentials for one peer+service pair. Returns HTTP 404 if the peer is not provisioned on that service. Response: `{"service_id", "username", "password"}`. | -| `GET` | `/api/peers//service-credentials` | Return filled `peer_config_template` values for all services the peer is provisioned on (see below). | - -**Admin UI:** The Email, Calendar, and Files service pages in the admin dashboard each have an **Accounts** tab. From there, admins can provision and deprovision peer accounts, and reveal stored credentials for a provisioned peer. This tab calls the same API endpoints listed above. - -### How `peer_config_template` connects to stored credentials - -`GET /api/peers//service-credentials` is the endpoint a peer device calls during first-time setup to configure email, CalDAV, and file sync clients. - -The route: - -1. Calls `AccountManager.get_all_credentials(peer_name)` → `{service_id: {field: value}}`. -2. For each service, calls `ServiceRegistry.get_peer_service_info(service_id, peer_name, domain, cred)`. -3. `get_peer_service_info` iterates over `peer_config_template` and replaces tokens: - - `{domain}` → effective cell domain - - `{peer.username}` → URL-percent-encoded peer username (safe='') - - `{peer.service_credentials..}` → the value from stored credentials - - `{config.}` → value from the service's merged config schema -4. Returns the filled template dict as the value for that service in the response. - -Response shape: - -```json -{ - "peer": "alice", - "services": { - "email": { - "imap_host": "mail.alice.pic.ngo", - "username": "alice@alice.pic.ngo", - "password": "" - }, - "files": { - "url": "https://files.alice.pic.ngo/dav/alice/", - "username": "alice", - "password": "" - } - } -} -``` - -If a service has no `peer_config_template` in its manifest, `get_peer_service_info` returns `None` and the raw credential dict is used as the fallback. - -### Container lifecycle routes - -The following PIC API endpoints are available for all services (builtins and store services). These are called by the web UI and can be called directly from the PIC admin API. - -| Method | Path | Description | -|---|---|---| -| `GET` | `/api/services/catalog//status` | Return container status. Builtins query the main compose stack; store services query their own compose project. Response includes a `containers` array with one entry per container. | -| `POST` | `/api/services/catalog//restart` | Restart the service containers. Builtins restart via the main compose stack; store services restart via their own compose project. | -| `POST` | `/api/services/catalog//reconfigure` | Re-render the compose file from the template and re-apply with `up -d` (rolling update). Store services only — builtins are reconfigured through their own settings routes. The request body must include a `compose_template` field containing the new template content. | - -### Store service HTTP API - -When `accounts.manager` is `"http"`, PIC will call your container's HTTP API for account operations. **HTTP dispatch is not yet wired up in `AccountManager`** — the current dispatch table covers only `email_manager`, `calendar_manager`, and `file_manager` (used by the reference services). Implement this interface now so your service is ready when HTTP dispatch ships. - -The base path is `/service-api/accounts` on your container's internal address. There is no authentication on this API — it is reachable only from within the `cell-network` Docker network. - -**Create account** - -``` -POST /service-api/accounts -Content-Type: application/json - -{ "username": "alice", "password": "auto-generated-by-pic" } -``` - -PIC generates the password and passes it to your service. Return HTTP 200 with `{"ok": true}` on success. Return HTTP 400 or 409 with `{"ok": false, "error": "..."}` for expected errors (duplicate username, invalid input). Return HTTP 500 for unexpected internal errors. - -**Delete account** - -``` -DELETE /service-api/accounts/{username} -``` - -Return HTTP 200 with `{"ok": true}` on success. Return HTTP 404 with `{"ok": false, "error": "not found"}` if the account does not exist. - -**List accounts** - -``` -GET /service-api/accounts -``` - -Return `{"accounts": ["alice", "bob"]}` — an array of all provisioned usernames. - ---- - -## 5. Backup integration - -Declare `has_storage: true` in `capabilities` and fill in the `backup` block. PIC's `ServiceRegistry.get_backup_plan()` returns the combined backup declarations for all installed services. The backup runner reads from that method. - -### Why docker exec instead of bind mounts - -The API container only has access to `data/api/` on the host filesystem. Service data (mailboxes, calendar collections, file trees) lives in other containers' volumes. Rather than mount every service volume into the API container — which would require compose changes per service — PIC streams data using `docker exec tar czf - `. This works for any container on the Docker host regardless of how its volumes are configured. - -### `volumes` entries - -Each object in the `volumes` array describes one directory to capture: - -| Field | Description | -|---|---| -| `container` | Name of the running container to exec into (e.g. `"cell-notes"`). | -| `path` | Absolute path inside that container to archive (e.g. `"/data/notes"`). | -| `name` | Archive filename stem. PIC saves the archive as `.tar.gz` under `service_data//` in the backup directory. | - -A service with multiple containers or multiple data directories lists one entry per directory. - -**Security note:** The backup commands use `docker exec -- tar -C -czf - .` (note the `--` separator before the container name) to prevent option injection. The container name is also validated against `^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,63}$` before the command is run. - -### `config_paths` - -Paths in `config_paths` are relative to the PIC project root on the host and are copied directly into the snapshot (no docker exec). Use this for configuration files the service reads at startup, not for user data. - -### Full example - -```json -"backup": { - "volumes": [ - {"container": "cell-notes", "path": "/data/notes", "name": "notes_data"} - ], - "config_paths": ["config/notes"] -} -``` - -This produces one archive `notes_data.tar.gz` (streamed from the `cell-notes` container) plus a direct copy of `config/notes/` from the host. - -### Restore - -PIC restores each volume entry by piping the archive back via `docker exec -i -- tar -C -xzf -`. The `-C ` flag bounds extraction to the declared volume path — the same path used during backup. Archive entries are relative paths (the backup uses `tar -C -czf - .`), so files land in exactly the location declared in the manifest `volumes` entry. The target container must be running at restore time. - ---- - -## 6. Egress routing - -When `has_egress` is `true`, the cell admin can assign a specific outbound interface to your service. PIC enforces the selection using `fwmark` rules and policy routing in the `cell-wireguard` container via the `ConnectivityManager`. - -The valid values for `egress.allowed` and what they mean: - -| Value | Path | -|---|---| -| `default` | Default route through the cell's WAN interface (no VPN). | -| `wireguard_ext` | Traffic leaves through `wg_ext0` (fwmark `0x10`, table 110). Requires the `wireguard-ext` store service. | -| `openvpn` | Traffic leaves through `tun0` (fwmark `0x20`, table 120). Requires the `openvpn-client` store service. | -| `tor` | Traffic is redirected to the Tor transparent proxy on port 9040 (fwmark `0x30`, table 130). Requires the `tor` store service. | -| `sshuttle` | Traffic is redirected to the sshuttle transparent proxy on port 12300 (fwmark `0x40`, table 140). Requires the `sshuttle` store service. | -| `proxy` | Traffic is redirected to the redsocks transparent proxy on port 12345 (fwmark `0x50`, table 150). Requires the `proxy` store service. | - -List only the interfaces that make sense for your service in `allowed`. The `default` value is used when the admin has not changed anything. Always include `default` in `allowed` so the admin has a way to use the normal path. - -The egress field in the manifest tells PIC what options to present in the UI. Actual enforcement requires the cell to have the corresponding exit type configured (an OpenVPN config uploaded, a WireGuard external config active, etc.). If the chosen exit is not active, packets will be dropped by the kill-switch FORWARD rule in `cell-wireguard`. - ---- - -## 7. Quick-start example - -This section walks through a minimal working example: a static website served from Nginx with no accounts, no backup, and no egress policy. - -### `manifest.json` - -```json -{ - "schema_version": 3, - "id": "homepage", - "name": "Homepage", - "description": "A static homepage served from your cell", - "version": "1.0.0", - "author": "acme", - "kind": "store", - "min_pic_version": "1.0", - - "capabilities": { - "has_subdomain": true, - "has_accounts": false, - "has_admin_config": false, - "has_storage": false, - "has_egress": false, - "has_api_hooks": false - }, - - "subdomain": "home", - "extra_subdomains": [], - "backend": "cell-homepage:80", - - "containers": ["cell-homepage"], - - "image": "git.pic.ngo/roof/pic-homepage:latest", - "container_name": "cell-homepage", - - "volumes": [ - { "name": "homepage-html", "mount": "/usr/share/nginx/html" } - ], - - "env": [], - - "iptables_rules": [ - { - "type": "ACCEPT", - "dest_ip": "${SERVICE_IP}", - "dest_port": 80, - "proto": "tcp" - } - ], - - "caddy_route": { - "subdomain": "home" - }, - - "compose": null -} -``` - -### What PIC does on install - -1. Downloads this manifest from the store index. -2. Validates every field (image allowlist, volume safety, reserved subdomains, iptables rule format). -3. Allocates a static IP from the service pool (`172.20.0.20`–`172.20.0.254`). -4. Writes a Docker Compose override file that starts `cell-homepage` with the allocated IP on `cell-network`. -5. Runs `docker compose up -d cell-homepage`. -6. Applies the `iptables_rules` in `cell-wireguard` so peers can reach the container. -7. Regenerates the Caddyfile so `home.` proxies to `cell-homepage:80`. - -The result is that any WireGuard peer can reach `https://home.alice.pic.ngo/` immediately after installation. - ---- - -## 8. Reference implementations - -The `email`, `calendar`, and `files` services in `pic-services/services/` are the canonical examples of a complete store service. They demonstrate the full feature set: - -| Service | Notable features demonstrated | -|---|---| -| `email` | `has_accounts`, `has_egress`, multi-container (`cell-mail` + `cell-rainloop`), `extra_backends`, custom image baking defaults via Dockerfile | -| `calendar` | `has_accounts`, CalDAV `peer_config_template`, htpasswd account provisioning | -| `files` | `has_accounts`, `has_storage`, WebDAV + Filegator `extra_backends`, `backup.volumes` with multiple entries | - -When in doubt about how to structure your manifest or compose template, use these as the reference. - ---- - -## 9. Submitting to the store - -### Package format - -A store service package is a ZIP archive containing: - -``` -homepage-1.0.0.zip -├── manifest.json (required) -├── compose-template.yml (recommended for multi-container services) -└── install.sh (optional post-install script) -``` - -`install.sh` is executed on the cell host after the container starts. Keep it minimal — initialise data structures, create default config files. Do not use it to install system packages or modify files outside the PIC project root. - -### Store index entry - -The store index at `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json` is a JSON array. Each entry looks like: - -```json -{ - "id": "homepage", - "name": "Homepage", - "description": "A static homepage served from your cell", - "version": "1.0.0", - "author": "acme" -} -``` - -PIC fetches the full manifest from `https://git.pic.ngo/roof/pic-services/raw/branch/main/services/{id}/manifest.json` when the admin clicks install. - -### Submission process - -1. Fork `https://git.pic.ngo/roof/pic-services`. -2. Create a directory `services//` and add your `manifest.json`. -3. Open a pull request against `main`. - -The review checks the following before merging: - -**Security** -- Image hosted on `git.pic.ngo/roof/*`. No external registries. -- No volume mounts to system paths or to the PIC project root. -- `iptables_rules` only declare `ACCEPT` rules (no DROP, no REJECT, no chain redirects). -- `env` values contain only alphanumeric characters and a small set of safe punctuation. -- `install.sh` does not call `apt`, `yum`, `curl | bash`, or modify files outside the project. - -**Correctness** -- `subdomain` does not collide with the reserved list or with any existing store service. -- `backend` points to the declared `container_name`. -- If `has_accounts: true`, the container responds correctly on all three `/service-api/accounts` endpoints. -- If `has_storage: true`, every `volumes` entry names a container that is running and a path that exists inside it. - -**Quality** -- `description` is one sentence, no marketing language. -- `version` is a valid semver string. -- `config_schema` labels are in plain English, sentence case. - -### Versioning - -Increment `version` in `manifest.json` with every change you submit. PIC does not auto-update installed services; the admin manually runs an update. When an update is available, the UI shows the version mismatch between the installed record and the store index. - ---- - -## Appendix: manifest field quick reference - -| Field | Required | Notes | -|---|---|---| -| `schema_version` | yes | Must be `3` | -| `id` | yes | | -| `name` | yes | | -| `description` | yes | | -| `version` | yes | | -| `author` | yes | | -| `kind` | yes | Must be `"store"` | -| `min_pic_version` | no | | -| `capabilities.*` | yes | All six flags must be present | -| `subdomain` | if `has_subdomain` | | -| `extra_subdomains` | no | | -| `backend` | if `has_subdomain` | | -| `extra_backends` | no | | -| `containers` | no | Informational | -| `config_schema` | if `has_admin_config` | | -| `peer_config_template` | if `has_accounts` | | -| `accounts` | if `has_accounts` | | -| `compose` | no | Always `null` — compose config goes in `compose-template.yml` | -| `backup` | if `has_storage` | | -| `egress` | if `has_egress` | | -| `storage` | if `has_storage` | | -| `image` | yes | Must match `git.pic.ngo/roof/*` | -| `container_name` | yes | Must match `^cell-[a-z0-9][a-z0-9-]{0,30}$` | -| `volumes` | no | | -| `env` | no | | -| `iptables_rules` | no | | -| `caddy_route` | no | | diff --git a/webui/README.md b/webui/README.md deleted file mode 100644 index 3ad5b56..0000000 --- a/webui/README.md +++ /dev/null @@ -1,138 +0,0 @@ -# Personal Internet Cell - Web UI - -A modern React-based web interface for managing your Personal Internet Cell. - -## Features - -- **Dashboard**: Overview of cell status and services -- **Peer Management**: Add, remove, and configure WireGuard peers -- **Network Services**: DNS, DHCP, and NTP management -- **WireGuard**: VPN configuration and status -- **Email Services**: Postfix and Dovecot management -- **Calendar Services**: Radicale CalDAV/CardDAV management -- **File Storage**: WebDAV file storage management -- **Routing**: Advanced VPN gateway and routing configuration -- **Logs**: System logs and monitoring -- **Settings**: Cell configuration and security settings - -## Tech Stack - -- **React 19**: Modern React with hooks -- **Vite**: Fast build tool and dev server -- **Tailwind CSS**: Utility-first CSS framework -- **Lucide React**: Beautiful icons -- **React Router**: Client-side routing -- **Axios**: HTTP client for API communication - -## Development - -### Prerequisites - -- Node.js 18+ and npm -- Personal Internet Cell backend running on port 3000 - -### Setup - -1. Install dependencies: - ```bash - bun install - ``` - -2. Start the development server: - ```bash - npm run dev - ``` - -3. Open your browser to `http://localhost:5173` - -### Development Features - -- **Hot Reload**: Changes reflect immediately -- **API Proxy**: Requests to `/api/*` are proxied to `http://localhost:3000` -- **TypeScript Support**: Full TypeScript support available -- **ESLint**: Code linting and formatting - -## Building for Production - -### Build - -```bash -npm run build -``` - -This creates a `dist/` directory with optimized production files. - -### Preview - -```bash -npm run preview -``` - -This serves the built files locally for testing. - -## API Integration - -The Web UI communicates with the Personal Internet Cell backend API: - -- **Base URL**: `http://localhost:3000` (development) -- **Health Check**: `/health` -- **API Endpoints**: `/api/*` - -### Environment Variables - -Create a `.env` file to customize the API URL: - -```env -VITE_API_URL=http://localhost:3000 -``` - -## Project Structure - -``` -src/ -├── components/ # Reusable UI components -│ └── Sidebar.jsx # Navigation sidebar -├── pages/ # Page components -│ ├── Dashboard.jsx # Main dashboard -│ ├── Peers.jsx # Peer management -│ ├── NetworkServices.jsx -│ ├── WireGuard.jsx # VPN configuration -│ ├── Email.jsx # Email services -│ ├── Calendar.jsx # Calendar services -│ ├── Files.jsx # File storage -│ ├── Routing.jsx # Routing configuration -│ ├── Logs.jsx # System logs -│ └── Settings.jsx # Cell settings -├── services/ # API services -│ └── api.js # API client and endpoints -├── App.jsx # Main app component -├── main.jsx # App entry point -└── index.css # Global styles -``` - -## Styling - -The Web UI uses Tailwind CSS with custom components: - -- **Cards**: `.card` for content containers -- **Buttons**: `.btn`, `.btn-primary`, `.btn-secondary`, etc. -- **Inputs**: `.input` for form fields -- **Status Indicators**: `.status-indicator`, `.status-online`, etc. - -## Browser Support - -- Chrome 90+ -- Firefox 88+ -- Safari 14+ -- Edge 90+ - -## Contributing - -1. Follow the existing code style -2. Use TypeScript for new components -3. Add tests for new features -4. Update documentation as needed - -## License - -Part of the Personal Internet Cell project.