roof/pic
1
0
Fork 0
Code Issues Pull Requests Actions Wiki Activity

docs: bring all docs current with this session's changes
Unit Tests / test (push) Successful in 12m12s
Details

Browse Source 
Update README, QUICKSTART, wiki, service-developer-guide, and CLAUDE.md for:
optional store services (email/calendar/files), sshuttle+proxy egress exits,
provider-aware Network Services/DNS overview, DHCP/dnsmasq removal, split-horizon
VPN DNS, container hardening (slim images, unprivileged WireGuard, webui port 8080,
pinned ntp/coredns), installer changes (host NTP, PIC_DEBUG, clean output, systemd),
and the backup overhaul (full secrets coverage + optional passphrase encryption).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dmitrii Iurco
2026-06-10 15:56:03 -04:00
parent 82a0c0e9bd
commit 8a9f4f50c6
5 changed files with 196 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Personal Internet Cell – Project Wiki.md
+133 -28
View File
@@ -2,7 +2,7 @@
## Overview
Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, DHCP, NTP, WireGuard VPN, email, calendar/contacts, file storage, HTTPS reverse proxy, a certificate authority, and optional services — all managed from a single REST API and React web UI.
Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, NTP, WireGuard VPN, HTTPS reverse proxy, a certificate authority, and optional services (email, calendar/contacts, file storage, connectivity exits, and more) — all managed from a single REST API and React web UI.
The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers.
@@ -16,13 +16,15 @@ The goal is to give a person full ownership of their core internet services on t
4. [Authentication](#authentication)
5. [API Reference](#api-reference)
6. [DDNS](#ddns)
7. [Services UI](#services-ui)
8. [Service Store (Add-ons)](#service-store-add-ons)
9. [Cell-to-Cell Networking](#cell-to-cell-networking)
10. [Extended Connectivity](#extended-connectivity)
11. [Security Model](#security-model)
12. [Testing](#testing)
13. [Development](#development)
7. [Network Services and DNS](#network-services-and-dns)
8. [Services UI](#services-ui)
9. [Service Store (Add-ons)](#service-store-add-ons)
10. [Backup and Restore](#backup-and-restore)
11. [Cell-to-Cell Networking](#cell-to-cell-networking)
12. [Extended Connectivity](#extended-connectivity)
13. [Security Model](#security-model)
14. [Testing](#testing)
15. [Development](#development)
---
@@ -31,20 +33,20 @@ The goal is to give a person full ownership of their core internet services on t
```
Browser / WireGuard peer
└── Caddy (:80/:443) reverse proxy, TLS termination
└── React SPA (:8081) Vite + Tailwind (Nginx in container)
└── React SPA (:8081→8080) Vite + Tailwind (Nginx in container)
└── Flask API (:3000) REST API, bound to 127.0.0.1
├── NetworkManager CoreDNS, dnsmasq, chrony
├── NetworkManager CoreDNS, chrony (split-horizon DNS)
├── WireGuardManager WireGuard VPN peer lifecycle
├── PeerRegistry peer registration and trust
├── EmailManager Postfix + Dovecot
├── CalendarManager Radicale CalDAV/CardDAV
├── FileManager WebDAV + Filegator
├── EmailManager Postfix + Dovecot (optional service)
├── CalendarManager Radicale CalDAV/CardDAV (optional service)
├── FileManager WebDAV + Filegator (optional service)
├── RoutingManager iptables NAT and routing
├── FirewallManager iptables firewall rules
├── VaultManager internal CA, cert lifecycle, Age encryption
├── VaultManager internal CA, cert lifecycle, Fernet encryption
├── ContainerManager Docker SDK
├── CellLinkManager cell-to-cell WireGuard links
├── ConnectivityManager exit routing (WG ext, OpenVPN, Tor)
├── ConnectivityManager exit routing (WG ext, OpenVPN, Tor, sshuttle, proxy)
├── DDNSManager dynamic DNS heartbeat
├── ServiceStoreManager optional service install/remove
├── CaddyManager Caddyfile generation and reload
@@ -52,7 +54,9 @@ Browser / WireGuard peer
└── SetupManager first-run wizard state
```
The 7 core containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default). Static IPs per container are defined in `docker-compose.yml`. Installed optional services join the same network via their own compose projects, managed by `ServiceComposer`.
Six core containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default): `cell-caddy`, `cell-dns`, `cell-ntp`, `cell-wireguard`, `cell-api`, `cell-webui`. Static IPs per container are defined in `docker-compose.yml`. Installed optional services join the same network via their own compose projects, managed by `ServiceComposer`.
The WireGuard container runs unprivileged — it uses `NET_ADMIN` only and requires the WireGuard module on the host kernel (Linux 5.6+ or a loadable module). The `cell-api` and `cell-webui` images are slim builds. The CoreDNS image is pinned to a specific digest.
Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.
@@ -73,7 +77,7 @@ The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers
| Manager | Responsibilities |
|---|---|
| `NetworkManager` | CoreDNS zone files, dnsmasq DHCP config and lease monitoring, chrony NTP |
| `NetworkManager` | CoreDNS zone files and split-horizon DNS, chrony NTP |
| `WireGuardManager` | Key generation, `wg0.conf` generation, peer add/remove, route sync |
| `PeerRegistry` | Peer registration, trust tracking, peer statistics |
| `EmailManager` | docker-mailserver accounts, mailbox config, alias management |
@@ -84,7 +88,7 @@ The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers
| `VaultManager` | Internal CA (self-signed root), TLS cert issue/revoke, Age public key |
| `ContainerManager` | Docker container/image/volume management via SDK |
| `CellLinkManager` | Site-to-site WireGuard links to other PIC cells, peer-sync protocol |
| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, or Tor |
| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, Tor, sshuttle, or proxy (redsocks) |
| `DDNSManager` | Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns) |
| `ServiceStoreManager` | Fetch manifest index, install/remove optional services |
| `CaddyManager` | Caddyfile generation, reload-on-change |
@@ -104,6 +108,16 @@ The wizard collects:
- **Services to install** — optional services (email, calendar, files) to install after setup; each starts a background install via `ServiceStoreManager`
- **Admin password** — minimum 12 characters
Domain modes and their TLS behavior:
| Mode | Certificate |
|---|---|
| `pic_ngo` | Wildcard Let's Encrypt cert via DNS-01 (requires accurate host clock for ACME + DDNS token) |
| `cloudflare` | Wildcard Let's Encrypt cert via Cloudflare DNS-01 |
| `duckdns` | Let's Encrypt via DuckDNS DNS-01 |
| `http01` | Let's Encrypt per-subdomain cert via HTTP-01 (no wildcard; port 80 must be reachable) |
| `lan` | Internal CA only; no internet required |
On completion:
1. Admin account is created in `data/auth_users.json`
2. Cell identity is written to `config/api/cell_config.json`
@@ -175,9 +189,9 @@ Auth enforcement is active once any user exists in the store. If the store is em
| POST | `/api/setup/step` | Submit current step |
| POST | `/api/setup/complete` | Finalize setup |
### Network Services (`/api/dns/`, `/api/dhcp/`, `/api/ntp/`, `/api/network/`)
### Network Services (`/api/dns/`, `/api/ntp/`, `/api/network/`)
DNS records, DHCP leases and reservations, NTP status, network connectivity test.
DNS overview (effective domain, public records, internal records, per-mode actions), NTP status, network connectivity test.
### WireGuard (`/api/wireguard/`, `/api/peers/`)
@@ -213,7 +227,7 @@ List connected cells, add/remove cell links, peer-sync.
### Connectivity (`/api/connectivity/`)
List exit nodes, configure WireGuard external / OpenVPN / Tor exits, assign per-peer exit policy.
List exit nodes, configure WireGuard external / OpenVPN / Tor / sshuttle / proxy exits, assign per-peer exit policy, assign per-service egress.
### Service Store (`/api/store/`)
@@ -249,6 +263,90 @@ Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`.
---
## Network Services and DNS
### Split-horizon DNS
PIC operates a split-horizon DNS configuration for the cell domain.
- **Outside the VPN** — the cell domain (e.g. `myhome.pic.ngo`) resolves to the public IP via the DDNS provider.
- **Inside the VPN** — CoreDNS answers the same cell domain with the WireGuard internal IP of `cell-caddy`. Traffic therefore flows through the WireGuard tunnel and Caddy serves it on both the public and WireGuard interface.
`NetworkManager.update_split_horizon_zone()` writes a zone file for the effective domain and reloads CoreDNS via SIGUSR1 whenever the cell name or domain mode changes.
### Network Services page
The **Network Services** page (`/network`) shows a provider-aware DNS overview:
- Current domain mode label and effective domain
- Public DNS records (A records registered with the DDNS provider)
- Service subdomains (shown for `pic_ngo` and `duckdns` modes)
- Internal records served by CoreDNS on the WireGuard network
- Per-mode action buttons (e.g. force-refresh DDNS, reload CoreDNS)
- NTP status (running/stopped, current time source)
DHCP has been removed from PIC. There is no `cell-dhcp` container and no DHCP configuration in the UI. The `cell-dns` container runs CoreDNS only.
### NTP
The `cell-ntp` container runs chrony in a pinned Alpine image. It provides NTP to WireGuard peers that configure the cell as their NTP server. The installer also enables host-level chrony to keep the host clock accurate for ACME certificate issuance and DDNS TOTP tokens.
---
## Backup and Restore
### What `make backup` captures
`make backup` creates `backups/cell-backup-<timestamp>.tar.gz` containing:
- `config/` — all service configuration including `cell_config.json`
- `data/` excluding logs (`data/logs/`) and internal config-backup snapshots (`data/api/config_backups/`)
- `docker-compose.yml` and `Makefile`
The archive is written mode `0600`. It contains key material (WireGuard keys, internal CA, vault fernet.key, admin credentials, DDNS token, cell links, Caddy ACME certs). Store it securely.
Data volumes of installed store services (email mailboxes, calendar collections, file trees) are **not** included in `make backup`. They are captured by the API-driven backup described below.
### API-driven backup (`POST /api/config/backup`)
The API backup captures everything `make backup` does, plus:
- `.env`
- The Caddyfile and Corefile (runtime-generated)
- DNS zone files
- WireGuard key material and live peer configs
- Vault directory (CA, certificates, fernet.key, trust store)
- Per-service connectivity configs (sshuttle keys, redsocks config, OpenVPN configs, WireGuard external config)
- Auth users, Flask secret key, DDNS token, cell links, peer service credentials
- Caddy issued ACME certificates and ACME state
- Live service data volumes (streamed via `docker exec tar`)
**Passphrase encryption**: pass `{"passphrase": "..."}` in the request body to encrypt the archive. The encrypted file is named `<backup_id>.tar.gz.age` and uses Fernet with an scrypt-derived key. The plaintext staging directory is removed immediately after encryption. Supply the same passphrase when calling `POST /api/config/restore/<backup_id>`.
Both backup and encrypted archive files are written mode `0600`.
### Restore
**From `make backup`:**
```bash
tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz
make restart
```
**From API backup:**
Call `POST /api/config/restore/<backup_id>` (with `{"passphrase": "..."}` for encrypted archives). The restore process:
1. Restores the vault first (fernet.key must be present before any encrypted secrets are read)
2. Restores identity, `.env`, WireGuard key material, cell links
3. Restores Caddy ACME certs, Caddyfile, Corefile, DNS zones
4. Restores connectivity configs, auth users, DDNS token
5. Restores service user account files
6. Reloads `cell_config.json` into memory
7. Restores live service data volumes (if service containers are running)
8. Calls `_reapply_runtime_state()` — regenerates Caddyfile and Corefile from the restored config and re-applies routing rules
After an API restore, run `make restart` to ensure all containers pick up the restored configuration.
---
## Services UI
### Navigation
@@ -298,7 +396,7 @@ The config form and users list are not shown to peers.
### Settings page
The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS/DHCP/NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.
The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS, NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.
### Relevant API endpoints
@@ -345,14 +443,21 @@ Access control is per-service (calendar, files, mail, WebDAV) and enforced at th
## Extended Connectivity
`ConnectivityManager` provides per-peer exit routing: traffic from a specific WireGuard peer can be routed through an alternate exit instead of the cell's default gateway.
`ConnectivityManager` provides per-peer and per-service exit routing: traffic from a specific WireGuard peer (or a specific installed service) can be routed through an alternate exit instead of the cell's default gateway.
Supported exits:
- **WireGuard external** — another WireGuard endpoint (e.g. a VPS)
- **OpenVPN** — OpenVPN client running in a container
- **Tor** — Tor SOCKS proxy with transparent redirection
All exit types are optional store services installed from the Services catalog. Each exit type corresponds to a store service ID:
Routing uses fwmark and `ip rule` / `ip route` in separate routing tables. Configuration is via `PUT /api/connectivity/peers/<peer_name>/exit`.
| Exit type | Store service | Mechanism |
|---|---|---|
| `wireguard_ext` | `wireguard-ext` | WireGuard client tunnel to an external server; iface `wg_ext0`; fwmark `0x10`, table 110 |
| `openvpn` | `openvpn-client` | OpenVPN client tunnel; iface `tun0`; fwmark `0x20`, table 120 |
| `tor` | `tor` | Transparent proxy → Tor SOCKS on port 9040; fwmark `0x30`, table 130 |
| `sshuttle` | `sshuttle` | SSH tunnel via sshuttle to any SSH server; transparent proxy on port 12300; fwmark `0x40`, table 140 |
| `proxy` | `proxy` | HTTP or SOCKS5 upstream proxy via redsocks transparent redirection on port 12345; fwmark `0x50`, table 150 |
Routing uses fwmark and `ip rule` / `ip route` in separate routing tables inside the `cell-wireguard` container. A kill-switch FORWARD rule drops traffic for configured exits if the exit container is not active.
Configuration is via `PUT /api/connectivity/peers/<peer_name>/exit`. Service-level egress is declared in the service manifest's `egress` block and configured via the Connectivity page in the UI.
---