pic/Personal Internet Cell – Project Wiki.md

# Personal Internet Cell – Project Wiki

## Overview

Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, DHCP, NTP, WireGuard VPN, email, calendar/contacts, file storage, HTTPS reverse proxy, a certificate authority, and optional services — all managed from a single REST API and React web UI.

The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers.

---

## Table of Contents

1. [Architecture](#architecture)
2. [Service Managers](#service-managers)
3. [First-Run Wizard](#first-run-wizard)
4. [Authentication](#authentication)
5. [API Reference](#api-reference)
6. [DDNS](#ddns)
7. [Service Store](#service-store)
8. [Cell-to-Cell Networking](#cell-to-cell-networking)
9. [Extended Connectivity](#extended-connectivity)
10. [Security Model](#security-model)
11. [Testing](#testing)
12. [Development](#development)

---

## Architecture

```
Browser / WireGuard peer
  └── Caddy (:80/:443)             reverse proxy, TLS termination
        └── React SPA (:8081)      Vite + Tailwind (Nginx in container)
        └── Flask API (:3000)      REST API, bound to 127.0.0.1
              ├── NetworkManager       CoreDNS, dnsmasq, chrony
              ├── WireGuardManager     WireGuard VPN peer lifecycle
              ├── PeerRegistry         peer registration and trust
              ├── EmailManager         Postfix + Dovecot
              ├── CalendarManager      Radicale CalDAV/CardDAV
              ├── FileManager          WebDAV + Filegator
              ├── RoutingManager       iptables NAT and routing
              ├── FirewallManager      iptables firewall rules
              ├── VaultManager         internal CA, cert lifecycle, Age encryption
              ├── ContainerManager     Docker SDK
              ├── CellLinkManager      cell-to-cell WireGuard links
              ├── ConnectivityManager  exit routing (WG ext, OpenVPN, Tor)
              ├── DDNSManager          dynamic DNS heartbeat
              ├── ServiceStoreManager  optional service install/remove
              ├── CaddyManager         Caddyfile generation and reload
              ├── AuthManager          session auth, RBAC
              └── SetupManager         first-run wizard state
```

All 12 service containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default). Static IPs per container are defined in `docker-compose.yml`.

Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.

---

## Service Managers

All managers inherit `BaseServiceManager` (`api/base_service_manager.py`), which provides:
- `get_status()` — current running state
- `get_config()` / `update_config()` — config read/write
- `test_connectivity()` — reachability check
- `get_logs()` — last N lines from the service log
- `restart_service()` — container restart via Docker SDK

The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers (e.g., `CONFIG_CHANGED`, `SERVICE_STARTED`). Dependencies are declared in the bus (wireguard depends on network; email depends on network and vault).

### Manager summary

| Manager | Responsibilities |
|---|---|
| `NetworkManager` | CoreDNS zone files, dnsmasq DHCP config and lease monitoring, chrony NTP |
| `WireGuardManager` | Key generation, `wg0.conf` generation, peer add/remove, route sync |
| `PeerRegistry` | Peer registration, trust tracking, peer statistics |
| `EmailManager` | docker-mailserver accounts, mailbox config, alias management |
| `CalendarManager` | Radicale user/calendar/contacts lifecycle |
| `FileManager` | WebDAV user directories, Filegator access |
| `RoutingManager` | NAT rules, per-peer routing policy, fwmark-based exit routing |
| `FirewallManager` | iptables INPUT/FORWARD/OUTPUT rule management |
| `VaultManager` | Internal CA (self-signed root), TLS cert issue/revoke, Age public key |
| `ContainerManager` | Docker container/image/volume management via SDK |
| `CellLinkManager` | Site-to-site WireGuard links to other PIC cells, peer-sync protocol |
| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, or Tor |
| `DDNSManager` | Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns) |
| `ServiceStoreManager` | Fetch manifest index, install/remove optional services |
| `CaddyManager` | Caddyfile generation, reload-on-change |
| `AuthManager` | bcrypt password store, session management, admin/peer RBAC |
| `SetupManager` | First-run wizard state, setup-complete flag |

---

## First-Run Wizard

On first start, `SetupManager.is_setup_complete()` returns `False`. The `enforce_setup` before-request hook returns HTTP 428 for all `/api/*` requests except `/api/setup/*` and `/health`, redirecting clients to `/setup`.

The wizard collects:
- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`)
- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01`
- **Timezone**
- **Initial services to enable**
- **Admin password** — minimum 12 characters

On completion:
1. Admin account is created in `data/auth_users.json`
2. Cell identity is written to `config/api/cell_config.json`
3. Caddy config is generated
4. If domain mode is `pic_ngo`, the cell registers `<name>.pic.ngo` with the DDNS service

Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`.

---

## Authentication

`AuthManager` stores bcrypt-hashed credentials in `data/auth_users.json`. Two roles:

| Role | Access |
|---|---|
| `admin` | All `/api/*` endpoints except `/api/peer/*` |
| `peer` | `/api/peer/*` only (peer dashboard, key exchange) |

Session auth flow:
- `POST /api/auth/login` — creates a Flask session
- `GET /api/auth/me` — current session info
- `POST /api/auth/logout` — clears session
- `POST /api/auth/change-password` — change own password
- `POST /api/auth/admin/reset-password` — admin resets another user's password

CSRF protection: all `POST`, `PUT`, `DELETE`, `PATCH` on `/api/*` (except `/api/auth/*` and `/api/setup/*`) require the `X-CSRF-Token` header matching the session token, obtained via `GET /api/auth/csrf-token`.

Cell-to-cell peer-sync endpoints (`/api/cells/peer-sync/*`) use source-IP + WireGuard public key auth, not session cookies.

Auth enforcement is active once any user exists in the store. If the store is empty (fresh install before wizard), all requests bypass auth — `enforce_setup` already blocks them with 428.

---

## API Reference

**Base URL:** `http://localhost:3000`  
**Auth:** session cookie (`X-CSRF-Token` header required for mutations)

### Core

| Method | Path | Description |
|---|---|---|
| GET | `/health` | Health check (always public) |
| GET | `/api/status` | All-service status summary |
| GET | `/api/config` | Full cell config |
| PUT | `/api/config` | Update cell config |
| GET | `/api/health/history` | Recent health check history |

### Auth (`/api/auth/`)

| Method | Path | Description |
|---|---|---|
| POST | `/api/auth/login` | Create session |
| POST | `/api/auth/logout` | Destroy session |
| GET | `/api/auth/me` | Current user info |
| GET | `/api/auth/csrf-token` | Get CSRF token |
| POST | `/api/auth/change-password` | Change own password |
| POST | `/api/auth/admin/reset-password` | Admin: reset another user's password |
| GET | `/api/auth/users` | Admin: list users |

### Setup (`/api/setup/`)

| Method | Path | Description |
|---|---|---|
| GET | `/api/setup/status` | Setup complete flag + current step |
| GET | `/api/setup/step` | Current wizard step data |
| POST | `/api/setup/step` | Submit current step |
| POST | `/api/setup/complete` | Finalize setup |

### Network Services (`/api/dns/`, `/api/dhcp/`, `/api/ntp/`, `/api/network/`)

DNS records, DHCP leases and reservations, NTP status, network connectivity test.

### WireGuard (`/api/wireguard/`, `/api/peers/`)

Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.

### Email (`/api/email/`)

User account management, mailbox config, alias management, connectivity test.

### Calendar (`/api/calendar/`)

User, calendar, and contacts (CardDAV) management.

### Files (`/api/files/`)

WebDAV user management, file upload/download/delete, folder management.

### Routing (`/api/routing/`)

NAT rules, peer routes, exit node configuration.

### Vault (`/api/vault/`)

Certificate issue/revoke, CA certificate, trust key management, Age public key.

### Containers (`/api/containers/`)

List, start, stop, inspect containers; manage images and volumes.

### Cell Network (`/api/cells/`)

List connected cells, add/remove cell links, peer-sync.

### Connectivity (`/api/connectivity/`)

List exit nodes, configure WireGuard external / OpenVPN / Tor exits, assign per-peer exit policy.

### Service Store (`/api/store/`)

List available services, install, remove.

### Logs (`/api/logs/`)

Per-service log retrieval, log search, log statistics.

---

## DDNS

`DDNSManager` maintains a `<cell-name>.pic.ngo` DNS A record pointing at the cell's public IP. A background thread runs every 5 minutes and calls `provider.update(token, ip)` only when the IP changes.

Registration happens during the setup wizard (if domain mode is `pic_ngo`) via `provider.register(name, ip)`, which returns a bearer token stored in `data/api/.ddns_token`.

DDNS config lives in `cell_config.json` under the top-level `ddns` key:

```json
{
  "ddns": {
    "provider": "pic_ngo",
    "api_base_url": "https://ddns.pic.ngo",
    "totp_secret": "<base32 secret>"
  }
}
```

Registration requires a time-based OTP (`X-Register-OTP` header) derived from the shared `REGISTER_TOTP_SECRET` on the DDNS server. This prevents unauthorized subdomain registration.

Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`.

---

## Service Store

`ServiceStoreManager` fetches a manifest index from `http://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest declares:
- Container image
- Caddy routes (added to the Caddyfile)
- iptables rules
- Environment variables
- Health check endpoint

`POST /api/store/install` pulls the image, writes the Caddy route, applies iptables rules, and starts the container. `POST /api/store/remove` reverses this.

---

## Cell-to-Cell Networking

`CellLinkManager` manages WireGuard site-to-site tunnels between PIC cells. Each link is a WireGuard peer configured with a dedicated `/32` address and allowed-IPs covering the remote cell's subnet.

The peer-sync protocol (`/api/cells/peer-sync/`) exchanges public keys and allowed networks between cells using source-IP + WireGuard public key authentication (no session required).

Access control is per-service (calendar, files, mail, WebDAV) and enforced at the iptables level.

---

## Extended Connectivity

`ConnectivityManager` provides per-peer exit routing: traffic from a specific WireGuard peer can be routed through an alternate exit instead of the cell's default gateway.

Supported exits:
- **WireGuard external** — another WireGuard endpoint (e.g. a VPS)
- **OpenVPN** — OpenVPN client running in a container
- **Tor** — Tor SOCKS proxy with transparent redirection

Routing uses fwmark and `ip rule` / `ip route` in separate routing tables. Configuration is via `PUT /api/connectivity/peers/<peer_name>/exit`.

---

## Security Model

- **No open ports for the API** — Flask API binds to `127.0.0.1:3000` only; Caddy proxies HTTPS requests to it.
- **Session auth** — bcrypt passwords, Flask server-side sessions, CSRF double-submit.
- **Setup wizard gate** — all `/api/*` requests return 428 until setup is complete.
- **Role separation** — admin cannot access peer endpoints; peer cannot access admin endpoints.
- **HTTPS everywhere** — Caddy handles TLS; internal services are reached via reverse proxy paths.
- **Internal CA** — VaultManager issues certificates for services that don't use Let's Encrypt.
- **Docker socket isolation** — the Docker socket is mounted only into `cell-api`; other containers have no Docker access.
- **iptables firewall** — FirewallManager manages INPUT/FORWARD rules; WireGuard peer isolation is enforced at the packet level.

---

## Testing

```bash
make test            # unit tests (pytest, ~1500 functions)
make test-coverage   # coverage report in htmlcov/
```

Test layout:
- `tests/` — unit and endpoint tests; no running services required
- `tests/integration/` — require a running PIC stack
- `tests/e2e/` — Playwright UI tests and WireGuard integration tests

CI: Gitea Actions runs `pytest tests/ --ignore=tests/e2e --ignore=tests/integration` on every push.

---

## Development

```bash
# Full stack in Docker
make start
make stop
make logs

# Flask API without Docker (port 3000)
pip install -r api/requirements.txt
python api/app.py

# React UI dev server (port 5173, proxies /api → :3000)
cd webui && npm install && npm run dev

# Rebuild containers after code change
make build-api
make build-webui
```

Key files:
- `api/app.py` — Flask app, blueprint registration, before-request hooks, health monitor thread
- `api/managers.py` — singleton instantiation of all service managers
- `api/base_service_manager.py` — abstract base class all managers implement
- `api/config_manager.py` — `cell_config.json` read/write/validate/backup
- `api/service_bus.py` — pub/sub event system
- `webui/src/services/api.js` — Axios API client used by all UI pages
- `docker-compose.yml` — container definitions and network topology
- `Makefile` — all operational commands