docs: Phase 7 — update docs to reflect optional services migration
Email, calendar, and files are now optional store services, not always-on builtins. Updated README, QUICKSTART, Wiki, and service-developer-guide to reflect: dynamic nav, optional service install flow, correct egress identifiers (wireguard_ext/default vs wireguard/cell_internet), removed builtin/store distinction from manifest reference, 7 core containers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -52,7 +52,7 @@ Browser / WireGuard peer
|
|||||||
└── SetupManager first-run wizard state
|
└── SetupManager first-run wizard state
|
||||||
```
|
```
|
||||||
|
|
||||||
All 12 service containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default). Static IPs per container are defined in `docker-compose.yml`.
|
The 7 core containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default). Static IPs per container are defined in `docker-compose.yml`. Installed optional services join the same network via their own compose projects, managed by `ServiceComposer`.
|
||||||
|
|
||||||
Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.
|
Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.
|
||||||
|
|
||||||
@@ -101,7 +101,7 @@ The wizard collects:
|
|||||||
- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`)
|
- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`)
|
||||||
- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01`
|
- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01`
|
||||||
- **Timezone**
|
- **Timezone**
|
||||||
- **Initial services to enable**
|
- **Services to install** — optional services (email, calendar, files) to install after setup; each starts a background install via `ServiceStoreManager`
|
||||||
- **Admin password** — minimum 12 characters
|
- **Admin password** — minimum 12 characters
|
||||||
|
|
||||||
On completion:
|
On completion:
|
||||||
@@ -109,6 +109,7 @@ On completion:
|
|||||||
2. Cell identity is written to `config/api/cell_config.json`
|
2. Cell identity is written to `config/api/cell_config.json`
|
||||||
3. Caddy config is generated
|
3. Caddy config is generated
|
||||||
4. If domain mode is `pic_ngo`, the cell registers `<name>.pic.ngo` with the DDNS service
|
4. If domain mode is `pic_ngo`, the cell registers `<name>.pic.ngo` with the DDNS service
|
||||||
|
5. Each selected service is installed in a background thread
|
||||||
|
|
||||||
Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`.
|
Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`.
|
||||||
|
|
||||||
@@ -182,17 +183,17 @@ DNS records, DHCP leases and reservations, NTP status, network connectivity test
|
|||||||
|
|
||||||
Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.
|
Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.
|
||||||
|
|
||||||
### Email (`/api/email/`)
|
### Email (`/api/email/`) _(available when email service is installed)_
|
||||||
|
|
||||||
User account management, mailbox config, alias management, connectivity test.
|
User account management, mailbox config, alias management, connectivity test. Returns HTTP 404 when the email service is not installed (except `/api/email/status`).
|
||||||
|
|
||||||
### Calendar (`/api/calendar/`)
|
### Calendar (`/api/calendar/`) _(available when calendar service is installed)_
|
||||||
|
|
||||||
User, calendar, and contacts (CardDAV) management.
|
User, calendar, and contacts (CardDAV) management. Returns HTTP 404 when the calendar service is not installed (except `/api/calendar/status`).
|
||||||
|
|
||||||
### Files (`/api/files/`)
|
### Files (`/api/files/`) _(available when files service is installed)_
|
||||||
|
|
||||||
WebDAV user management, file upload/download/delete, folder management.
|
WebDAV user management, file upload/download/delete, folder management. Returns HTTP 404 when the files service is not installed (except `/api/files/status`).
|
||||||
|
|
||||||
### Routing (`/api/routing/`)
|
### Routing (`/api/routing/`)
|
||||||
|
|
||||||
@@ -252,7 +253,7 @@ Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`.
|
|||||||
|
|
||||||
### Navigation
|
### Navigation
|
||||||
|
|
||||||
The left-hand navigation contains a **Services** group (previously labelled "Store"). Both admin and peer users see this group. It has three collapsible sub-items: **Email**, **Calendar**, and **Files**.
|
The left-hand navigation contains a **Services** group. Both admin and peer users see it. Sub-items for installed services (Email, Calendar, Files, etc.) are added dynamically: the UI fetches `GET /api/services/active` on load and after each install/uninstall. Services not yet installed do not appear in the nav.
|
||||||
|
|
||||||
Legacy paths redirect to their new canonical locations:
|
Legacy paths redirect to their new canonical locations:
|
||||||
|
|
||||||
@@ -265,9 +266,13 @@ Legacy paths redirect to their new canonical locations:
|
|||||||
|
|
||||||
### Services page (`/services`)
|
### Services page (`/services`)
|
||||||
|
|
||||||
The top of the page shows a **Built-in** section with three cards — Email, Calendar, and Files. Each card has a **Manage** link that navigates to the corresponding sub-page.
|
A single unified catalog of all available services from the store index. Each card shows:
|
||||||
|
- Service name, description, version
|
||||||
|
- **Install** button (not installed) or **Uninstall** button (installed)
|
||||||
|
- **Open** link for installed services (navigates to the service sub-page)
|
||||||
|
- Running/stopped status dot for installed services
|
||||||
|
|
||||||
Below the Built-in section, the optional add-on store lists third-party services that can be installed or removed (see [Service Store (Add-ons)](#service-store-add-ons)).
|
The `pic-services-changed` custom DOM event is dispatched after install/uninstall, causing the nav to re-fetch active services immediately.
|
||||||
|
|
||||||
### Service sub-pages — admin view
|
### Service sub-pages — admin view
|
||||||
|
|
||||||
@@ -276,19 +281,18 @@ Each sub-page at `/services/email`, `/services/calendar`, and `/services/files`
|
|||||||
1. **Connection info** — hostnames, ports, and protocol details (e.g. IMAP/SMTP/Webmail, CalDAV/CardDAV, WebDAV/Filegator).
|
1. **Connection info** — hostnames, ports, and protocol details (e.g. IMAP/SMTP/Webmail, CalDAV/CardDAV, WebDAV/Filegator).
|
||||||
2. **Service status** — current running state fetched from the API.
|
2. **Service status** — current running state fetched from the API.
|
||||||
3. **Users list** — accounts registered with that service.
|
3. **Users list** — accounts registered with that service.
|
||||||
4. **Inline config form** — editable fields for that service's settings:
|
4. **Inline config form** — editable fields for that service's settings.
|
||||||
- Email: ports and mail domain.
|
|
||||||
- Calendar: port and data directory.
|
|
||||||
- Files: ports, data directory, and per-user quota.
|
|
||||||
|
|
||||||
Config forms save automatically with an 800 ms debounce after the last change. Dirty state persists through navigation: if a user edits the form and navigates away before the debounce fires, clicking **Apply Now** on return still saves the pending changes.
|
If the service is not installed, the page shows a `ServiceNotInstalledBanner` with a link to the catalog for admins, or a "contact your admin" message for peer users. All non-status API routes for uninstalled services return HTTP 404.
|
||||||
|
|
||||||
|
Config forms save automatically with an 800 ms debounce after the last change.
|
||||||
|
|
||||||
### Service sub-pages — peer view
|
### Service sub-pages — peer view
|
||||||
|
|
||||||
Peers access the same URLs without an admin gate. The peer view shows only:
|
Peers access the same URLs. The peer view shows only:
|
||||||
|
|
||||||
- Connection info (hostnames, ports, copy buttons).
|
- Connection info (hostnames, ports, copy buttons).
|
||||||
- Personal credentials for that service (email address, CalDAV username, or WebDAV username), fetched from `/api/peer/*`.
|
- Personal credentials for that service, fetched from `/api/peer/*`.
|
||||||
|
|
||||||
The config form and users list are not shown to peers.
|
The config form and users list are not shown to peers.
|
||||||
|
|
||||||
@@ -296,22 +300,36 @@ The config form and users list are not shown to peers.
|
|||||||
|
|
||||||
The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS/DHCP/NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.
|
The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS/DHCP/NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.
|
||||||
|
|
||||||
### API change
|
### Relevant API endpoints
|
||||||
|
|
||||||
`GET /api/config` now includes an `installed_services` field — a dict keyed by service ID — listing services currently installed on the cell.
|
| Method | Path | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| GET | `/api/services/active` | List installed services with id, name, subdomain, capabilities |
|
||||||
|
| GET | `/api/config` | Full cell config, includes `installed_services` dict |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Service Store (Add-ons)
|
## Service Store (Add-ons)
|
||||||
|
|
||||||
`ServiceStoreManager` fetches a manifest index from `http://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest declares:
|
Email, calendar, and file storage are store services — not part of the core stack. All optional functionality ships through this mechanism.
|
||||||
- Container image
|
|
||||||
- Caddy routes (added to the Caddyfile)
|
|
||||||
- iptables rules
|
|
||||||
- Environment variables
|
|
||||||
- Health check endpoint
|
|
||||||
|
|
||||||
`POST /api/store/install` pulls the image, writes the Caddy route, applies iptables rules, and starts the container. `POST /api/store/remove` reverses this.
|
`ServiceStoreManager` fetches a manifest index from `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest (`schema_version: 3`) declares:
|
||||||
|
|
||||||
|
- Container image and compose template
|
||||||
|
- Caddy subdomain routes
|
||||||
|
- Capabilities: `has_subdomain`, `has_accounts`, `has_admin_config`, `has_storage`, `has_egress`
|
||||||
|
- Account provisioning interface (`accounts.manager`)
|
||||||
|
- Backup declarations (`backup.volumes`, `backup.config_paths`)
|
||||||
|
- Egress routing policy (`egress.allowed`)
|
||||||
|
- Per-peer connection info template (`peer_config_template`)
|
||||||
|
|
||||||
|
`POST /api/store/install` fetches the manifest and compose template, validates them, renders the template with PIC-specific variables (`${PIC_DOMAIN}`, `${PIC_DATA_DIR}`, etc.), writes a per-service compose file, and brings the containers up via `ServiceComposer`. Caddy routes and DNS entries are applied automatically.
|
||||||
|
|
||||||
|
`POST /api/store/remove` checks for dependent services, stops and removes containers, and regenerates Caddy.
|
||||||
|
|
||||||
|
**`ServiceComposer`** (`api/service_composer.py`) manages the per-service compose lifecycle independently of the main stack. Each service gets its own compose project at `data/services/<id>/docker-compose.yml`. On startup, `reapply_active_services()` brings up containers for all recorded installs.
|
||||||
|
|
||||||
|
See `docs/service-developer-guide.md` for the full manifest schema reference and submission process.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -354,7 +372,7 @@ Routing uses fwmark and `ip rule` / `ip route` in separate routing tables. Confi
|
|||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
make test # unit tests (pytest, ~1500 functions)
|
make test # unit tests (pytest, ~1900+ functions)
|
||||||
make test-coverage # coverage report in htmlcov/
|
make test-coverage # coverage report in htmlcov/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+29
-3
@@ -8,7 +8,8 @@ This guide walks through a first-time PIC installation on a clean Linux host.
|
|||||||
|
|
||||||
- Linux x86-64 host — Debian, Ubuntu, Fedora, RHEL, or Alpine
|
- Linux x86-64 host — Debian, Ubuntu, Fedora, RHEL, or Alpine
|
||||||
- 2 GB+ RAM, 10 GB+ disk
|
- 2 GB+ RAM, 10 GB+ disk
|
||||||
- Ports 53, 80, 443, 51820/udp, 25, 587, 993 available
|
- Always-required ports: 53, 80, 443, 51820/udp
|
||||||
|
- Email service only (when installed): 25, 587, 993
|
||||||
|
|
||||||
The installer handles all software dependencies (git, docker, make, etc.) automatically.
|
The installer handles all software dependencies (git, docker, make, etc.) automatically.
|
||||||
|
|
||||||
@@ -71,10 +72,10 @@ The wizard asks for:
|
|||||||
- `http01` — Let's Encrypt via HTTP-01 (no wildcard; cell must be reachable on port 80)
|
- `http01` — Let's Encrypt via HTTP-01 (no wildcard; cell must be reachable on port 80)
|
||||||
- `lan` — internal CA, no internet required (for LAN-only installs)
|
- `lan` — internal CA, no internet required (for LAN-only installs)
|
||||||
- **Timezone**
|
- **Timezone**
|
||||||
- **Services to enable** — email, calendar, files, WireGuard
|
- **Services to install** — email, calendar, files (optional; installed in the background after setup completes; can be added later via the Services store page)
|
||||||
- **Admin password** — minimum 12 characters, must contain uppercase, lowercase, and a digit
|
- **Admin password** — minimum 12 characters, must contain uppercase, lowercase, and a digit
|
||||||
|
|
||||||
Click **Complete Setup**. The wizard creates the admin account, writes cell identity to `config/api/cell_config.json`, and redirects to the login page.
|
Click **Complete Setup**. The wizard creates the admin account, writes cell identity to `config/api/cell_config.json`, and redirects to the login page. Any services you selected begin installing in the background.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -101,6 +102,31 @@ Once connected, `*.cell` names resolve through the cell's CoreDNS and traffic ca
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Installing and managing services
|
||||||
|
|
||||||
|
Email, calendar, and file storage are optional services installed from the built-in service store. They are not running by default.
|
||||||
|
|
||||||
|
**To install a service:**
|
||||||
|
|
||||||
|
1. Go to **Services** in the sidebar.
|
||||||
|
2. Find the service card (Email, Calendar, Files, or any other listed service).
|
||||||
|
3. Click **Install**. PIC fetches the manifest, starts the container, and wires up DNS and Caddy routes automatically.
|
||||||
|
4. The service appears in the sidebar navigation once installation completes.
|
||||||
|
|
||||||
|
**To check service status:**
|
||||||
|
|
||||||
|
The Services page shows each installed service as "running" or "stopped". You can also check via the API:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://<host-ip>:3000/api/services/active
|
||||||
|
```
|
||||||
|
|
||||||
|
**To uninstall a service:**
|
||||||
|
|
||||||
|
Click **Uninstall** on the service card. The container is stopped and removed. Data in `data/services/<id>/` is kept on disk unless you delete it manually.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Day-to-day operations
|
## Day-to-day operations
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -16,15 +16,11 @@ Browser
|
|||||||
├── cell-dhcp :67/udp dnsmasq
|
├── cell-dhcp :67/udp dnsmasq
|
||||||
├── cell-ntp :123/udp chrony
|
├── cell-ntp :123/udp chrony
|
||||||
├── cell-wireguard :51820/udp WireGuard VPN
|
├── cell-wireguard :51820/udp WireGuard VPN
|
||||||
├── cell-mail :25/:587/:993 Postfix + Dovecot
|
|
||||||
├── cell-radicale 127.0.0.1:5232 CalDAV/CardDAV
|
|
||||||
├── cell-webdav 127.0.0.1:8080 WebDAV
|
|
||||||
├── cell-rainloop 127.0.0.1:8888 Webmail (RainLoop)
|
|
||||||
├── cell-filegator 127.0.0.1:8082 File manager (Filegator)
|
|
||||||
└── cell-webui :8081 React UI (Nginx)
|
└── cell-webui :8081 React UI (Nginx)
|
||||||
|
(+ per-service containers, started when a service is installed)
|
||||||
```
|
```
|
||||||
|
|
||||||
All containers run on a custom Docker bridge network (`cell-network`, default subnet `172.20.0.0/16`). Static IPs per container are set in `docker-compose.yml` and can be overridden via `.env`.
|
Core containers run on a Docker bridge network (`cell-network`, default subnet `172.20.0.0/16`). Static IPs per container are set in `docker-compose.yml` and can be overridden via `.env`. Installed service containers join the same network with their own compose projects managed by `ServiceComposer`.
|
||||||
|
|
||||||
The Flask API (`api/app.py`) contains REST endpoints and a background health-monitoring thread. Service managers are instantiated as singletons in `api/managers.py`. The single source of truth for runtime configuration is `config/api/cell_config.json`, managed by `ConfigManager`.
|
The Flask API (`api/app.py`) contains REST endpoints and a background health-monitoring thread. Service managers are instantiated as singletons in `api/managers.py`. The single source of truth for runtime configuration is `config/api/cell_config.json`, managed by `ConfigManager`.
|
||||||
|
|
||||||
@@ -46,9 +42,9 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g
|
|||||||
- **Cell-to-cell networking** — WireGuard-based site-to-site links between PIC cells with service-level access control (calendar, files, mail, WebDAV) and a peer-sync protocol.
|
- **Cell-to-cell networking** — WireGuard-based site-to-site links between PIC cells with service-level access control (calendar, files, mail, WebDAV) and a peer-sync protocol.
|
||||||
- **Certificate authority** — `vault_manager` issues and revokes TLS certificates for internal services.
|
- **Certificate authority** — `vault_manager` issues and revokes TLS certificates for internal services.
|
||||||
- **Network services** — CoreDNS (`.cell` TLD), dnsmasq DHCP, chrony NTP.
|
- **Network services** — CoreDNS (`.cell` TLD), dnsmasq DHCP, chrony NTP.
|
||||||
- **Email** — Postfix + Dovecot via `docker-mailserver`.
|
- **Email** _(optional, install via Service Store)_ — Postfix + Dovecot via `docker-mailserver`.
|
||||||
- **Calendar/contacts** — Radicale CalDAV/CardDAV.
|
- **Calendar/contacts** _(optional, install via Service Store)_ — Radicale CalDAV/CardDAV.
|
||||||
- **File storage** — WebDAV with per-user accounts; Filegator for browser-based file management.
|
- **File storage** _(optional, install via Service Store)_ — WebDAV with per-user accounts; Filegator for browser-based file management.
|
||||||
- **Container manager** — start/stop/inspect containers, pull images, manage volumes via the Docker SDK.
|
- **Container manager** — start/stop/inspect containers, pull images, manage volumes via the Docker SDK.
|
||||||
- **Firewall manager** — iptables rule management (`firewall_manager.py`).
|
- **Firewall manager** — iptables rule management (`firewall_manager.py`).
|
||||||
- **Structured logging** — JSON logs with rotation (5 MB / 5 backups per service), log search, and per-service verbosity control.
|
- **Structured logging** — JSON logs with rotation (5 MB / 5 backups per service), log search, and per-service verbosity control.
|
||||||
@@ -61,7 +57,7 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g
|
|||||||
- Docker Engine and Docker Compose (v2 plugin or v1 standalone)
|
- Docker Engine and Docker Compose (v2 plugin or v1 standalone)
|
||||||
- Python 3.10+ (for `make setup` and local development; not needed at runtime)
|
- Python 3.10+ (for `make setup` and local development; not needed at runtime)
|
||||||
- 2 GB+ RAM, 10 GB+ disk
|
- 2 GB+ RAM, 10 GB+ disk
|
||||||
- Ports available: 53, 67/udp, 80, 443, 51820/udp, 25, 587, 993
|
- Ports available: 53, 67/udp, 80, 443, 51820/udp (plus 25, 587, 993 when the email service is installed)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -87,22 +83,13 @@ Port assignments and container IPs are configured in `.env` in the project root.
|
|||||||
| Variable | Default | Description |
|
| Variable | Default | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `CELL_NETWORK` | `172.20.0.0/16` | Docker bridge subnet |
|
| `CELL_NETWORK` | `172.20.0.0/16` | Docker bridge subnet |
|
||||||
| `CADDY_IP` through `FILEGATOR_IP` | `172.20.0.2`–`.13` | Static IP per container |
|
| `CADDY_IP` through `WG_IP` | `172.20.0.2`–`.9` | Static IP per core container |
|
||||||
| `DNS_PORT` | `53` | DNS (UDP + TCP) |
|
| `DNS_PORT` | `53` | DNS (UDP + TCP) |
|
||||||
| `DHCP_PORT` | `67` | DHCP (UDP) |
|
| `DHCP_PORT` | `67` | DHCP (UDP) |
|
||||||
| `NTP_PORT` | `123` | NTP (UDP) |
|
| `NTP_PORT` | `123` | NTP (UDP) |
|
||||||
| `WG_PORT` | `51820` | WireGuard listen port (UDP) |
|
| `WG_PORT` | `51820` | WireGuard listen port (UDP) |
|
||||||
| `API_PORT` | `3000` | Flask API (127.0.0.1 only) |
|
| `API_PORT` | `3000` | Flask API (127.0.0.1 only) |
|
||||||
| `WEBUI_PORT` | `8081` | React UI |
|
| `WEBUI_PORT` | `8081` | React UI |
|
||||||
| `MAIL_SMTP_PORT` | `25` | SMTP |
|
|
||||||
| `MAIL_SUBMISSION_PORT` | `587` | SMTP submission |
|
|
||||||
| `MAIL_IMAP_PORT` | `993` | IMAP |
|
|
||||||
| `RADICALE_PORT` | `5232` | CalDAV (127.0.0.1 only) |
|
|
||||||
| `WEBDAV_PORT` | `8080` | WebDAV (127.0.0.1 only) |
|
|
||||||
| `RAINLOOP_PORT` | `8888` | Webmail |
|
|
||||||
| `FILEGATOR_PORT` | `8082` | File manager UI |
|
|
||||||
| `WEBDAV_USER` | `admin` | WebDAV basic-auth username |
|
|
||||||
| `WEBDAV_PASS` | _(unset)_ | WebDAV basic-auth password |
|
|
||||||
| `FLASK_DEBUG` | _(unset)_ | Set to `1` for Flask debug mode; do not use in production |
|
| `FLASK_DEBUG` | _(unset)_ | Set to `1` for Flask debug mode; do not use in production |
|
||||||
| `PUID` / `PGID` | current user | UID/GID passed to the WireGuard container |
|
| `PUID` / `PGID` | current user | UID/GID passed to the WireGuard container |
|
||||||
|
|
||||||
@@ -116,18 +103,14 @@ Cell identity (cell name, domain mode, timezone) is set through the first-run wi
|
|||||||
|
|
||||||
- `80` / `443` — Caddy (HTTP/HTTPS reverse proxy)
|
- `80` / `443` — Caddy (HTTP/HTTPS reverse proxy)
|
||||||
- `51820/udp` — WireGuard
|
- `51820/udp` — WireGuard
|
||||||
- `25` / `587` / `993` — mail
|
|
||||||
- `53` — DNS
|
- `53` — DNS
|
||||||
- `67/udp` — DHCP
|
- `67/udp` — DHCP
|
||||||
- `8081` — Web UI
|
- `8081` — Web UI
|
||||||
|
- `25` / `587` / `993` — mail _(only when the email service is installed)_
|
||||||
|
|
||||||
**Ports bound to `127.0.0.1` only:**
|
**Ports bound to `127.0.0.1` only:**
|
||||||
|
|
||||||
- `3000` — Flask API
|
- `3000` — Flask API
|
||||||
- `5232` — Radicale (CalDAV)
|
|
||||||
- `8080` — WebDAV
|
|
||||||
- `8888` — Webmail
|
|
||||||
- `8082` — Filegator
|
|
||||||
|
|
||||||
The API uses session-based authentication (admin and peer roles). The Docker socket is mounted into `cell-api`; treat access to port 3000 as equivalent to root access on the host.
|
The API uses session-based authentication (admin and peer roles). The Docker socket is mounted into `cell-api`; treat access to port 3000 as equivalent to root access on the host.
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,8 @@ This guide is for developers who want to build services that integrate with Pers
|
|||||||
5. [Backup integration](#5-backup-integration)
|
5. [Backup integration](#5-backup-integration)
|
||||||
6. [Egress routing](#6-egress-routing)
|
6. [Egress routing](#6-egress-routing)
|
||||||
7. [Quick-start example](#7-quick-start-example)
|
7. [Quick-start example](#7-quick-start-example)
|
||||||
8. [Submitting to the store](#8-submitting-to-the-store)
|
8. [Reference implementations](#8-reference-implementations)
|
||||||
|
9. [Submitting to the store](#9-submitting-to-the-store)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -29,19 +30,15 @@ A PIC service is a Docker container (or a set of containers) that plugs into the
|
|||||||
- Which paths to include in automated backups
|
- Which paths to include in automated backups
|
||||||
- Which outbound network interfaces the service is allowed to use
|
- Which outbound network interfaces the service is allowed to use
|
||||||
|
|
||||||
PIC has two kinds of services:
|
All PIC services are **store services** — optional packages installed by the cell admin from the `pic-services` catalog. PIC downloads the manifest, renders a per-service Docker Compose file, and starts the containers. The core PIC stack (DNS, DHCP, NTP, WireGuard, Caddy, API, WebUI) runs independently of any installed services.
|
||||||
|
|
||||||
**Builtins** ship with PIC itself (email, calendar, files). Their containers are declared in the main `docker-compose.yml`. Their manifests live under `api/services/builtins/<id>/manifest.json`. They use Python manager classes for account operations because they run inside the same Docker network and share internal access.
|
The email, calendar, and files services (in `pic-services/services/`) are the reference implementations and show the full feature set. The `ServiceRegistry` in `api/service_registry.py` is the single source of truth for all installed services. `CaddyManager`, the backup system, and the peer services endpoint all read from it rather than from hardcoded lists.
|
||||||
|
|
||||||
**Store services** are third-party packages installed by the cell admin from the PIC service store. PIC downloads the manifest from the store index, allocates a static IP for the container in the service pool (`172.20.x.x`, offsets 20–254 within your `ip_range` subnet), generates a Docker Compose override file, and starts the container. Store services use a small HTTP API (described in section 4) for account operations.
|
|
||||||
|
|
||||||
The `ServiceRegistry` in `api/service_registry.py` is the single source of truth for both kinds. `CaddyManager`, the backup system, and the peer services endpoint all read from it rather than from hardcoded lists.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Manifest reference
|
## 2. Manifest reference
|
||||||
|
|
||||||
The manifest is a JSON file with `"schema_version": 3`. Every field is described below. The three builtin manifests (`email`, `calendar`, `files`) are the canonical examples.
|
The manifest is a JSON file with `"schema_version": 3`. Every field is described below. The `email`, `calendar`, and `files` manifests in `pic-services/services/` are the canonical reference examples.
|
||||||
|
|
||||||
### Top-level identity fields
|
### Top-level identity fields
|
||||||
|
|
||||||
@@ -53,7 +50,7 @@ The manifest is a JSON file with `"schema_version": 3`. Every field is described
|
|||||||
| `description` | string | yes | One-sentence description shown in the UI. |
|
| `description` | string | yes | One-sentence description shown in the UI. |
|
||||||
| `version` | string | yes | Semver string for the service package itself (e.g. `"1.0.0"`). |
|
| `version` | string | yes | Semver string for the service package itself (e.g. `"1.0.0"`). |
|
||||||
| `author` | string | yes | Your name or organisation. |
|
| `author` | string | yes | Your name or organisation. |
|
||||||
| `kind` | string | yes | `"builtin"` for built-in services, `"store"` for third-party packages. |
|
| `kind` | string | yes | Must be `"store"`. |
|
||||||
| `min_pic_version` | string | no | Minimum PIC version required (e.g. `"1.0"`). |
|
| `min_pic_version` | string | no | Minimum PIC version required (e.g. `"1.0"`). |
|
||||||
|
|
||||||
```json
|
```json
|
||||||
@@ -99,7 +96,7 @@ These fields are only read when `has_subdomain` is `true`.
|
|||||||
|
|
||||||
| Field | Type | Required | Description |
|
| Field | Type | Required | Description |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `subdomain` | string | yes (if `has_subdomain`) | The primary subdomain (e.g. `"notes"`). Results in `notes.<cell-domain>`. Must not collide with reserved names: `api`, `webui`, `admin`, `www`, `mail`, `ns1`, `ns2`, `git`, `registry`, `install`. |
|
| `subdomain` | string | yes (if `has_subdomain`) | The primary subdomain (e.g. `"notes"`). Results in `notes.<cell-domain>`. Must not collide with reserved names: `api`, `webui`, `admin`, `www`, `ns1`, `ns2`, `git`, `registry`, `install`. |
|
||||||
| `extra_subdomains` | array of strings | no | Additional subdomains that point to the same backend (e.g. `["webmail"]`). |
|
| `extra_subdomains` | array of strings | no | Additional subdomains that point to the same backend (e.g. `["webmail"]`). |
|
||||||
| `backend` | string | yes (if `has_subdomain`) | The container-name:port combination that Caddy proxies to (e.g. `"cell-notes:8080"`). Uses Docker DNS on the `cell-network`. |
|
| `backend` | string | yes (if `has_subdomain`) | The container-name:port combination that Caddy proxies to (e.g. `"cell-notes:8080"`). Uses Docker DNS on the `cell-network`. |
|
||||||
| `extra_backends` | object | no | Maps extra subdomain names to separate backends. Key is the subdomain string; value is the backend string. The email service uses this to send `webdav.*` to a different container than `files.*`. |
|
| `extra_backends` | object | no | Maps extra subdomain names to separate backends. Key is the subdomain string; value is the backend string. The email service uses this to send `webdav.*` to a different container than `files.*`. |
|
||||||
@@ -176,7 +173,7 @@ Required when `has_accounts` is `true`.
|
|||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `manager` | string | For builtins: the Python manager name used internally (`"email_manager"`, `"calendar_manager"`, `"file_manager"`). For store services: set to `"http"` to indicate the HTTP API (section 4). |
|
| `manager` | string | Set to `"http"` for store services — PIC will call your container's HTTP API for account operations (see section 4). The reference services (`email`, `calendar`, `files`) use internal manager names (`"email_manager"`, `"calendar_manager"`, `"file_manager"`). |
|
||||||
| `credentials` | array of strings | Names of credential fields this service issues per peer. Most services use `["password"]`. The names appear in `peer_config_template` tokens. |
|
| `credentials` | array of strings | Names of credential fields this service issues per peer. Most services use `["password"]`. The names appear in `peer_config_template` tokens. |
|
||||||
|
|
||||||
```json
|
```json
|
||||||
@@ -188,7 +185,7 @@ Required when `has_accounts` is `true`.
|
|||||||
|
|
||||||
### `compose`
|
### `compose`
|
||||||
|
|
||||||
For builtins set this to `null` — their containers come from the main `docker-compose.yml`. For store services this field is unused at the manifest level; compose configuration is provided via `compose-template.yml` in the package (see section 3).
|
Unused at the manifest level. Compose configuration is provided via `compose-template.yml` in the service package (see section 3). Set to `null` in the manifest.
|
||||||
|
|
||||||
### `backup`
|
### `backup`
|
||||||
|
|
||||||
@@ -221,12 +218,12 @@ Required when `has_egress` is `true`. Declares which outbound network interfaces
|
|||||||
| `default` | string | The interface selected when the admin has not changed anything. |
|
| `default` | string | The interface selected when the admin has not changed anything. |
|
||||||
| `allowed` | array of strings | The complete set of interfaces the admin can choose from. |
|
| `allowed` | array of strings | The complete set of interfaces the admin can choose from. |
|
||||||
|
|
||||||
Valid interface identifiers: `cell_internet`, `openvpn`, `wireguard`, `tor`, `pic_cell`.
|
Valid interface identifiers: `default`, `wireguard_ext`, `openvpn`, `tor`.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
"egress": {
|
"egress": {
|
||||||
"default": "cell_internet",
|
"default": "default",
|
||||||
"allowed": ["cell_internet", "openvpn", "wireguard", "tor", "pic_cell"]
|
"allowed": ["default", "wireguard_ext", "openvpn", "tor"]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -315,7 +312,7 @@ This section covers two related things: the `AccountManager` class that is PIC's
|
|||||||
|
|
||||||
### How AccountManager works
|
### How AccountManager works
|
||||||
|
|
||||||
`AccountManager` (`api/account_manager.py`) is the single entry point for all account operations across every service type. It is instantiated once in `api/managers.py` and holds references to the three builtin service managers (`email_manager`, `calendar_manager`, `file_manager`).
|
`AccountManager` (`api/account_manager.py`) is the single entry point for all account operations across every service type. It is instantiated once in `api/managers.py` and holds references to the service managers used by the reference services (`email_manager`, `calendar_manager`, `file_manager`).
|
||||||
|
|
||||||
When a peer account is provisioned, `AccountManager`:
|
When a peer account is provisioned, `AccountManager`:
|
||||||
|
|
||||||
@@ -463,9 +460,9 @@ The following PIC API endpoints are available for all services (builtins and sto
|
|||||||
| `POST` | `/api/services/catalog/<id>/restart` | Restart the service containers. Builtins restart via the main compose stack; store services restart via their own compose project. |
|
| `POST` | `/api/services/catalog/<id>/restart` | Restart the service containers. Builtins restart via the main compose stack; store services restart via their own compose project. |
|
||||||
| `POST` | `/api/services/catalog/<id>/reconfigure` | Re-render the compose file from the template and re-apply with `up -d` (rolling update). Store services only — builtins are reconfigured through their own settings routes. The request body must include a `compose_template` field containing the new template content. |
|
| `POST` | `/api/services/catalog/<id>/reconfigure` | Re-render the compose file from the template and re-apply with `up -d` (rolling update). Store services only — builtins are reconfigured through their own settings routes. The request body must include a `compose_template` field containing the new template content. |
|
||||||
|
|
||||||
### Store service HTTP API (planned)
|
### Store service HTTP API
|
||||||
|
|
||||||
When `accounts.manager` is `"http"`, PIC will call your container's HTTP API for account operations. This path is not yet implemented in `AccountManager`; the dispatch table only covers `email_manager`, `calendar_manager`, and `file_manager`. The interface below is the planned contract — implement it now so your service is ready when HTTP dispatch is wired up.
|
When `accounts.manager` is `"http"`, PIC will call your container's HTTP API for account operations. **HTTP dispatch is not yet wired up in `AccountManager`** — the current dispatch table covers only `email_manager`, `calendar_manager`, and `file_manager` (used by the reference services). Implement this interface now so your service is ready when HTTP dispatch ships.
|
||||||
|
|
||||||
The base path is `/service-api/accounts` on your container's internal address. There is no authentication on this API — it is reachable only from within the `cell-network` Docker network.
|
The base path is `/service-api/accounts` on your container's internal address. There is no authentication on this API — it is reachable only from within the `cell-network` Docker network.
|
||||||
|
|
||||||
@@ -551,13 +548,12 @@ The valid values for `egress.allowed` and what they mean:
|
|||||||
|
|
||||||
| Value | Path |
|
| Value | Path |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `cell_internet` | Default route through the cell's WAN interface (no VPN). |
|
| `default` | Default route through the cell's WAN interface (no VPN). |
|
||||||
|
| `wireguard_ext` | Traffic leaves through `wg_ext0` (fwmark `0x10`, table 110). |
|
||||||
| `openvpn` | Traffic leaves through `tun0` (fwmark `0x20`, table 120). |
|
| `openvpn` | Traffic leaves through `tun0` (fwmark `0x20`, table 120). |
|
||||||
| `wireguard` | Traffic leaves through a second WireGuard interface `wg_ext0` (fwmark `0x10`, table 110). |
|
|
||||||
| `tor` | Traffic is redirected to the Tor transparent proxy on port 9040 (fwmark `0x30`, table 130). |
|
| `tor` | Traffic is redirected to the Tor transparent proxy on port 9040 (fwmark `0x30`, table 130). |
|
||||||
| `pic_cell` | Traffic routes through a connected peer cell via a site-to-site WireGuard link. |
|
|
||||||
|
|
||||||
List only the interfaces that make sense for your service in `allowed`. Listing all five is fine if your service is general-purpose. The `default` value is used when the admin has not changed anything. Always include `cell_internet` in `allowed` so the admin has a way to use the normal path.
|
List only the interfaces that make sense for your service in `allowed`. The `default` value is used when the admin has not changed anything. Always include `default` in `allowed` so the admin has a way to use the normal path.
|
||||||
|
|
||||||
The egress field in the manifest tells PIC what options to present in the UI. Actual enforcement requires the cell to have the corresponding exit type configured (an OpenVPN config uploaded, a WireGuard external config active, etc.). If the chosen exit is not active, packets will be dropped by the kill-switch FORWARD rule in `cell-wireguard`.
|
The egress field in the manifest tells PIC what options to present in the UI. Actual enforcement requires the cell to have the corresponding exit type configured (an OpenVPN config uploaded, a WireGuard external config active, etc.). If the chosen exit is not active, packets will be dropped by the kill-switch FORWARD rule in `cell-wireguard`.
|
||||||
|
|
||||||
@@ -635,7 +631,21 @@ The result is that any WireGuard peer can reach `https://home.alice.pic.ngo/` im
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Submitting to the store
|
## 8. Reference implementations
|
||||||
|
|
||||||
|
The `email`, `calendar`, and `files` services in `pic-services/services/` are the canonical examples of a complete store service. They demonstrate the full feature set:
|
||||||
|
|
||||||
|
| Service | Notable features demonstrated |
|
||||||
|
|---|---|
|
||||||
|
| `email` | `has_accounts`, `has_egress`, multi-container (`cell-mail` + `cell-rainloop`), `extra_backends`, custom image baking defaults via Dockerfile |
|
||||||
|
| `calendar` | `has_accounts`, CalDAV `peer_config_template`, htpasswd account provisioning |
|
||||||
|
| `files` | `has_accounts`, `has_storage`, WebDAV + Filegator `extra_backends`, `backup.volumes` with multiple entries |
|
||||||
|
|
||||||
|
When in doubt about how to structure your manifest or compose template, use these as the reference.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Submitting to the store
|
||||||
|
|
||||||
### Package format
|
### Package format
|
||||||
|
|
||||||
@@ -700,32 +710,32 @@ Increment `version` in `manifest.json` with every change you submit. PIC does no
|
|||||||
|
|
||||||
## Appendix: manifest field quick reference
|
## Appendix: manifest field quick reference
|
||||||
|
|
||||||
| Field | Builtin | Store | Notes |
|
| Field | Required | Notes |
|
||||||
|---|---|---|---|
|
|---|---|---|
|
||||||
| `schema_version` | required | required | Must be `3` |
|
| `schema_version` | yes | Must be `3` |
|
||||||
| `id` | required | required | |
|
| `id` | yes | |
|
||||||
| `name` | required | required | |
|
| `name` | yes | |
|
||||||
| `description` | required | required | |
|
| `description` | yes | |
|
||||||
| `version` | required | required | |
|
| `version` | yes | |
|
||||||
| `author` | required | required | |
|
| `author` | yes | |
|
||||||
| `kind` | required | required | `"builtin"` or `"store"` |
|
| `kind` | yes | Must be `"store"` |
|
||||||
| `min_pic_version` | optional | optional | |
|
| `min_pic_version` | no | |
|
||||||
| `capabilities.*` | required | required | All six flags must be present |
|
| `capabilities.*` | yes | All six flags must be present |
|
||||||
| `subdomain` | if `has_subdomain` | if `has_subdomain` | |
|
| `subdomain` | if `has_subdomain` | |
|
||||||
| `extra_subdomains` | optional | optional | |
|
| `extra_subdomains` | no | |
|
||||||
| `backend` | if `has_subdomain` | if `has_subdomain` | |
|
| `backend` | if `has_subdomain` | |
|
||||||
| `extra_backends` | optional | optional | |
|
| `extra_backends` | no | |
|
||||||
| `containers` | optional | optional | Informational |
|
| `containers` | no | Informational |
|
||||||
| `config_schema` | if `has_admin_config` | if `has_admin_config` | |
|
| `config_schema` | if `has_admin_config` | |
|
||||||
| `peer_config_template` | if `has_accounts` | if `has_accounts` | |
|
| `peer_config_template` | if `has_accounts` | |
|
||||||
| `accounts` | if `has_accounts` | if `has_accounts` | |
|
| `accounts` | if `has_accounts` | |
|
||||||
| `compose` | `null` | `null` | |
|
| `compose` | no | Always `null` — compose config goes in `compose-template.yml` |
|
||||||
| `backup` | if `has_storage` | if `has_storage` | |
|
| `backup` | if `has_storage` | |
|
||||||
| `egress` | if `has_egress` | if `has_egress` | |
|
| `egress` | if `has_egress` | |
|
||||||
| `storage` | if `has_storage` | if `has_storage` | |
|
| `storage` | if `has_storage` | |
|
||||||
| `image` | not used | required | |
|
| `image` | yes | Must match `git.pic.ngo/roof/*` |
|
||||||
| `container_name` | not used | required | |
|
| `container_name` | yes | Must match `^cell-[a-z0-9][a-z0-9-]{0,30}$` |
|
||||||
| `volumes` | not used | optional | |
|
| `volumes` | no | |
|
||||||
| `env` | not used | optional | |
|
| `env` | no | |
|
||||||
| `iptables_rules` | not used | optional | |
|
| `iptables_rules` | no | |
|
||||||
| `caddy_route` | not used | optional | |
|
| `caddy_route` | no | |
|
||||||
|
|||||||
Reference in New Issue
Block a user