Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 roof
|
4b3d695805 | ||
|
roof
|
2ab3d2d5ac | ||
|
roof
|
c430a392b8 | ||
|
roof
|
fa00c90328 |
@@ -91,3 +91,5 @@ backups/
|
||||
# Coverage data
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
CLAUDE.md
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file is the primary context source for Claude Code in this repository. Read it fully before touching any code.
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Personal Internet Cell (PIC)** is a self-hosted digital infrastructure platform for individuals who want full ownership of their core internet services without relying on cloud providers.
|
||||
|
||||
A PIC instance runs DNS, NTP, WireGuard VPN, an HTTPS reverse proxy (Caddy), an internal certificate authority, and — as optional store services — email (SMTP/IMAP), calendar/contacts (CalDAV/CardDAV), file storage (WebDAV), and extended-connectivity exits (WireGuard-ext, OpenVPN, Tor, sshuttle, proxy) — all managed from a single REST API and a React web UI. No manual config-file editing is required for normal operations.
|
||||
|
||||
**Primary users:** technically capable individuals, homelab operators, small families or teams.
|
||||
|
||||
**What the product optimizes for:**
|
||||
- One-command install, browser-based first-run wizard, no manual `.env` editing for identity
|
||||
- Everything managed through the API and UI — the user should never need to `ssh` for day-to-day operations
|
||||
- Security by default: session auth, CSRF protection, WireGuard isolation, internal CA, no open API port
|
||||
- Reliability and observability: structured logs, health monitoring, automated config backups
|
||||
|
||||
**Key constraints:**
|
||||
- Runs on a single Linux host with Docker; no Kubernetes, no swarm
|
||||
- Must work on Debian, Ubuntu, Fedora, RHEL, and Alpine
|
||||
- The Flask API must never be exposed directly; Caddy always proxies it
|
||||
- All secrets live in `data/` (git-ignored), never in the repo
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
### Backend
|
||||
- **Python 3.11** — Flask REST API (`api/app.py`)
|
||||
- **Flask** — routing, sessions, before-request hooks (enforce_setup, enforce_auth, check_csrf)
|
||||
- **bcrypt** — password hashing in `AuthManager`
|
||||
- **Docker SDK for Python** — container lifecycle in `ContainerManager`
|
||||
- **PyNaCl / Age** — encryption in `VaultManager`
|
||||
- **pyotp** — TOTP for DDNS registration
|
||||
|
||||
### Frontend
|
||||
- **React 18** — SPA
|
||||
- **Vite** — dev server and build (proxies `/api` → `:3000`)
|
||||
- **Tailwind CSS** — all styling; no custom CSS files
|
||||
- **Axios** — all API calls go through `src/services/api.js`
|
||||
|
||||
### Infrastructure
|
||||
- **Docker Compose** — all 12+ service containers
|
||||
- **Caddy** — reverse proxy, TLS termination (Let's Encrypt DNS-01 or HTTP-01 or internal CA)
|
||||
- **CoreDNS** — `.cell` TLD authoritative DNS + split-horizon for the effective domain
|
||||
- **chrony** — NTP
|
||||
- **WireGuard** — VPN (kernel module, not userspace)
|
||||
- **Postfix + Dovecot** — email via `docker-mailserver`
|
||||
- **Radicale** — CalDAV/CardDAV
|
||||
- **PowerDNS** — authoritative DNS on the DDNS VPS (separate repo: `pic-ddns`)
|
||||
|
||||
### CI/CD
|
||||
- **Gitea Actions** — unit tests on every push, image builds on tag
|
||||
- **act_runner** — self-hosted runner on pic0 (192.168.31.51)
|
||||
- **Gitea Container Registry** — images pushed to `git.pic.ngo`
|
||||
|
||||
Do not introduce: Redux, styled-components, SQLAlchemy, Celery, or any async framework (asyncio/FastAPI) into the main API unless explicitly requested.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser / WireGuard peer
|
||||
└── Caddy (:80/:443) TLS termination, reverse proxy
|
||||
└── React SPA (:8081) Vite + Tailwind (Nginx in container)
|
||||
└── Flask API (:3000) REST API, bound to 127.0.0.1 only
|
||||
├── NetworkManager CoreDNS, chrony
|
||||
├── WireGuardManager WireGuard peer lifecycle
|
||||
├── PeerRegistry peer registration and trust
|
||||
├── EmailManager Postfix + Dovecot
|
||||
├── CalendarManager Radicale CalDAV/CardDAV
|
||||
├── FileManager WebDAV + Filegator
|
||||
├── RoutingManager iptables NAT and routing
|
||||
├── FirewallManager iptables INPUT/FORWARD rules
|
||||
├── VaultManager internal CA, TLS certs, Age encryption
|
||||
├── ContainerManager Docker SDK
|
||||
├── CellLinkManager site-to-site WireGuard links
|
||||
├── ConnectivityManager per-peer exit routing (WG ext, OpenVPN, Tor)
|
||||
├── DDNSManager dynamic DNS heartbeat
|
||||
├── ServiceStoreManager optional service install/remove
|
||||
├── CaddyManager Caddyfile generation and reload
|
||||
├── AuthManager bcrypt passwords, session auth, RBAC
|
||||
└── SetupManager first-run wizard state
|
||||
```
|
||||
|
||||
### Key files
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `api/app.py` | Flask app, all REST endpoints, before-request hooks, health monitor thread |
|
||||
| `api/managers.py` | Singleton instantiation of all service managers |
|
||||
| `api/base_service_manager.py` | Abstract base class: `get_status`, `get_config`, `update_config`, `validate_config`, `test_connectivity`, `get_logs`, `restart_service` |
|
||||
| `api/config_manager.py` | Single source of truth for `cell_config.json` — all read/write goes through here |
|
||||
| `api/service_bus.py` | Pub/sub event system between managers |
|
||||
| `webui/src/services/api.js` | Axios API client — all UI→API calls |
|
||||
| `docker-compose.yml` | Container definitions and network topology |
|
||||
| `Makefile` | All operational commands |
|
||||
| `install.sh` | Bash installer served via `https://install.pic.ngo` |
|
||||
|
||||
### Directory layout
|
||||
|
||||
```
|
||||
api/ Flask API and all service managers
|
||||
webui/ React SPA (Vite + Tailwind)
|
||||
tests/ pytest unit tests (no running services required)
|
||||
tests/integration/ require a running PIC stack
|
||||
tests/e2e/ Playwright UI and WireGuard e2e tests
|
||||
config/ Runtime config per service (mostly git-ignored)
|
||||
data/ Runtime secrets and state (fully git-ignored)
|
||||
scripts/ Setup and maintenance scripts
|
||||
install.sh One-line installer entry point
|
||||
Makefile All make targets
|
||||
docker-compose.yml
|
||||
```
|
||||
|
||||
### Config and secrets
|
||||
|
||||
- Runtime config: `config/api/cell_config.json` — managed by `ConfigManager`, never edit directly
|
||||
- Secrets and user data: `data/` — git-ignored, contains `auth_users.json`, WireGuard keys, DDNS token, CA key
|
||||
- DDNS config lives under the top-level `ddns` key in `cell_config.json`, accessed via `config_manager.configs.get('ddns', {})`
|
||||
- Do not read `_identity.domain` expecting a dict — it is a plain string (the domain mode, e.g. `"pic_ngo"`)
|
||||
|
||||
### Before-request hooks (app.py)
|
||||
|
||||
Three hooks run on every request in this order:
|
||||
1. `enforce_setup` — returns 428 for all `/api/*` except `/api/setup/*` and `/health` until setup is complete. Skipped when `app.config['TESTING']` is True.
|
||||
2. `enforce_auth` — returns 401 if no session; returns 503 if users file exists but is empty (misconfiguration). Skipped when `app.config['TESTING']` is True.
|
||||
3. `check_csrf` — requires `X-CSRF-Token` header on all mutating requests except `/api/auth/*` and `/api/setup/*`.
|
||||
|
||||
---
|
||||
|
||||
## Coding Conventions
|
||||
|
||||
### Python (API)
|
||||
|
||||
- All managers inherit `BaseServiceManager` — always implement all abstract methods
|
||||
- Use `self.logger` (from `BaseServiceManager`) — never `print()` or raw `logging`
|
||||
- Config reads go through `self.config_manager` — never open `cell_config.json` directly
|
||||
- Use `threading.RLock` for shared state; managers run in a multi-threaded Flask app
|
||||
- Do not use `any` typing; be explicit
|
||||
- Keep Flask route handlers thin — business logic belongs in the manager, not in `app.py`
|
||||
- Error responses must be JSON: `jsonify({'error': '...'}), <status_code>`
|
||||
- Do not catch bare `Exception` and silently swallow it — log at minimum
|
||||
|
||||
### JavaScript (webui)
|
||||
|
||||
- All API calls go through `src/services/api.js` — never use `fetch` or a new Axios instance directly
|
||||
- Use functional components; no class components
|
||||
- Tailwind utilities only — no inline styles, no custom CSS files
|
||||
- Keep page components in `src/pages/`, reusable UI in `src/components/`
|
||||
- State: local `useState`/`useEffect` is fine; no Redux or global state library
|
||||
|
||||
### General
|
||||
|
||||
- No comments that describe *what* the code does — only *why* if non-obvious
|
||||
- No dead code, no commented-out blocks
|
||||
- No backwards-compat shims for things being removed
|
||||
- Prefer editing existing files over creating new ones
|
||||
- Tests that write to disk: mock `builtins.open` with `OSError` rather than relying on `/nonexistent/path` (CI runs as root and can create any path)
|
||||
|
||||
---
|
||||
|
||||
## Testing and Quality
|
||||
|
||||
Before considering any task complete:
|
||||
1. Run `make test` — all 1500+ unit tests must pass
|
||||
2. Fix failures before committing — the pre-commit hook will block the commit anyway
|
||||
|
||||
### Rules
|
||||
|
||||
- Use `unittest.mock` / `pytest-mock` for all Docker, filesystem, and subprocess calls
|
||||
- Tests must pass in CI (rootless environment where filesystem assumptions don't hold)
|
||||
- When testing write-failure paths, mock `builtins.open` with `side_effect=OSError` — do not rely on unwritable paths
|
||||
- Integration tests (`tests/integration/`) require a running stack — exclude from CI with `--ignore=tests/integration`
|
||||
- E2e tests (`tests/e2e/`) require Playwright — exclude from CI with `--ignore=tests/e2e`
|
||||
- Add tests for any new API endpoint, manager method, or utility function
|
||||
- Do not add tests for Flask routing boilerplate or trivial getters — test behaviour, not structure
|
||||
|
||||
---
|
||||
|
||||
## File Placement Rules
|
||||
|
||||
| New thing | Where it goes |
|
||||
|---|---|
|
||||
| New service manager | `api/<name>_manager.py`, registered in `api/managers.py` and wired into `app.py` |
|
||||
| New API endpoints | `app.py` — grouped with the relevant manager's existing endpoints |
|
||||
| New React page | `webui/src/pages/` |
|
||||
| Reusable UI component | `webui/src/components/` |
|
||||
| New pytest test file | `tests/test_<module>.py` |
|
||||
| Operational script | `scripts/` |
|
||||
| Documentation | Update `README.md`, `QUICKSTART.md`, or `Personal Internet Cell – Project Wiki.md` as appropriate |
|
||||
|
||||
Do not create a new abstraction for a single use case. Do not create near-duplicate files — edit the existing one.
|
||||
|
||||
---
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- **Never expose the Flask API port (3000) directly** — it must always be behind Caddy
|
||||
- **Never commit secrets** — `data/`, `.env`, `*.key`, `*.pem` are all git-ignored; keep it that way
|
||||
- **Do not modify `enforce_setup` or `enforce_auth` hooks** without understanding the full auth flow — these are the security boundary
|
||||
- **Do not change the `cell_config.json` schema** without updating `ConfigManager` validation and all manager reads
|
||||
- **Do not rename API route paths** without checking the webui `api.js` client and any external callers
|
||||
- **Do not modify WireGuard key generation** — losing the server private key means all peers must be re-provisioned
|
||||
- Flag any change to auth flow, CSRF logic, or session management as security-sensitive before implementing
|
||||
|
||||
---
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Stack lifecycle (always use make — never call docker/docker-compose directly)
|
||||
make start # build and start all containers
|
||||
make stop # stop all containers
|
||||
make restart # restart containers
|
||||
make status # container status + API health check
|
||||
make logs # follow all container logs
|
||||
make logs-api # follow API logs only
|
||||
make logs-caddy # follow Caddy logs
|
||||
make shell-api # shell inside the API container
|
||||
make build-api # rebuild API image after code change
|
||||
make build-webui # rebuild webui image after code change
|
||||
|
||||
# Tests
|
||||
make test # pytest tests/ --ignore=tests/e2e --ignore=tests/integration
|
||||
make test-coverage # coverage report in htmlcov/
|
||||
pytest tests/test_<module>.py -v # single test file
|
||||
|
||||
# Local dev (no Docker)
|
||||
pip install -r api/requirements.txt
|
||||
python3 api/app.py # Flask API on :3000
|
||||
|
||||
cd webui && npm install && npm run dev # React UI on :5173 (proxies /api → :3000)
|
||||
|
||||
# Peer / WireGuard
|
||||
make list-peers
|
||||
make show-routes
|
||||
|
||||
# Admin password
|
||||
make show-admin-password
|
||||
make reset-admin-password
|
||||
|
||||
# Backup / restore
|
||||
make backup
|
||||
make restore
|
||||
|
||||
# Maintenance
|
||||
make update # git pull + rebuild + restart
|
||||
make uninstall # stop containers; prompt to delete config/ and data/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Topology
|
||||
|
||||
| Machine | IP | Role |
|
||||
|---|---|---|
|
||||
| pic0 | 192.168.31.51 | Dev machine — you are here. Run all commands directly. |
|
||||
| pic1 | 192.168.31.52 | Test/staging PIC instance |
|
||||
| Gitea | 192.168.31.50 | Self-hosted git server (`gitea@192.168.31.50:roof/pic.git`) |
|
||||
| DDNS VPS | 192.168.31.101 (LAN) / 178.168.15.65 (public) | PowerDNS + FastAPI for `*.pic.ngo` DDNS |
|
||||
|
||||
The `roof` user on pic0 has passwordless sudo and is in the `docker` group — use both freely.
|
||||
|
||||
---
|
||||
|
||||
## AI Collaboration Rules
|
||||
|
||||
These rules apply to every Claude Code session in this repo:
|
||||
|
||||
- **Read memory first** — load `/home/roof/.claude/projects/-home-roof/memory/MEMORY.md` at session start; follow referenced memory files for relevant context.
|
||||
- **You are on pic0** — execute commands directly here; do not ask the user to run them.
|
||||
- **`make` is the only container interface** — never call `docker` or `docker-compose` directly. All container lifecycle goes through `make start`, `make stop`, `make build`, `make logs`, etc.
|
||||
- **Use specialized agents** — spawn `pic-remote` for VPS/pic1 SSH tasks, `pic-qa` for test writing, `pic-architect` for design decisions, `pic-designer` for UI review, `pic-devops` for docker-compose/Makefile changes, `pic-writer` for documentation.
|
||||
- **Test before commit** — run `make test` and fix all failures before staging. The pre-commit hook enforces this, but run it manually first.
|
||||
- **No skipping hooks** — never use `--no-verify` unless the only change is documentation or a workflow file with no Python/JS.
|
||||
- **Commits need context** — write commit messages that explain *why*, not just *what*. Always add the Co-Authored-By trailer.
|
||||
@@ -1,521 +0,0 @@
|
||||
# Personal Internet Cell – Project Wiki
|
||||
|
||||
## Overview
|
||||
|
||||
Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, NTP, WireGuard VPN, HTTPS reverse proxy, a certificate authority, and optional services (email, calendar/contacts, file storage, connectivity exits, and more) — all managed from a single REST API and React web UI.
|
||||
|
||||
The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture](#architecture)
|
||||
2. [Service Managers](#service-managers)
|
||||
3. [First-Run Wizard](#first-run-wizard)
|
||||
4. [Authentication](#authentication)
|
||||
5. [API Reference](#api-reference)
|
||||
6. [DDNS](#ddns)
|
||||
7. [Network Services and DNS](#network-services-and-dns)
|
||||
8. [Services UI](#services-ui)
|
||||
9. [Service Store (Add-ons)](#service-store-add-ons)
|
||||
10. [Backup and Restore](#backup-and-restore)
|
||||
11. [Cell-to-Cell Networking](#cell-to-cell-networking)
|
||||
12. [Extended Connectivity](#extended-connectivity)
|
||||
13. [Security Model](#security-model)
|
||||
14. [Testing](#testing)
|
||||
15. [Development](#development)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser / WireGuard peer
|
||||
└── Caddy (:80/:443) reverse proxy, TLS termination
|
||||
└── React SPA (:8081→8080) Vite + Tailwind (Nginx in container)
|
||||
└── Flask API (:3000) REST API, bound to 127.0.0.1
|
||||
├── NetworkManager CoreDNS, chrony (split-horizon DNS)
|
||||
├── WireGuardManager WireGuard VPN peer lifecycle
|
||||
├── PeerRegistry peer registration and trust
|
||||
├── EmailManager Postfix + Dovecot (optional service)
|
||||
├── CalendarManager Radicale CalDAV/CardDAV (optional service)
|
||||
├── FileManager WebDAV + Filegator (optional service)
|
||||
├── RoutingManager iptables NAT and routing
|
||||
├── FirewallManager iptables firewall rules
|
||||
├── VaultManager internal CA, cert lifecycle, Fernet encryption
|
||||
├── ContainerManager Docker SDK
|
||||
├── CellLinkManager cell-to-cell WireGuard links
|
||||
├── ConnectivityManager exit routing (WG ext, OpenVPN, Tor, sshuttle, proxy)
|
||||
├── DDNSManager dynamic DNS heartbeat
|
||||
├── ServiceStoreManager optional service install/remove
|
||||
├── CaddyManager Caddyfile generation and reload
|
||||
├── AuthManager session auth, RBAC
|
||||
└── SetupManager first-run wizard state
|
||||
```
|
||||
|
||||
Six core containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default): `cell-caddy`, `cell-dns`, `cell-ntp`, `cell-wireguard`, `cell-api`, `cell-webui`. Static IPs per container are defined in `docker-compose.yml`. Installed optional services join the same network via their own compose projects, managed by `ServiceComposer`.
|
||||
|
||||
The WireGuard container runs unprivileged — it uses `NET_ADMIN` only and requires the WireGuard module on the host kernel (Linux 5.6+ or a loadable module). The `cell-api` and `cell-webui` images are slim builds. The CoreDNS image is pinned to a specific digest.
|
||||
|
||||
Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.
|
||||
|
||||
---
|
||||
|
||||
## Service Managers
|
||||
|
||||
All managers inherit `BaseServiceManager` (`api/base_service_manager.py`), which provides:
|
||||
- `get_status()` — current running state
|
||||
- `get_config()` / `update_config()` — config read/write
|
||||
- `test_connectivity()` — reachability check
|
||||
- `get_logs()` — last N lines from the service log
|
||||
- `restart_service()` — container restart via Docker SDK
|
||||
|
||||
The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers (e.g., `CONFIG_CHANGED`, `SERVICE_STARTED`). Dependencies are declared in the bus (wireguard depends on network; email depends on network and vault).
|
||||
|
||||
### Manager summary
|
||||
|
||||
| Manager | Responsibilities |
|
||||
|---|---|
|
||||
| `NetworkManager` | CoreDNS zone files and split-horizon DNS, chrony NTP |
|
||||
| `WireGuardManager` | Key generation, `wg0.conf` generation, peer add/remove, route sync |
|
||||
| `PeerRegistry` | Peer registration, trust tracking, peer statistics |
|
||||
| `EmailManager` | docker-mailserver accounts, mailbox config, alias management |
|
||||
| `CalendarManager` | Radicale user/calendar/contacts lifecycle |
|
||||
| `FileManager` | WebDAV user directories, Filegator access |
|
||||
| `RoutingManager` | NAT rules, per-peer routing policy, fwmark-based exit routing |
|
||||
| `FirewallManager` | iptables INPUT/FORWARD/OUTPUT rule management |
|
||||
| `VaultManager` | Internal CA (self-signed root), TLS cert issue/revoke, Age public key |
|
||||
| `ContainerManager` | Docker container/image/volume management via SDK |
|
||||
| `CellLinkManager` | Site-to-site WireGuard links to other PIC cells, peer-sync protocol |
|
||||
| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, Tor, sshuttle, or proxy (redsocks) |
|
||||
| `DDNSManager` | Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns) |
|
||||
| `ServiceStoreManager` | Fetch manifest index, install/remove optional services |
|
||||
| `CaddyManager` | Caddyfile generation, reload-on-change |
|
||||
| `AuthManager` | bcrypt password store, session management, admin/peer RBAC |
|
||||
| `SetupManager` | First-run wizard state, setup-complete flag |
|
||||
|
||||
---
|
||||
|
||||
## First-Run Wizard
|
||||
|
||||
On first start, `SetupManager.is_setup_complete()` returns `False`. The `enforce_setup` before-request hook returns HTTP 428 for all `/api/*` requests except `/api/setup/*` and `/health`, redirecting clients to `/setup`.
|
||||
|
||||
The wizard collects:
|
||||
- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`)
|
||||
- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01`
|
||||
- **Timezone**
|
||||
- **Services to install** — optional services (email, calendar, files) to install after setup; each starts a background install via `ServiceStoreManager`
|
||||
- **Admin password** — minimum 12 characters
|
||||
|
||||
Domain modes and their TLS behavior:
|
||||
|
||||
| Mode | Certificate |
|
||||
|---|---|
|
||||
| `pic_ngo` | Wildcard Let's Encrypt cert via DNS-01 (requires accurate host clock for ACME + DDNS token) |
|
||||
| `cloudflare` | Wildcard Let's Encrypt cert via Cloudflare DNS-01 |
|
||||
| `duckdns` | Let's Encrypt via DuckDNS DNS-01 |
|
||||
| `http01` | Let's Encrypt per-subdomain cert via HTTP-01 (no wildcard; port 80 must be reachable) |
|
||||
| `lan` | Internal CA only; no internet required |
|
||||
|
||||
On completion:
|
||||
1. Admin account is created in `data/auth_users.json`
|
||||
2. Cell identity is written to `config/api/cell_config.json`
|
||||
3. Caddy config is generated
|
||||
4. If domain mode is `pic_ngo`, the cell registers `<name>.pic.ngo` with the DDNS service
|
||||
5. Each selected service is installed in a background thread
|
||||
|
||||
Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
`AuthManager` stores bcrypt-hashed credentials in `data/auth_users.json`. Two roles:
|
||||
|
||||
| Role | Access |
|
||||
|---|---|
|
||||
| `admin` | All `/api/*` endpoints except `/api/peer/*` |
|
||||
| `peer` | `/api/peer/*` only (peer dashboard, key exchange) |
|
||||
|
||||
Session auth flow:
|
||||
- `POST /api/auth/login` — creates a Flask session
|
||||
- `GET /api/auth/me` — current session info
|
||||
- `POST /api/auth/logout` — clears session
|
||||
- `POST /api/auth/change-password` — change own password
|
||||
- `POST /api/auth/admin/reset-password` — admin resets another user's password
|
||||
|
||||
CSRF protection: all `POST`, `PUT`, `DELETE`, `PATCH` on `/api/*` (except `/api/auth/*` and `/api/setup/*`) require the `X-CSRF-Token` header matching the session token, obtained via `GET /api/auth/csrf-token`.
|
||||
|
||||
Cell-to-cell peer-sync endpoints (`/api/cells/peer-sync/*`) use source-IP + WireGuard public key auth, not session cookies.
|
||||
|
||||
Auth enforcement is active once any user exists in the store. If the store is empty (fresh install before wizard), all requests bypass auth — `enforce_setup` already blocks them with 428.
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
**Base URL:** `http://localhost:3000`
|
||||
**Auth:** session cookie (`X-CSRF-Token` header required for mutations)
|
||||
|
||||
### Core
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| GET | `/health` | Health check (always public) |
|
||||
| GET | `/api/status` | All-service status summary |
|
||||
| GET | `/api/config` | Full cell config |
|
||||
| PUT | `/api/config` | Update cell config |
|
||||
| GET | `/api/health/history` | Recent health check history |
|
||||
|
||||
### Auth (`/api/auth/`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| POST | `/api/auth/login` | Create session |
|
||||
| POST | `/api/auth/logout` | Destroy session |
|
||||
| GET | `/api/auth/me` | Current user info |
|
||||
| GET | `/api/auth/csrf-token` | Get CSRF token |
|
||||
| POST | `/api/auth/change-password` | Change own password |
|
||||
| POST | `/api/auth/admin/reset-password` | Admin: reset another user's password |
|
||||
| GET | `/api/auth/users` | Admin: list users |
|
||||
|
||||
### Setup (`/api/setup/`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/setup/status` | Setup complete flag + current step |
|
||||
| GET | `/api/setup/step` | Current wizard step data |
|
||||
| POST | `/api/setup/step` | Submit current step |
|
||||
| POST | `/api/setup/complete` | Finalize setup |
|
||||
|
||||
### Network Services (`/api/dns/`, `/api/ntp/`, `/api/network/`)
|
||||
|
||||
DNS overview (effective domain, public records, internal records, per-mode actions), NTP status, network connectivity test.
|
||||
|
||||
### WireGuard (`/api/wireguard/`, `/api/peers/`)
|
||||
|
||||
Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.
|
||||
|
||||
### Email (`/api/email/`) _(available when email service is installed)_
|
||||
|
||||
User account management, mailbox config, alias management, connectivity test. Returns HTTP 404 when the email service is not installed (except `/api/email/status`).
|
||||
|
||||
### Calendar (`/api/calendar/`) _(available when calendar service is installed)_
|
||||
|
||||
User, calendar, and contacts (CardDAV) management. Returns HTTP 404 when the calendar service is not installed (except `/api/calendar/status`).
|
||||
|
||||
### Files (`/api/files/`) _(available when files service is installed)_
|
||||
|
||||
WebDAV user management, file upload/download/delete, folder management. Returns HTTP 404 when the files service is not installed (except `/api/files/status`).
|
||||
|
||||
### Routing (`/api/routing/`)
|
||||
|
||||
NAT rules, peer routes, exit node configuration.
|
||||
|
||||
### Vault (`/api/vault/`)
|
||||
|
||||
Certificate issue/revoke, CA certificate, trust key management, Age public key.
|
||||
|
||||
### Containers (`/api/containers/`)
|
||||
|
||||
List, start, stop, inspect containers; manage images and volumes.
|
||||
|
||||
### Cell Network (`/api/cells/`)
|
||||
|
||||
List connected cells, add/remove cell links, peer-sync.
|
||||
|
||||
### Connectivity (`/api/connectivity/`)
|
||||
|
||||
List exit nodes, configure WireGuard external / OpenVPN / Tor / sshuttle / proxy exits, assign per-peer exit policy, assign per-service egress.
|
||||
|
||||
### Service Store (`/api/store/`)
|
||||
|
||||
List available services, install, remove.
|
||||
|
||||
### Logs (`/api/logs/`)
|
||||
|
||||
Per-service log retrieval, log search, log statistics.
|
||||
|
||||
---
|
||||
|
||||
## DDNS
|
||||
|
||||
`DDNSManager` maintains a `<cell-name>.pic.ngo` DNS A record pointing at the cell's public IP. A background thread runs every 5 minutes and calls `provider.update(token, ip)` only when the IP changes.
|
||||
|
||||
Registration happens during the setup wizard (if domain mode is `pic_ngo`) via `provider.register(name, ip)`, which returns a bearer token stored in `data/api/.ddns_token`.
|
||||
|
||||
DDNS config lives in `cell_config.json` under the top-level `ddns` key:
|
||||
|
||||
```json
|
||||
{
|
||||
"ddns": {
|
||||
"provider": "pic_ngo",
|
||||
"api_base_url": "https://ddns.pic.ngo",
|
||||
"totp_secret": "<base32 secret>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Registration requires a time-based OTP (`X-Register-OTP` header) derived from the shared `REGISTER_TOTP_SECRET` on the DDNS server. This prevents unauthorized subdomain registration.
|
||||
|
||||
Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`.
|
||||
|
||||
---
|
||||
|
||||
## Network Services and DNS
|
||||
|
||||
### Split-horizon DNS
|
||||
|
||||
PIC operates a split-horizon DNS configuration for the cell domain.
|
||||
|
||||
- **Outside the VPN** — the cell domain (e.g. `myhome.pic.ngo`) resolves to the public IP via the DDNS provider.
|
||||
- **Inside the VPN** — CoreDNS answers the same cell domain with the WireGuard internal IP of `cell-caddy`. Traffic therefore flows through the WireGuard tunnel and Caddy serves it on both the public and WireGuard interface.
|
||||
|
||||
`NetworkManager.update_split_horizon_zone()` writes a zone file for the effective domain and reloads CoreDNS via SIGUSR1 whenever the cell name or domain mode changes.
|
||||
|
||||
### Network Services page
|
||||
|
||||
The **Network Services** page (`/network`) shows a provider-aware DNS overview:
|
||||
- Current domain mode label and effective domain
|
||||
- Public DNS records (A records registered with the DDNS provider)
|
||||
- Service subdomains (shown for `pic_ngo` and `duckdns` modes)
|
||||
- Internal records served by CoreDNS on the WireGuard network
|
||||
- Per-mode action buttons (e.g. force-refresh DDNS, reload CoreDNS)
|
||||
- NTP status (running/stopped, current time source)
|
||||
|
||||
DHCP has been removed from PIC. There is no `cell-dhcp` container and no DHCP configuration in the UI. The `cell-dns` container runs CoreDNS only.
|
||||
|
||||
### NTP
|
||||
|
||||
The `cell-ntp` container runs chrony in a pinned Alpine image. It provides NTP to WireGuard peers that configure the cell as their NTP server. The installer also enables host-level chrony to keep the host clock accurate for ACME certificate issuance and DDNS TOTP tokens.
|
||||
|
||||
---
|
||||
|
||||
## Backup and Restore
|
||||
|
||||
### What `make backup` captures
|
||||
|
||||
`make backup` creates `backups/cell-backup-<timestamp>.tar.gz` containing:
|
||||
- `config/` — all service configuration including `cell_config.json`
|
||||
- `data/` excluding logs (`data/logs/`) and internal config-backup snapshots (`data/api/config_backups/`)
|
||||
- `docker-compose.yml` and `Makefile`
|
||||
|
||||
The archive is written mode `0600`. It contains key material (WireGuard keys, internal CA, vault fernet.key, admin credentials, DDNS token, cell links, Caddy ACME certs). Store it securely.
|
||||
|
||||
Data volumes of installed store services (email mailboxes, calendar collections, file trees) are **not** included in `make backup`. They are captured by the API-driven backup described below.
|
||||
|
||||
### API-driven backup (`POST /api/config/backup`)
|
||||
|
||||
The API backup captures everything `make backup` does, plus:
|
||||
- `.env`
|
||||
- The Caddyfile and Corefile (runtime-generated)
|
||||
- DNS zone files
|
||||
- WireGuard key material and live peer configs
|
||||
- Vault directory (CA, certificates, fernet.key, trust store)
|
||||
- Per-service connectivity configs (sshuttle keys, redsocks config, OpenVPN configs, WireGuard external config)
|
||||
- Auth users, Flask secret key, DDNS token, cell links, peer service credentials
|
||||
- Caddy issued ACME certificates and ACME state
|
||||
- Live service data volumes (streamed via `docker exec tar`)
|
||||
|
||||
**Passphrase encryption**: pass `{"passphrase": "..."}` in the request body to encrypt the archive. The encrypted file is named `<backup_id>.tar.gz.age` and uses Fernet with an scrypt-derived key. The plaintext staging directory is removed immediately after encryption. Supply the same passphrase when calling `POST /api/config/restore/<backup_id>`.
|
||||
|
||||
Both backup and encrypted archive files are written mode `0600`.
|
||||
|
||||
### Restore
|
||||
|
||||
**From `make backup`:**
|
||||
|
||||
```bash
|
||||
tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz
|
||||
make restart
|
||||
```
|
||||
|
||||
**From API backup:**
|
||||
|
||||
Call `POST /api/config/restore/<backup_id>` (with `{"passphrase": "..."}` for encrypted archives). The restore process:
|
||||
1. Restores the vault first (fernet.key must be present before any encrypted secrets are read)
|
||||
2. Restores identity, `.env`, WireGuard key material, cell links
|
||||
3. Restores Caddy ACME certs, Caddyfile, Corefile, DNS zones
|
||||
4. Restores connectivity configs, auth users, DDNS token
|
||||
5. Restores service user account files
|
||||
6. Reloads `cell_config.json` into memory
|
||||
7. Restores live service data volumes (if service containers are running)
|
||||
8. Calls `_reapply_runtime_state()` — regenerates Caddyfile and Corefile from the restored config and re-applies routing rules
|
||||
|
||||
After an API restore, run `make restart` to ensure all containers pick up the restored configuration.
|
||||
|
||||
---
|
||||
|
||||
## Services UI
|
||||
|
||||
### Navigation
|
||||
|
||||
The left-hand navigation contains a **Services** group. Both admin and peer users see it. Sub-items for installed services (Email, Calendar, Files, etc.) are added dynamically: the UI fetches `GET /api/services/active` on load and after each install/uninstall. Services not yet installed do not appear in the nav.
|
||||
|
||||
Legacy paths redirect to their new canonical locations:
|
||||
|
||||
| Old path | New path |
|
||||
|---|---|
|
||||
| `/email` | `/services/email` |
|
||||
| `/calendar` | `/services/calendar` |
|
||||
| `/files` | `/services/files` |
|
||||
| `/store` | `/services` |
|
||||
|
||||
### Services page (`/services`)
|
||||
|
||||
A single unified catalog of all available services from the store index. Each card shows:
|
||||
- Service name, description, version
|
||||
- **Install** button (not installed) or **Uninstall** button (installed)
|
||||
- **Open** link for installed services (navigates to the service sub-page)
|
||||
- Running/stopped status dot for installed services
|
||||
|
||||
The `pic-services-changed` custom DOM event is dispatched after install/uninstall, causing the nav to re-fetch active services immediately.
|
||||
|
||||
### Service sub-pages — admin view
|
||||
|
||||
Each sub-page at `/services/email`, `/services/calendar`, and `/services/files` shows:
|
||||
|
||||
1. **Connection info** — hostnames, ports, and protocol details (e.g. IMAP/SMTP/Webmail, CalDAV/CardDAV, WebDAV/Filegator).
|
||||
2. **Service status** — current running state fetched from the API.
|
||||
3. **Users list** — accounts registered with that service.
|
||||
4. **Inline config form** — editable fields for that service's settings.
|
||||
|
||||
If the service is not installed, the page shows a `ServiceNotInstalledBanner` with a link to the catalog for admins, or a "contact your admin" message for peer users. All non-status API routes for uninstalled services return HTTP 404.
|
||||
|
||||
Config forms save automatically with an 800 ms debounce after the last change.
|
||||
|
||||
### Service sub-pages — peer view
|
||||
|
||||
Peers access the same URLs. The peer view shows only:
|
||||
|
||||
- Connection info (hostnames, ports, copy buttons).
|
||||
- Personal credentials for that service, fetched from `/api/peer/*`.
|
||||
|
||||
The config form and users list are not shown to peers.
|
||||
|
||||
### Settings page
|
||||
|
||||
The Email, Calendar, and Files configuration forms have been removed from the Settings page. Settings now covers: Identity, DDNS, Network (DNS, NTP), WireGuard, Routing & Firewall, Vault & Trust, and Backup & Restore.
|
||||
|
||||
### Relevant API endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/services/active` | List installed services with id, name, subdomain, capabilities |
|
||||
| GET | `/api/config` | Full cell config, includes `installed_services` dict |
|
||||
|
||||
---
|
||||
|
||||
## Service Store (Add-ons)
|
||||
|
||||
Email, calendar, and file storage are store services — not part of the core stack. All optional functionality ships through this mechanism.
|
||||
|
||||
`ServiceStoreManager` fetches a manifest index from `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest (`schema_version: 3`) declares:
|
||||
|
||||
- Container image and compose template
|
||||
- Caddy subdomain routes
|
||||
- Capabilities: `has_subdomain`, `has_accounts`, `has_admin_config`, `has_storage`, `has_egress`
|
||||
- Account provisioning interface (`accounts.manager`)
|
||||
- Backup declarations (`backup.volumes`, `backup.config_paths`)
|
||||
- Egress routing policy (`egress.allowed`)
|
||||
- Per-peer connection info template (`peer_config_template`)
|
||||
|
||||
`POST /api/store/install` fetches the manifest and compose template, validates them, renders the template with PIC-specific variables (`${PIC_DOMAIN}`, `${PIC_DATA_DIR}`, etc.), writes a per-service compose file, and brings the containers up via `ServiceComposer`. Caddy routes and DNS entries are applied automatically.
|
||||
|
||||
`POST /api/store/remove` checks for dependent services, stops and removes containers, and regenerates Caddy.
|
||||
|
||||
**`ServiceComposer`** (`api/service_composer.py`) manages the per-service compose lifecycle independently of the main stack. Each service gets its own compose project at `data/services/<id>/docker-compose.yml`. On startup, `reapply_active_services()` brings up containers for all recorded installs.
|
||||
|
||||
See `docs/service-developer-guide.md` for the full manifest schema reference and submission process.
|
||||
|
||||
---
|
||||
|
||||
## Cell-to-Cell Networking
|
||||
|
||||
`CellLinkManager` manages WireGuard site-to-site tunnels between PIC cells. Each link is a WireGuard peer configured with a dedicated `/32` address and allowed-IPs covering the remote cell's subnet.
|
||||
|
||||
The peer-sync protocol (`/api/cells/peer-sync/`) exchanges public keys and allowed networks between cells using source-IP + WireGuard public key authentication (no session required).
|
||||
|
||||
Access control is per-service (calendar, files, mail, WebDAV) and enforced at the iptables level.
|
||||
|
||||
---
|
||||
|
||||
## Extended Connectivity
|
||||
|
||||
`ConnectivityManager` provides per-peer and per-service exit routing: traffic from a specific WireGuard peer (or a specific installed service) can be routed through an alternate exit instead of the cell's default gateway.
|
||||
|
||||
All exit types are optional store services installed from the Services catalog. Each exit type corresponds to a store service ID:
|
||||
|
||||
| Exit type | Store service | Mechanism |
|
||||
|---|---|---|
|
||||
| `wireguard_ext` | `wireguard-ext` | WireGuard client tunnel to an external server; iface `wg_ext0`; fwmark `0x10`, table 110 |
|
||||
| `openvpn` | `openvpn-client` | OpenVPN client tunnel; iface `tun0`; fwmark `0x20`, table 120 |
|
||||
| `tor` | `tor` | Transparent proxy → Tor SOCKS on port 9040; fwmark `0x30`, table 130 |
|
||||
| `sshuttle` | `sshuttle` | SSH tunnel via sshuttle to any SSH server; transparent proxy on port 12300; fwmark `0x40`, table 140 |
|
||||
| `proxy` | `proxy` | HTTP or SOCKS5 upstream proxy via redsocks transparent redirection on port 12345; fwmark `0x50`, table 150 |
|
||||
|
||||
Routing uses fwmark and `ip rule` / `ip route` in separate routing tables inside the `cell-wireguard` container. A kill-switch FORWARD rule drops traffic for configured exits if the exit container is not active.
|
||||
|
||||
Configuration is via `PUT /api/connectivity/peers/<peer_name>/exit`. Service-level egress is declared in the service manifest's `egress` block and configured via the Connectivity page in the UI.
|
||||
|
||||
---
|
||||
|
||||
## Security Model
|
||||
|
||||
- **No open ports for the API** — Flask API binds to `127.0.0.1:3000` only; Caddy proxies HTTPS requests to it.
|
||||
- **Session auth** — bcrypt passwords, Flask server-side sessions, CSRF double-submit.
|
||||
- **Setup wizard gate** — all `/api/*` requests return 428 until setup is complete.
|
||||
- **Role separation** — admin cannot access peer endpoints; peer cannot access admin endpoints.
|
||||
- **HTTPS everywhere** — Caddy handles TLS; internal services are reached via reverse proxy paths.
|
||||
- **Internal CA** — VaultManager issues certificates for services that don't use Let's Encrypt.
|
||||
- **Docker socket isolation** — the Docker socket is mounted only into `cell-api`; other containers have no Docker access.
|
||||
- **iptables firewall** — FirewallManager manages INPUT/FORWARD rules; WireGuard peer isolation is enforced at the packet level.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
make test # unit tests (pytest, ~1900+ functions)
|
||||
make test-coverage # coverage report in htmlcov/
|
||||
```
|
||||
|
||||
Test layout:
|
||||
- `tests/` — unit and endpoint tests; no running services required
|
||||
- `tests/integration/` — require a running PIC stack
|
||||
- `tests/e2e/` — Playwright UI tests and WireGuard integration tests
|
||||
|
||||
CI: Gitea Actions runs `pytest tests/ --ignore=tests/e2e --ignore=tests/integration` on every push.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Full stack in Docker
|
||||
make start
|
||||
make stop
|
||||
make logs
|
||||
|
||||
# Flask API without Docker (port 3000)
|
||||
pip install -r api/requirements.txt
|
||||
python api/app.py
|
||||
|
||||
# React UI dev server (port 5173, proxies /api → :3000)
|
||||
cd webui && npm install && npm run dev
|
||||
|
||||
# Rebuild containers after code change
|
||||
make build-api
|
||||
make build-webui
|
||||
```
|
||||
|
||||
Key files:
|
||||
- `api/app.py` — Flask app, blueprint registration, before-request hooks, health monitor thread
|
||||
- `api/managers.py` — singleton instantiation of all service managers
|
||||
- `api/base_service_manager.py` — abstract base class all managers implement
|
||||
- `api/config_manager.py` — `cell_config.json` read/write/validate/backup
|
||||
- `api/service_bus.py` — pub/sub event system
|
||||
- `webui/src/services/api.js` — Axios API client used by all UI pages
|
||||
- `docker-compose.yml` — container definitions and network topology
|
||||
- `Makefile` — all operational commands
|
||||
-255
@@ -1,255 +0,0 @@
|
||||
# Quick Start
|
||||
|
||||
This guide walks through a first-time PIC installation on a clean Linux host.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Linux x86-64 host — Debian, Ubuntu, Fedora, RHEL, or Alpine
|
||||
- 2 GB+ RAM, 10 GB+ disk
|
||||
- Always-required ports: 53, 80, 443, 51820/udp
|
||||
- Email service only (when installed): 25, 587, 993
|
||||
- WireGuard kernel module available on the host (`modprobe wireguard`); required — userspace WireGuard is not supported
|
||||
|
||||
The installer handles all software dependencies (git, docker, make, etc.) automatically.
|
||||
|
||||
---
|
||||
|
||||
## Option A — One-line installer (recommended)
|
||||
|
||||
```bash
|
||||
curl -fsSL https://install.pic.ngo | sudo bash
|
||||
```
|
||||
|
||||
Always review the script before running it:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://install.pic.ngo | less
|
||||
```
|
||||
|
||||
The installer runs 7 steps and prints clean one-line progress for each. Run with `PIC_DEBUG=1` or `--debug` for full verbose output. A complete log is always written to `/var/log/pic-install.log`.
|
||||
|
||||
The installer:
|
||||
1. Detects your OS and installs Docker, git, make via the system package manager
|
||||
2. Installs and starts host NTP (chrony) — required for ACME certificate issuance and DDNS token registration
|
||||
3. Creates a `pic` system user and adds it to the `docker` group
|
||||
4. Clones the repository to `/opt/pic`
|
||||
5. Runs `make install` — generates keys and config, writes a systemd unit. The admin password is printed once here; it does not appear again.
|
||||
6. Runs `make start-core` to bring up the six core containers
|
||||
7. Enables the `pic` systemd unit so the stack starts on reboot, then waits for the API health check
|
||||
|
||||
When it finishes, open the URL it prints:
|
||||
|
||||
```
|
||||
http://<host-ip>:8081/setup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Option B — Manual install
|
||||
|
||||
Use this if you want to control where PIC is installed, or if you are installing on a machine that already has Docker.
|
||||
|
||||
```bash
|
||||
git clone https://git.pic.ngo/roof/pic.git pic
|
||||
cd pic
|
||||
sudo make install
|
||||
make start-core
|
||||
```
|
||||
|
||||
Then open `http://<host-ip>:8081/setup` in a browser.
|
||||
|
||||
Note: install host NTP before running `make install` if you plan to use `pic_ngo` domain mode. The installer does this automatically in Option A.
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y chrony && sudo systemctl enable --now chrony
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Complete the setup wizard
|
||||
|
||||
The setup wizard appears automatically on first start. All API requests redirect to `/setup` until it is finished.
|
||||
|
||||
The wizard asks for:
|
||||
|
||||
- **Cell name** — used for hostnames and DDNS subdomain. Lowercase letters, digits, hyphens, 2–31 characters. Example: `myhome`.
|
||||
- **Domain mode** — how HTTPS certificates are issued:
|
||||
- `pic_ngo` — automatic `<cell-name>.pic.ngo` subdomain with a wildcard Let's Encrypt cert via DNS-01 (recommended for internet-facing cells; requires accurate host clock)
|
||||
- `cloudflare` — wildcard Let's Encrypt cert via Cloudflare DNS-01 (bring your own domain)
|
||||
- `duckdns` — Let's Encrypt via DuckDNS DNS-01
|
||||
- `http01` — Let's Encrypt via HTTP-01 (no wildcard; cell must be reachable on port 80)
|
||||
- `lan` — internal CA only, no internet required (for LAN-only installs)
|
||||
- **Timezone**
|
||||
- **Services to install** — email, calendar, files (optional; installed in the background after setup completes; can be added later via the Services store page)
|
||||
- **Admin password** — minimum 12 characters, must contain uppercase, lowercase, and a digit
|
||||
|
||||
Click **Complete Setup**. The wizard creates the admin account, writes cell identity to `config/api/cell_config.json`, and redirects to the login page. Any services you selected begin installing in the background.
|
||||
|
||||
---
|
||||
|
||||
## Log in
|
||||
|
||||
After the wizard you are redirected to `/login`.
|
||||
|
||||
- **Username:** `admin`
|
||||
- **Password:** the password you set in the wizard
|
||||
|
||||
---
|
||||
|
||||
## Add a WireGuard peer
|
||||
|
||||
Go to **Peers** in the sidebar.
|
||||
|
||||
1. Click **Add Peer**.
|
||||
2. Enter a peer name (e.g. `laptop`).
|
||||
3. The API generates a key pair and assigns the next available VPN IP automatically.
|
||||
4. Click the QR code icon to display the peer configuration as a QR code.
|
||||
5. Scan the QR code with a WireGuard client (Android, iOS, or the WireGuard desktop app).
|
||||
|
||||
Once connected, `*.cell` names resolve through the cell's CoreDNS and traffic can be routed through the cell.
|
||||
|
||||
---
|
||||
|
||||
## Installing and managing services
|
||||
|
||||
Email, calendar, and file storage are optional services installed from the built-in service store. They are not running by default.
|
||||
|
||||
**To install a service:**
|
||||
|
||||
1. Go to **Services** in the sidebar.
|
||||
2. Find the service card (Email, Calendar, Files, or any other listed service).
|
||||
3. Click **Install**. PIC fetches the manifest, starts the container, and wires up DNS and Caddy routes automatically.
|
||||
4. The service appears in the sidebar navigation once installation completes.
|
||||
|
||||
**To check service status:**
|
||||
|
||||
The Services page shows each installed service as "running" or "stopped". You can also check via the API:
|
||||
|
||||
```bash
|
||||
curl -s http://<host-ip>:3000/api/services/active
|
||||
```
|
||||
|
||||
**To uninstall a service:**
|
||||
|
||||
Click **Uninstall** on the service card. The container is stopped and removed. Data in `data/services/<id>/` is kept on disk unless you delete it manually.
|
||||
|
||||
---
|
||||
|
||||
## Day-to-day operations
|
||||
|
||||
```bash
|
||||
# Check container status and API health
|
||||
make status
|
||||
|
||||
# Follow logs from all services
|
||||
make logs
|
||||
|
||||
# Follow logs from a single service
|
||||
make logs-api
|
||||
make logs-wireguard
|
||||
make logs-caddy
|
||||
|
||||
# Open a shell inside a container
|
||||
make shell-api
|
||||
make shell-dns
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup and restore
|
||||
|
||||
```bash
|
||||
make backup # archives config/ and data/ into backups/cell-backup-<timestamp>.tar.gz
|
||||
make restore # list available backups
|
||||
```
|
||||
|
||||
The backup archive is written mode `0600`. It contains secrets and key material — WireGuard keys, the internal CA, vault keys, admin credentials, DDNS token, cell links, and Caddy certificates. Store it securely.
|
||||
|
||||
Data volumes for installed store services (email mailboxes, calendar data, file storage) are captured separately via the API-driven backup (`POST /api/config/backup`), which also supports an optional passphrase for encryption at rest. The encrypted file is named `<backup_id>.tar.gz.age`.
|
||||
|
||||
To restore from a `make backup` archive:
|
||||
|
||||
```bash
|
||||
tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz
|
||||
make restart
|
||||
```
|
||||
|
||||
After restore, the API re-generates the Caddyfile and Corefile from the restored config and re-applies routing rules automatically.
|
||||
|
||||
---
|
||||
|
||||
## Updating PIC
|
||||
|
||||
```bash
|
||||
make update # git pull + rebuild + restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Uninstalling
|
||||
|
||||
```bash
|
||||
make uninstall # stops containers; prompts to also delete config/ and data/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Containers not starting
|
||||
|
||||
```bash
|
||||
make logs
|
||||
make logs-api
|
||||
```
|
||||
|
||||
Look for errors about missing config files or port conflicts.
|
||||
|
||||
### Port 53 already in use
|
||||
|
||||
On Ubuntu and Debian, `systemd-resolved` listens on port 53. Disable it:
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now systemd-resolved
|
||||
sudo rm /etc/resolv.conf
|
||||
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
|
||||
```
|
||||
|
||||
Then run `make start` again.
|
||||
|
||||
### WireGuard container fails to start
|
||||
|
||||
The WireGuard container runs unprivileged (NET_ADMIN only, no privileged mode). It requires the host kernel's WireGuard module — either compiled in (Linux 5.6+) or loadable.
|
||||
|
||||
```bash
|
||||
sudo modprobe wireguard
|
||||
```
|
||||
|
||||
On minimal installs you may need `wireguard-tools` and the kernel headers for the running kernel. On kernels that lack a builtin WireGuard module, check your distro's `wireguard-dkms` or `wireguard-linux-compat` package.
|
||||
|
||||
### API returns 428 and redirects to /setup
|
||||
|
||||
The first-run wizard has not been completed. Open `http://<host-ip>:8081` and finish the wizard.
|
||||
|
||||
### API returns 401 / UI shows "Not authenticated"
|
||||
|
||||
Your session expired or you have not logged in. Go to `http://<host-ip>:8081/login`.
|
||||
|
||||
### API returns 503 "Authentication not configured"
|
||||
|
||||
The auth file exists but contains no accounts. To recover:
|
||||
|
||||
```bash
|
||||
make reset-admin-password
|
||||
```
|
||||
|
||||
This generates a new admin password and prints it.
|
||||
|
||||
### Forgot the admin password
|
||||
|
||||
```bash
|
||||
make show-admin-password # print current password
|
||||
make reset-admin-password # generate a new random password
|
||||
```
|
||||
@@ -25,7 +25,7 @@ The Flask API (`api/app.py`) contains REST endpoints and a background health-mon
|
||||
|
||||
The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls go through `src/services/api.js` (Axios).
|
||||
|
||||
**Web UI pages:** Dashboard, Peers, Network Services, WireGuard, Email, Calendar, Files, Routing, Vault, Containers, Cell Network, Connectivity, Service Store, Logs, Settings.
|
||||
**Web UI pages:** Dashboard, Peers, Network Services, WireGuard, Connectivity (tunnels, proxies, SSH, Tor, cells, assignments), Services (store catalog + per-service pages), Routing, Vault, Containers, Activity, Logs, Settings — plus peer-facing My Services and Account pages.
|
||||
|
||||
---
|
||||
|
||||
@@ -35,9 +35,9 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g
|
||||
- **Session-based auth** — admin and peer roles. All `/api/*` endpoints require an authenticated session after setup. CSRF protection on all state-changing requests.
|
||||
- **WireGuard VPN** — peer lifecycle management, automatic key generation, QR code config export, per-peer routing policy.
|
||||
- **Caddy HTTPS** — automatic TLS via Let's Encrypt (DNS-01 or HTTP-01) or an internal CA, depending on domain mode.
|
||||
- **DDNS (pic.ngo)** — registers a `<cell-name>.pic.ngo` subdomain. Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`. A background thread re-publishes the public IP every 5 minutes.
|
||||
- **Service store** — install/remove optional third-party services from the `pic-services` index at `git.pic.ngo`. Manifests declare container images, Caddy routes, and iptables rules.
|
||||
- **Extended connectivity** — per-peer egress routing through alternate exits: WireGuard external, OpenVPN, Tor, sshuttle (SSH tunnel), or proxy (HTTP/SOCKS5 via redsocks). Exit nodes are optional store services. Per-service egress policy is also supported. Routing uses fwmark and `ip rule` in the WireGuard container.
|
||||
- **DDNS (pic.ngo)** — registers a `<cell-name>.pic.ngo` subdomain. Supported providers: `pic_ngo`, `cloudflare`, `duckdns`. A background thread re-publishes the public IP every 5 minutes.
|
||||
- **Service store** — install/remove optional third-party services from the `pic-services` index at `git.pic.ngo`. Manifests declare container images, Caddy routes, and iptables rules. Store images are digest-pinned and cosign-signed by the build pipeline; the cell verifies signatures before starting a container (enforced by default).
|
||||
- **Extended connectivity** — named connection instances per exit type: WireGuard external, OpenVPN, Tor, sshuttle (SSH tunnel), or proxy (HTTP/SOCKS5 via redsocks), plus cell-relay through another cell. Peers are assigned per-peer to a connection with configurable fail-open/fail-closed; per-connection health is tracked. Per-service egress policy is also supported. Routing uses per-instance fwmarks and `ip rule` in the WireGuard container.
|
||||
- **Cell-to-cell networking** — WireGuard-based site-to-site links between PIC cells with service-level access control (calendar, files, mail, WebDAV) and a peer-sync protocol.
|
||||
- **Certificate authority** — `vault_manager` issues and revokes TLS certificates for internal services.
|
||||
- **Network services** — CoreDNS (`.cell` TLD and split-horizon DNS for the cell domain), chrony NTP.
|
||||
@@ -48,6 +48,8 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g
|
||||
- **Container manager** — start/stop/inspect containers, pull images, manage volumes via the Docker SDK.
|
||||
- **Firewall manager** — iptables rule management (`firewall_manager.py`).
|
||||
- **Structured logging** — JSON logs with rotation (5 MB / 5 backups per service), log search, and per-service verbosity control.
|
||||
- **Audit log** — append-only, hash-chained change log of all admin actions, with CSV export and an Activity page in the UI.
|
||||
- **Backup / restore** — full backup of config, secrets, key material, and live service data volumes, with optional passphrase encryption; ordered restore with automatic runtime reapply.
|
||||
|
||||
---
|
||||
|
||||
@@ -61,9 +63,15 @@ The React frontend (`webui/`) is built with Vite + Tailwind CSS. All API calls g
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
Full documentation lives in the [project wiki](https://git.pic.ngo/roof/pic/wiki) — installation walkthrough, admin guide (setup, domains/TLS, services, connectivity, peers, backup, logging/audit, troubleshooting), user guide, and developer documentation (architecture, API reference, building store services, testing).
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
See [QUICKSTART.md](QUICKSTART.md) for step-by-step instructions.
|
||||
See the wiki's [Setup and First Run](https://git.pic.ngo/roof/pic/wiki/Admin-Setup) for step-by-step instructions.
|
||||
|
||||
The short version — one-line installer (recommended):
|
||||
|
||||
@@ -212,4 +220,4 @@ make reset-admin-password # generate and set a new random admin password
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
MIT.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1016,15 +1016,15 @@ class ConfigManager:
|
||||
# enforce — refuse to start a service whose image is undigested,
|
||||
# unsigned, or whose signature does not verify
|
||||
#
|
||||
# Default is "warn" until the publish pipeline signs all store images; a
|
||||
# later phase flips the default to "enforce". The section is backed up and
|
||||
# restored with the rest of cell_config.json automatically.
|
||||
# All store images are now signed + digest-pinned via the publish pipeline,
|
||||
# so the default is "enforce". The section is backed up and restored with
|
||||
# the rest of cell_config.json automatically.
|
||||
|
||||
def get_image_verification(self) -> Dict[str, Any]:
|
||||
"""Return the image verification config, e.g. {'mode': 'warn'}."""
|
||||
"""Return the image verification config, e.g. {'mode': 'enforce'}."""
|
||||
cfg = self.configs.get('image_verification')
|
||||
if not isinstance(cfg, dict) or cfg.get('mode') not in _IMAGE_VERIFY_MODES:
|
||||
cfg = {'mode': 'warn'}
|
||||
cfg = {'mode': 'enforce'}
|
||||
self.configs['image_verification'] = cfg
|
||||
return dict(cfg)
|
||||
|
||||
|
||||
@@ -285,7 +285,7 @@ class ServiceComposer:
|
||||
return getter()
|
||||
except Exception as e: # config corruption must not crash install
|
||||
logger.warning('service_composer: could not read verification mode: %s', e)
|
||||
return 'warn'
|
||||
return 'enforce'
|
||||
|
||||
def _cosign_verify(self, image_ref: str) -> Dict:
|
||||
"""Run `cosign verify` against the bundled public key for one image ref.
|
||||
|
||||
@@ -1,743 +0,0 @@
|
||||
# PIC Service Developer Guide
|
||||
|
||||
This guide is for developers who want to build services that integrate with Personal Internet Cell (PIC). It covers the manifest format, how PIC wires up routing, DNS, backup, and account provisioning for your service, and how to package and submit your work.
|
||||
|
||||
**Prerequisites:** you should be comfortable with Docker, Docker Compose, and basic Linux networking. You do not need to know Python to build a store service.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [What a PIC service is](#1-what-a-pic-service-is)
|
||||
2. [Manifest reference](#2-manifest-reference)
|
||||
3. [Compose template variables](#3-compose-template-variables)
|
||||
4. [Account provisioning interface](#4-account-provisioning-interface)
|
||||
5. [Backup integration](#5-backup-integration)
|
||||
6. [Egress routing](#6-egress-routing)
|
||||
7. [Quick-start example](#7-quick-start-example)
|
||||
8. [Reference implementations](#8-reference-implementations)
|
||||
9. [Submitting to the store](#9-submitting-to-the-store)
|
||||
|
||||
---
|
||||
|
||||
## 1. What a PIC service is
|
||||
|
||||
A PIC service is a Docker container (or a set of containers) that plugs into the PIC ecosystem through a single JSON file called the **manifest**. The manifest tells PIC everything it needs to know:
|
||||
|
||||
- How to route HTTPS traffic to the service through Caddy
|
||||
- What subdomains to expose
|
||||
- Which users get accounts on the service and what credentials they receive
|
||||
- Which paths to include in automated backups
|
||||
- Which outbound network interfaces the service is allowed to use
|
||||
|
||||
All PIC services are **store services** — optional packages installed by the cell admin from the `pic-services` catalog. PIC downloads the manifest, renders a per-service Docker Compose file, and starts the containers. The core PIC stack (DNS, NTP, WireGuard, Caddy, API, WebUI) runs independently of any installed services.
|
||||
|
||||
The email, calendar, and files services (in `pic-services/services/`) are the reference implementations and show the full feature set. The `ServiceRegistry` in `api/service_registry.py` is the single source of truth for all installed services. `CaddyManager`, the backup system, and the peer services endpoint all read from it rather than from hardcoded lists.
|
||||
|
||||
---
|
||||
|
||||
## 2. Manifest reference
|
||||
|
||||
The manifest is a JSON file with `"schema_version": 3`. Every field is described below. The `email`, `calendar`, and `files` manifests in `pic-services/services/` are the canonical reference examples.
|
||||
|
||||
### Top-level identity fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `schema_version` | integer | yes | Must be `3`. |
|
||||
| `id` | string | yes | Unique service identifier, lowercase, no spaces (e.g. `"notes"`). Must match the directory name for builtins, or the store index entry for store services. |
|
||||
| `name` | string | yes | Human-readable display name (e.g. `"Notes"`). |
|
||||
| `description` | string | yes | One-sentence description shown in the UI. |
|
||||
| `version` | string | yes | Semver string for the service package itself (e.g. `"1.0.0"`). |
|
||||
| `author` | string | yes | Your name or organisation. |
|
||||
| `kind` | string | yes | Must be `"store"`. |
|
||||
| `min_pic_version` | string | no | Minimum PIC version required (e.g. `"1.0"`). |
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 3,
|
||||
"id": "notes",
|
||||
"name": "Notes",
|
||||
"description": "Self-hosted Markdown notes with full-text search",
|
||||
"version": "1.0.0",
|
||||
"author": "acme",
|
||||
"kind": "store",
|
||||
"min_pic_version": "1.0"
|
||||
}
|
||||
```
|
||||
|
||||
### `capabilities`
|
||||
|
||||
A set of boolean flags that tell PIC which integrations to activate for your service.
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `has_subdomain` | bool | `false` | The service gets a subdomain and a Caddy reverse-proxy route. Requires `subdomain` and `backend`. |
|
||||
| `has_accounts` | bool | `false` | The service provisions per-peer accounts. Requires `accounts`. |
|
||||
| `has_admin_config` | bool | `false` | The service has admin-configurable fields. Requires `config_schema`. |
|
||||
| `has_storage` | bool | `false` | The service has data worth backing up. Requires `backup`. |
|
||||
| `has_egress` | bool | `false` | The admin can choose which outbound interface this service uses. Requires `egress`. |
|
||||
| `has_api_hooks` | bool | `false` | Reserved for future use; set `false`. |
|
||||
|
||||
```json
|
||||
"capabilities": {
|
||||
"has_subdomain": true,
|
||||
"has_accounts": true,
|
||||
"has_admin_config": false,
|
||||
"has_storage": true,
|
||||
"has_egress": false,
|
||||
"has_api_hooks": false
|
||||
}
|
||||
```
|
||||
|
||||
### `subdomain`, `extra_subdomains`, `backend`, `extra_backends`
|
||||
|
||||
These fields are only read when `has_subdomain` is `true`.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `subdomain` | string | yes (if `has_subdomain`) | The primary subdomain (e.g. `"notes"`). Results in `notes.<cell-domain>`. Must not collide with reserved names: `api`, `webui`, `admin`, `www`, `ns1`, `ns2`, `git`, `registry`, `install`. |
|
||||
| `extra_subdomains` | array of strings | no | Additional subdomains that point to the same backend (e.g. `["webmail"]`). |
|
||||
| `backend` | string | yes (if `has_subdomain`) | The container-name:port combination that Caddy proxies to (e.g. `"cell-notes:8080"`). Uses Docker DNS on the `cell-network`. |
|
||||
| `extra_backends` | object | no | Maps extra subdomain names to separate backends. Key is the subdomain string; value is the backend string. The email service uses this to send `webdav.*` to a different container than `files.*`. |
|
||||
|
||||
```json
|
||||
"subdomain": "notes",
|
||||
"extra_subdomains": [],
|
||||
"backend": "cell-notes:8080"
|
||||
```
|
||||
|
||||
**Validation at runtime:** `ServiceRegistry.get_caddy_routes()` validates all subdomain and backend values before passing them to CaddyManager or NetworkManager. Any entry whose `subdomain` does not match `^[a-z][a-z0-9-]{0,30}$`, whose `backend` does not match `^[A-Za-z0-9._-]+:\d{1,5}$`, or whose `subdomain` appears in the reserved list is silently skipped with a warning log. The same validation applies to `extra_subdomains` and `extra_backends` keys/values. For store services, this validation is also performed during installation by `ServiceStoreManager._validate_manifest()`.
|
||||
|
||||
### `containers`
|
||||
|
||||
Array of container names that belong to this service. Used by the UI and log viewer. For builtins this is informational; for store services PIC only manages the single container declared in the manifest.
|
||||
|
||||
```json
|
||||
"containers": ["cell-notes"]
|
||||
```
|
||||
|
||||
### `config_schema`
|
||||
|
||||
Defines admin-configurable fields for this service. When `has_admin_config` is `true`, the UI renders a settings form from this schema. PIC stores admin-saved values in `cell_config.json` and merges them with your `default` values at runtime. The merged result is available as the `config` key when `ServiceRegistry.get()` returns your service.
|
||||
|
||||
Each field is an object:
|
||||
|
||||
| Key | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `type` | string | yes | One of `"string"`, `"integer"`, `"boolean"`. |
|
||||
| `label` | string | yes | Human-readable label for the settings form. |
|
||||
| `required` | bool | no | Whether the field must have a value before the service starts. |
|
||||
| `default` | any | no | Default value used when the admin has not set one. |
|
||||
| `min` / `max` | integer | no (integer only) | Inclusive bounds for integer fields. |
|
||||
|
||||
```json
|
||||
"config_schema": {
|
||||
"port": {
|
||||
"type": "integer",
|
||||
"label": "Internal HTTP port",
|
||||
"default": 8080,
|
||||
"min": 1,
|
||||
"max": 65535
|
||||
},
|
||||
"storage_path": {
|
||||
"type": "string",
|
||||
"label": "Data directory inside container",
|
||||
"default": "/data/notes"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `peer_config_template`
|
||||
|
||||
When a peer is provisioned on this service, PIC fills this template and returns the result to the peer as their connection info. Template substitution tokens:
|
||||
|
||||
| Token | Replaced with |
|
||||
|---|---|
|
||||
| `{domain}` | The cell's public domain (e.g. `alice.pic.ngo`) |
|
||||
| `{peer.username}` | The peer's username |
|
||||
| `{peer.service_credentials.<id>.<field>}` | A credential value; `<id>` is the service `id`, `<field>` matches a name in `accounts.credentials` |
|
||||
| `{config.<key>}` | A value from the merged `config_schema` result |
|
||||
|
||||
```json
|
||||
"peer_config_template": {
|
||||
"url": "https://notes.{domain}/",
|
||||
"username": "{peer.username}",
|
||||
"password": "{peer.service_credentials.notes.password}"
|
||||
}
|
||||
```
|
||||
|
||||
### `accounts`
|
||||
|
||||
Required when `has_accounts` is `true`.
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `manager` | string | Set to `"http"` for store services — PIC will call your container's HTTP API for account operations (see section 4). The reference services (`email`, `calendar`, `files`) use internal manager names (`"email_manager"`, `"calendar_manager"`, `"file_manager"`). |
|
||||
| `credentials` | array of strings | Names of credential fields this service issues per peer. Most services use `["password"]`. The names appear in `peer_config_template` tokens. |
|
||||
|
||||
```json
|
||||
"accounts": {
|
||||
"manager": "http",
|
||||
"credentials": ["password"]
|
||||
}
|
||||
```
|
||||
|
||||
### `compose`
|
||||
|
||||
Unused at the manifest level. Compose configuration is provided via `compose-template.yml` in the service package (see section 3). Set to `null` in the manifest.
|
||||
|
||||
### `backup`
|
||||
|
||||
Required when `has_storage` is `true`. Tells PIC's backup system what to snapshot.
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `volumes` | array of objects | Container paths to stream out via `docker exec tar`. Each entry has three string fields: `container` (container name), `path` (absolute path inside the container), and `name` (archive filename stem). |
|
||||
| `config_paths` | array of strings | Paths relative to the PIC project root on the host that contain service configuration (not user data). Copied directly into the snapshot. |
|
||||
|
||||
Each entry in `volumes` produces an archive at `<name>.tar.gz` inside the snapshot. For example, `"name": "maildata"` produces `maildata.tar.gz`.
|
||||
|
||||
```json
|
||||
"backup": {
|
||||
"volumes": [
|
||||
{"container": "cell-notes", "path": "/data/notes", "name": "notes_data"}
|
||||
],
|
||||
"config_paths": ["config/notes"]
|
||||
}
|
||||
```
|
||||
|
||||
`ServiceRegistry.get_backup_plan()` aggregates these declarations across all installed services. The backup runner reads from that method rather than from any hardcoded list.
|
||||
|
||||
### `egress`
|
||||
|
||||
Required when `has_egress` is `true`. Declares which outbound network interfaces this service is permitted to use.
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `default` | string | The interface selected when the admin has not changed anything. |
|
||||
| `allowed` | array of strings | The complete set of interfaces the admin can choose from. |
|
||||
|
||||
Valid interface identifiers: `default`, `wireguard_ext`, `openvpn`, `tor`, `sshuttle`, `proxy`.
|
||||
|
||||
```json
|
||||
"egress": {
|
||||
"default": "default",
|
||||
"allowed": ["default", "wireguard_ext", "openvpn", "tor", "sshuttle", "proxy"]
|
||||
}
|
||||
```
|
||||
|
||||
How enforcement works is described in section 6.
|
||||
|
||||
### `storage`
|
||||
|
||||
Informational metadata used by the UI to show storage usage.
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `primary_path` | string | The path (relative to project root) that holds the bulk of user data. |
|
||||
| `quota_mb` | integer or null | Storage quota in megabytes; `null` means no limit. |
|
||||
|
||||
```json
|
||||
"storage": {
|
||||
"primary_path": "data/notes",
|
||||
"quota_mb": null
|
||||
}
|
||||
```
|
||||
|
||||
### Store-only manifest fields
|
||||
|
||||
Store services (where `kind` is `"store"`) have additional required fields that builtins do not use. These are validated by `ServiceStoreManager._validate_manifest()` before installation is permitted.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|---|---|---|---|
|
||||
| `image` | string | yes | Docker image to pull. Must match the pattern `git.pic.ngo/roof/*`. Images from other registries are rejected. |
|
||||
| `container_name` | string | yes | The name Docker gives the running container. |
|
||||
| `volumes` | array | no | Named volumes to mount. Each entry must have `name` (the volume name) and `mount` (the absolute path inside the container). Mounts to `/`, `/etc`, `/var`, `/proc`, `/sys`, `/dev`, `/app`, `/run`, `/boot`, and paths that are a prefix of the PIC project root are forbidden. |
|
||||
| `env` | array | no | Environment variables to pass. Each entry has `key` and `value`. Values must match `^[A-Za-z0-9._@:/+\-= ]*$`. |
|
||||
| `iptables_rules` | array | no | FORWARD ACCEPT rules PIC should install in `cell-wireguard`. Each rule must have `type: "ACCEPT"`, `dest_ip: "${SERVICE_IP}"`, an integer `dest_port` (1–65535), and an optional `proto` (`"tcp"` or `"udp"`, default `"tcp"`). The literal string `${SERVICE_IP}` is replaced with the allocated container IP at install time. |
|
||||
| `caddy_route` | object | no | If the service exposes a web UI, provide `subdomain` (must not be reserved; must match `^[a-z][a-z0-9-]{0,30}$`). PIC inserts the corresponding `reverse_proxy` directive into the Caddyfile. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Compose template variables
|
||||
|
||||
This section applies only to store services. Builtins define their containers directly in `docker-compose.yml`.
|
||||
|
||||
When you ship a store service, you include a `compose-template.yml` alongside your `manifest.json`. `ServiceComposer.render_template()` substitutes the variables below before writing the per-service `docker-compose.yml`.
|
||||
|
||||
| Variable | Syntax | Value |
|
||||
|---|---|---|
|
||||
| `${PIC_CFG_<KEY>}` | uppercase `config_schema` key | The admin-saved value for that field, or the `default` from the schema if the admin has not set it. For example, `config_schema.port` → `${PIC_CFG_PORT}`. |
|
||||
| `${PIC_SECRET_<NAME>}` | any name you choose | An auto-generated random secret produced by `secrets.token_urlsafe(24)` (~32 URL-safe base64 characters). Generated once on first install, then reused unchanged on every reconfigure. Stored per service in `data/service_secrets.json`. |
|
||||
| `${PIC_DOMAIN}` | literal | Effective domain from `ConfigManager` (e.g. `alice.pic.ngo`). |
|
||||
| `${PIC_CELL_NAME}` | literal | Cell name from the identity config (e.g. `alice`). |
|
||||
| `${PIC_SERVICE_ID}` | literal | The `id` field from the service manifest (e.g. `notes`). |
|
||||
|
||||
**Volume mounts**: Because docker compose runs inside the API container but the Docker daemon runs on the host, relative volume paths in compose templates resolve relative to the compose file's directory as seen by the HOST filesystem. To avoid path resolution surprises, prefer **named volumes** for service data (Docker manages them independently). If bind mounts are required, use absolute host paths with `${PIC_PROJECT_DIR}` once that variable is implemented, or document the expected host layout clearly.
|
||||
|
||||
Example `compose-template.yml` for a notes service:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
cell-notes:
|
||||
image: git.pic.ngo/roof/pic-notes:latest
|
||||
container_name: cell-notes
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
NOTES_PORT: "${PIC_CFG_PORT}"
|
||||
NOTES_DOMAIN: "${PIC_DOMAIN}"
|
||||
NOTES_DB_PASS: "${PIC_SECRET_DB_PASSWORD}"
|
||||
volumes:
|
||||
- notes-data:/data/notes
|
||||
networks:
|
||||
cell-network:
|
||||
ipv4_address: "${SERVICE_IP}"
|
||||
|
||||
volumes:
|
||||
notes-data:
|
||||
|
||||
networks:
|
||||
cell-network:
|
||||
external: true
|
||||
```
|
||||
|
||||
The `SERVICE_IP` variable is the IP PIC allocated from the service pool. It is always set automatically.
|
||||
|
||||
---
|
||||
|
||||
## 4. Account provisioning interface
|
||||
|
||||
This section covers two related things: the `AccountManager` class that is PIC's central credential dispatcher, and the HTTP API that store services must implement to receive account operations.
|
||||
|
||||
### How AccountManager works
|
||||
|
||||
`AccountManager` (`api/account_manager.py`) is the single entry point for all account operations across every service type. It is instantiated once in `api/managers.py` and holds references to the service managers used by the reference services (`email_manager`, `calendar_manager`, `file_manager`).
|
||||
|
||||
When a peer account is provisioned, `AccountManager`:
|
||||
|
||||
1. Looks up the service in `ServiceRegistry` and reads `accounts.manager` from the manifest.
|
||||
2. Dispatches to the appropriate internal manager method (for builtins) or to the service's HTTP API endpoint (for store services — not yet implemented; `"http"` manager support is planned).
|
||||
3. Stores the returned credentials in `data/peer_service_credentials.json` with permissions `0o600`.
|
||||
|
||||
Credentials are stored in plaintext. This is intentional: the peer credentials endpoint needs to return them verbatim for one-time client configuration. The `0o600` permission matches the pattern used for WireGuard keys and `data/service_secrets.json`.
|
||||
|
||||
The credentials file structure is:
|
||||
|
||||
```json
|
||||
{
|
||||
"<service_id>": {
|
||||
"<peer_username>": { "password": "..." }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Writes use a write-then-rename pattern (`tmp` → final path) with `os.fsync` to avoid partial-write corruption.
|
||||
|
||||
### Manifest `accounts` field
|
||||
|
||||
The `accounts` block in the manifest wires a service into `AccountManager`.
|
||||
|
||||
| Field | Type | Description |
|
||||
|---|---|---|
|
||||
| `manager` | string | Which underlying manager handles account operations. For builtins: `"email_manager"`, `"calendar_manager"`, or `"file_manager"`. |
|
||||
| `credentials` | array of strings | Names of the credential fields this service issues per peer. Most services use `["password"]`. These names are used as token keys in `peer_config_template`. |
|
||||
|
||||
```json
|
||||
"accounts": {
|
||||
"manager": "email_manager",
|
||||
"credentials": ["password"]
|
||||
}
|
||||
```
|
||||
|
||||
The `manager` value must match a key that `AccountManager` was instantiated with. If the manager name has no registered dispatch entry, `provision()` raises `ValueError` immediately.
|
||||
|
||||
### Provision flow
|
||||
|
||||
```
|
||||
POST /api/services/catalog/<service_id>/accounts
|
||||
Content-Type: application/json
|
||||
|
||||
{ "username": "alice", "password": "optional" }
|
||||
```
|
||||
|
||||
If `password` is omitted, `AccountManager` generates one with `secrets.token_urlsafe(16)`. The response on HTTP 201 is:
|
||||
|
||||
```json
|
||||
{ "service_id": "email", "username": "alice", "provisioned": true }
|
||||
```
|
||||
|
||||
The password is not echoed in the response. To retrieve stored credentials for a provisioned peer, call `GET /api/services/catalog/<id>/accounts/<username>/credentials`.
|
||||
|
||||
Internally, `AccountManager.provision(service_id, peer_username, password)`:
|
||||
|
||||
1. Resolves the service and its manager via `_resolve_service()`.
|
||||
2. Calls the appropriate `_provision_*` method, which delegates to the concrete manager:
|
||||
- `email_manager` → `create_email_user(username, domain, password)`
|
||||
- `calendar_manager` → `create_calendar_user(username, password)`
|
||||
- `file_manager` → `create_user(username, password)`
|
||||
3. Stores `{"password": "<value>"}` under `[service_id][peer_username]` in the credentials file.
|
||||
4. Returns the credential dict to the caller.
|
||||
|
||||
If the underlying manager call returns `False`, `provision()` raises `RuntimeError`. The route handler maps this to HTTP 500.
|
||||
|
||||
For email, the domain is read from the service's merged config (`svc['config']['domain']`). If that key is absent, provisioning raises `ValueError` before calling the manager.
|
||||
|
||||
### Deprovision flow
|
||||
|
||||
```
|
||||
DELETE /api/services/catalog/<service_id>/accounts/<username>
|
||||
```
|
||||
|
||||
`AccountManager.deprovision(service_id, peer_username)`:
|
||||
|
||||
1. Calls the appropriate `_deprovision_*` method on the underlying manager.
|
||||
2. Removes the peer's entry from the credentials file. If that leaves the service block empty, the service block itself is removed.
|
||||
3. Returns `True` if the underlying call succeeded.
|
||||
|
||||
The route returns HTTP 200 with `{"message": "..."}` on success, or HTTP 400 if the service does not exist or does not support accounts.
|
||||
|
||||
**Peer deletion** calls `AccountManager.deprovision_peer(peer_username)`, which iterates over every service the peer is provisioned on and calls `deprovision()` for each. Failures on individual services are logged and skipped rather than aborting the deletion — the method returns `{service_id: bool}` for every service attempted.
|
||||
|
||||
### PIC admin API endpoints for account management
|
||||
|
||||
These endpoints are in `api/routes/services.py` and `api/routes/peers.py`.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/services/catalog/<service_id>/accounts` | Return `{"service_id": "...", "accounts": ["alice", "bob"]}` — reads directly from the credentials file. |
|
||||
| `POST` | `/api/services/catalog/<service_id>/accounts` | Provision a peer account. Body: `{"username": "...", "password": "..."}` (password optional). Returns HTTP 201 with `{"service_id", "username", "provisioned": true}`. |
|
||||
| `DELETE` | `/api/services/catalog/<service_id>/accounts/<username>` | Deprovision the peer's account. Returns HTTP 200 on success, HTTP 400 if the service or username is unknown. |
|
||||
| `GET` | `/api/services/catalog/<service_id>/accounts/<username>/credentials` | Return stored credentials for one peer+service pair. Returns HTTP 404 if the peer is not provisioned on that service. Response: `{"service_id", "username", "password"}`. |
|
||||
| `GET` | `/api/peers/<peer_name>/service-credentials` | Return filled `peer_config_template` values for all services the peer is provisioned on (see below). |
|
||||
|
||||
**Admin UI:** The Email, Calendar, and Files service pages in the admin dashboard each have an **Accounts** tab. From there, admins can provision and deprovision peer accounts, and reveal stored credentials for a provisioned peer. This tab calls the same API endpoints listed above.
|
||||
|
||||
### How `peer_config_template` connects to stored credentials
|
||||
|
||||
`GET /api/peers/<peer_name>/service-credentials` is the endpoint a peer device calls during first-time setup to configure email, CalDAV, and file sync clients.
|
||||
|
||||
The route:
|
||||
|
||||
1. Calls `AccountManager.get_all_credentials(peer_name)` → `{service_id: {field: value}}`.
|
||||
2. For each service, calls `ServiceRegistry.get_peer_service_info(service_id, peer_name, domain, cred)`.
|
||||
3. `get_peer_service_info` iterates over `peer_config_template` and replaces tokens:
|
||||
- `{domain}` → effective cell domain
|
||||
- `{peer.username}` → URL-percent-encoded peer username (safe='')
|
||||
- `{peer.service_credentials.<service_id>.<field>}` → the value from stored credentials
|
||||
- `{config.<key>}` → value from the service's merged config schema
|
||||
4. Returns the filled template dict as the value for that service in the response.
|
||||
|
||||
Response shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"peer": "alice",
|
||||
"services": {
|
||||
"email": {
|
||||
"imap_host": "mail.alice.pic.ngo",
|
||||
"username": "alice@alice.pic.ngo",
|
||||
"password": "<stored>"
|
||||
},
|
||||
"files": {
|
||||
"url": "https://files.alice.pic.ngo/dav/alice/",
|
||||
"username": "alice",
|
||||
"password": "<stored>"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If a service has no `peer_config_template` in its manifest, `get_peer_service_info` returns `None` and the raw credential dict is used as the fallback.
|
||||
|
||||
### Container lifecycle routes
|
||||
|
||||
The following PIC API endpoints are available for all services (builtins and store services). These are called by the web UI and can be called directly from the PIC admin API.
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/services/catalog/<id>/status` | Return container status. Builtins query the main compose stack; store services query their own compose project. Response includes a `containers` array with one entry per container. |
|
||||
| `POST` | `/api/services/catalog/<id>/restart` | Restart the service containers. Builtins restart via the main compose stack; store services restart via their own compose project. |
|
||||
| `POST` | `/api/services/catalog/<id>/reconfigure` | Re-render the compose file from the template and re-apply with `up -d` (rolling update). Store services only — builtins are reconfigured through their own settings routes. The request body must include a `compose_template` field containing the new template content. |
|
||||
|
||||
### Store service HTTP API
|
||||
|
||||
When `accounts.manager` is `"http"`, PIC will call your container's HTTP API for account operations. **HTTP dispatch is not yet wired up in `AccountManager`** — the current dispatch table covers only `email_manager`, `calendar_manager`, and `file_manager` (used by the reference services). Implement this interface now so your service is ready when HTTP dispatch ships.
|
||||
|
||||
The base path is `/service-api/accounts` on your container's internal address. There is no authentication on this API — it is reachable only from within the `cell-network` Docker network.
|
||||
|
||||
**Create account**
|
||||
|
||||
```
|
||||
POST /service-api/accounts
|
||||
Content-Type: application/json
|
||||
|
||||
{ "username": "alice", "password": "auto-generated-by-pic" }
|
||||
```
|
||||
|
||||
PIC generates the password and passes it to your service. Return HTTP 200 with `{"ok": true}` on success. Return HTTP 400 or 409 with `{"ok": false, "error": "..."}` for expected errors (duplicate username, invalid input). Return HTTP 500 for unexpected internal errors.
|
||||
|
||||
**Delete account**
|
||||
|
||||
```
|
||||
DELETE /service-api/accounts/{username}
|
||||
```
|
||||
|
||||
Return HTTP 200 with `{"ok": true}` on success. Return HTTP 404 with `{"ok": false, "error": "not found"}` if the account does not exist.
|
||||
|
||||
**List accounts**
|
||||
|
||||
```
|
||||
GET /service-api/accounts
|
||||
```
|
||||
|
||||
Return `{"accounts": ["alice", "bob"]}` — an array of all provisioned usernames.
|
||||
|
||||
---
|
||||
|
||||
## 5. Backup integration
|
||||
|
||||
Declare `has_storage: true` in `capabilities` and fill in the `backup` block. PIC's `ServiceRegistry.get_backup_plan()` returns the combined backup declarations for all installed services. The backup runner reads from that method.
|
||||
|
||||
### Why docker exec instead of bind mounts
|
||||
|
||||
The API container only has access to `data/api/` on the host filesystem. Service data (mailboxes, calendar collections, file trees) lives in other containers' volumes. Rather than mount every service volume into the API container — which would require compose changes per service — PIC streams data using `docker exec <container> tar czf - <path>`. This works for any container on the Docker host regardless of how its volumes are configured.
|
||||
|
||||
### `volumes` entries
|
||||
|
||||
Each object in the `volumes` array describes one directory to capture:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
| `container` | Name of the running container to exec into (e.g. `"cell-notes"`). |
|
||||
| `path` | Absolute path inside that container to archive (e.g. `"/data/notes"`). |
|
||||
| `name` | Archive filename stem. PIC saves the archive as `<name>.tar.gz` under `service_data/<service_id>/` in the backup directory. |
|
||||
|
||||
A service with multiple containers or multiple data directories lists one entry per directory.
|
||||
|
||||
**Security note:** The backup commands use `docker exec -- <container> tar -C <path> -czf - .` (note the `--` separator before the container name) to prevent option injection. The container name is also validated against `^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,63}$` before the command is run.
|
||||
|
||||
### `config_paths`
|
||||
|
||||
Paths in `config_paths` are relative to the PIC project root on the host and are copied directly into the snapshot (no docker exec). Use this for configuration files the service reads at startup, not for user data.
|
||||
|
||||
### Full example
|
||||
|
||||
```json
|
||||
"backup": {
|
||||
"volumes": [
|
||||
{"container": "cell-notes", "path": "/data/notes", "name": "notes_data"}
|
||||
],
|
||||
"config_paths": ["config/notes"]
|
||||
}
|
||||
```
|
||||
|
||||
This produces one archive `notes_data.tar.gz` (streamed from the `cell-notes` container) plus a direct copy of `config/notes/` from the host.
|
||||
|
||||
### Restore
|
||||
|
||||
PIC restores each volume entry by piping the archive back via `docker exec -i -- <container> tar -C <path> -xzf -`. The `-C <path>` flag bounds extraction to the declared volume path — the same path used during backup. Archive entries are relative paths (the backup uses `tar -C <path> -czf - .`), so files land in exactly the location declared in the manifest `volumes` entry. The target container must be running at restore time.
|
||||
|
||||
---
|
||||
|
||||
## 6. Egress routing
|
||||
|
||||
When `has_egress` is `true`, the cell admin can assign a specific outbound interface to your service. PIC enforces the selection using `fwmark` rules and policy routing in the `cell-wireguard` container via the `ConnectivityManager`.
|
||||
|
||||
The valid values for `egress.allowed` and what they mean:
|
||||
|
||||
| Value | Path |
|
||||
|---|---|
|
||||
| `default` | Default route through the cell's WAN interface (no VPN). |
|
||||
| `wireguard_ext` | Traffic leaves through `wg_ext0` (fwmark `0x10`, table 110). Requires the `wireguard-ext` store service. |
|
||||
| `openvpn` | Traffic leaves through `tun0` (fwmark `0x20`, table 120). Requires the `openvpn-client` store service. |
|
||||
| `tor` | Traffic is redirected to the Tor transparent proxy on port 9040 (fwmark `0x30`, table 130). Requires the `tor` store service. |
|
||||
| `sshuttle` | Traffic is redirected to the sshuttle transparent proxy on port 12300 (fwmark `0x40`, table 140). Requires the `sshuttle` store service. |
|
||||
| `proxy` | Traffic is redirected to the redsocks transparent proxy on port 12345 (fwmark `0x50`, table 150). Requires the `proxy` store service. |
|
||||
|
||||
List only the interfaces that make sense for your service in `allowed`. The `default` value is used when the admin has not changed anything. Always include `default` in `allowed` so the admin has a way to use the normal path.
|
||||
|
||||
The egress field in the manifest tells PIC what options to present in the UI. Actual enforcement requires the cell to have the corresponding exit type configured (an OpenVPN config uploaded, a WireGuard external config active, etc.). If the chosen exit is not active, packets will be dropped by the kill-switch FORWARD rule in `cell-wireguard`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Quick-start example
|
||||
|
||||
This section walks through a minimal working example: a static website served from Nginx with no accounts, no backup, and no egress policy.
|
||||
|
||||
### `manifest.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 3,
|
||||
"id": "homepage",
|
||||
"name": "Homepage",
|
||||
"description": "A static homepage served from your cell",
|
||||
"version": "1.0.0",
|
||||
"author": "acme",
|
||||
"kind": "store",
|
||||
"min_pic_version": "1.0",
|
||||
|
||||
"capabilities": {
|
||||
"has_subdomain": true,
|
||||
"has_accounts": false,
|
||||
"has_admin_config": false,
|
||||
"has_storage": false,
|
||||
"has_egress": false,
|
||||
"has_api_hooks": false
|
||||
},
|
||||
|
||||
"subdomain": "home",
|
||||
"extra_subdomains": [],
|
||||
"backend": "cell-homepage:80",
|
||||
|
||||
"containers": ["cell-homepage"],
|
||||
|
||||
"image": "git.pic.ngo/roof/pic-homepage:latest",
|
||||
"container_name": "cell-homepage",
|
||||
|
||||
"volumes": [
|
||||
{ "name": "homepage-html", "mount": "/usr/share/nginx/html" }
|
||||
],
|
||||
|
||||
"env": [],
|
||||
|
||||
"iptables_rules": [
|
||||
{
|
||||
"type": "ACCEPT",
|
||||
"dest_ip": "${SERVICE_IP}",
|
||||
"dest_port": 80,
|
||||
"proto": "tcp"
|
||||
}
|
||||
],
|
||||
|
||||
"caddy_route": {
|
||||
"subdomain": "home"
|
||||
},
|
||||
|
||||
"compose": null
|
||||
}
|
||||
```
|
||||
|
||||
### What PIC does on install
|
||||
|
||||
1. Downloads this manifest from the store index.
|
||||
2. Validates every field (image allowlist, volume safety, reserved subdomains, iptables rule format).
|
||||
3. Allocates a static IP from the service pool (`172.20.0.20`–`172.20.0.254`).
|
||||
4. Writes a Docker Compose override file that starts `cell-homepage` with the allocated IP on `cell-network`.
|
||||
5. Runs `docker compose up -d cell-homepage`.
|
||||
6. Applies the `iptables_rules` in `cell-wireguard` so peers can reach the container.
|
||||
7. Regenerates the Caddyfile so `home.<cell-domain>` proxies to `cell-homepage:80`.
|
||||
|
||||
The result is that any WireGuard peer can reach `https://home.alice.pic.ngo/` immediately after installation.
|
||||
|
||||
---
|
||||
|
||||
## 8. Reference implementations
|
||||
|
||||
The `email`, `calendar`, and `files` services in `pic-services/services/` are the canonical examples of a complete store service. They demonstrate the full feature set:
|
||||
|
||||
| Service | Notable features demonstrated |
|
||||
|---|---|
|
||||
| `email` | `has_accounts`, `has_egress`, multi-container (`cell-mail` + `cell-rainloop`), `extra_backends`, custom image baking defaults via Dockerfile |
|
||||
| `calendar` | `has_accounts`, CalDAV `peer_config_template`, htpasswd account provisioning |
|
||||
| `files` | `has_accounts`, `has_storage`, WebDAV + Filegator `extra_backends`, `backup.volumes` with multiple entries |
|
||||
|
||||
When in doubt about how to structure your manifest or compose template, use these as the reference.
|
||||
|
||||
---
|
||||
|
||||
## 9. Submitting to the store
|
||||
|
||||
### Package format
|
||||
|
||||
A store service package is a ZIP archive containing:
|
||||
|
||||
```
|
||||
homepage-1.0.0.zip
|
||||
├── manifest.json (required)
|
||||
├── compose-template.yml (recommended for multi-container services)
|
||||
└── install.sh (optional post-install script)
|
||||
```
|
||||
|
||||
`install.sh` is executed on the cell host after the container starts. Keep it minimal — initialise data structures, create default config files. Do not use it to install system packages or modify files outside the PIC project root.
|
||||
|
||||
### Store index entry
|
||||
|
||||
The store index at `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json` is a JSON array. Each entry looks like:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "homepage",
|
||||
"name": "Homepage",
|
||||
"description": "A static homepage served from your cell",
|
||||
"version": "1.0.0",
|
||||
"author": "acme"
|
||||
}
|
||||
```
|
||||
|
||||
PIC fetches the full manifest from `https://git.pic.ngo/roof/pic-services/raw/branch/main/services/{id}/manifest.json` when the admin clicks install.
|
||||
|
||||
### Submission process
|
||||
|
||||
1. Fork `https://git.pic.ngo/roof/pic-services`.
|
||||
2. Create a directory `services/<your-id>/` and add your `manifest.json`.
|
||||
3. Open a pull request against `main`.
|
||||
|
||||
The review checks the following before merging:
|
||||
|
||||
**Security**
|
||||
- Image hosted on `git.pic.ngo/roof/*`. No external registries.
|
||||
- No volume mounts to system paths or to the PIC project root.
|
||||
- `iptables_rules` only declare `ACCEPT` rules (no DROP, no REJECT, no chain redirects).
|
||||
- `env` values contain only alphanumeric characters and a small set of safe punctuation.
|
||||
- `install.sh` does not call `apt`, `yum`, `curl | bash`, or modify files outside the project.
|
||||
|
||||
**Correctness**
|
||||
- `subdomain` does not collide with the reserved list or with any existing store service.
|
||||
- `backend` points to the declared `container_name`.
|
||||
- If `has_accounts: true`, the container responds correctly on all three `/service-api/accounts` endpoints.
|
||||
- If `has_storage: true`, every `volumes` entry names a container that is running and a path that exists inside it.
|
||||
|
||||
**Quality**
|
||||
- `description` is one sentence, no marketing language.
|
||||
- `version` is a valid semver string.
|
||||
- `config_schema` labels are in plain English, sentence case.
|
||||
|
||||
### Versioning
|
||||
|
||||
Increment `version` in `manifest.json` with every change you submit. PIC does not auto-update installed services; the admin manually runs an update. When an update is available, the UI shows the version mismatch between the installed record and the store index.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: manifest field quick reference
|
||||
|
||||
| Field | Required | Notes |
|
||||
|---|---|---|
|
||||
| `schema_version` | yes | Must be `3` |
|
||||
| `id` | yes | |
|
||||
| `name` | yes | |
|
||||
| `description` | yes | |
|
||||
| `version` | yes | |
|
||||
| `author` | yes | |
|
||||
| `kind` | yes | Must be `"store"` |
|
||||
| `min_pic_version` | no | |
|
||||
| `capabilities.*` | yes | All six flags must be present |
|
||||
| `subdomain` | if `has_subdomain` | |
|
||||
| `extra_subdomains` | no | |
|
||||
| `backend` | if `has_subdomain` | |
|
||||
| `extra_backends` | no | |
|
||||
| `containers` | no | Informational |
|
||||
| `config_schema` | if `has_admin_config` | |
|
||||
| `peer_config_template` | if `has_accounts` | |
|
||||
| `accounts` | if `has_accounts` | |
|
||||
| `compose` | no | Always `null` — compose config goes in `compose-template.yml` |
|
||||
| `backup` | if `has_storage` | |
|
||||
| `egress` | if `has_egress` | |
|
||||
| `storage` | if `has_storage` | |
|
||||
| `image` | yes | Must match `git.pic.ngo/roof/*` |
|
||||
| `container_name` | yes | Must match `^cell-[a-z0-9][a-z0-9-]{0,30}$` |
|
||||
| `volumes` | no | |
|
||||
| `env` | no | |
|
||||
| `iptables_rules` | no | |
|
||||
| `caddy_route` | no | |
|
||||
@@ -518,7 +518,7 @@ class TestEmailManagerApply(unittest.TestCase):
|
||||
|
||||
|
||||
class TestImageVerificationConfig(unittest.TestCase):
|
||||
"""image_verification config round-trip and warn-by-default behaviour."""
|
||||
"""image_verification config round-trip; default is now 'enforce' (P3)."""
|
||||
|
||||
def setUp(self):
|
||||
self.temp_dir = tempfile.mkdtemp()
|
||||
@@ -530,9 +530,9 @@ class TestImageVerificationConfig(unittest.TestCase):
|
||||
def tearDown(self):
|
||||
shutil.rmtree(self.temp_dir)
|
||||
|
||||
def test_default_mode_is_warn(self):
|
||||
self.assertEqual(self.cm.get_image_verification_mode(), 'warn')
|
||||
self.assertEqual(self.cm.get_image_verification(), {'mode': 'warn'})
|
||||
def test_default_mode_is_enforce(self):
|
||||
self.assertEqual(self.cm.get_image_verification_mode(), 'enforce')
|
||||
self.assertEqual(self.cm.get_image_verification(), {'mode': 'enforce'})
|
||||
|
||||
def test_set_and_get_round_trip(self):
|
||||
for mode in ('off', 'warn', 'enforce'):
|
||||
@@ -548,9 +548,9 @@ class TestImageVerificationConfig(unittest.TestCase):
|
||||
with self.assertRaises(ValueError):
|
||||
self.cm.set_image_verification_mode('paranoid')
|
||||
|
||||
def test_corrupt_section_falls_back_to_warn(self):
|
||||
def test_corrupt_section_falls_back_to_enforce(self):
|
||||
self.cm.configs['image_verification'] = {'mode': 'bogus'}
|
||||
self.assertEqual(self.cm.get_image_verification_mode(), 'warn')
|
||||
self.assertEqual(self.cm.get_image_verification_mode(), 'enforce')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -874,6 +874,46 @@ class TestImageVerification(unittest.TestCase):
|
||||
self.assertTrue(result['ok'])
|
||||
c.up.assert_called_once()
|
||||
|
||||
def test_default_mode_is_enforce(self):
|
||||
"""get_image_verification_mode default is 'enforce' (P3 flip)."""
|
||||
from config_manager import ConfigManager as RealCM
|
||||
import tempfile, shutil
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
try:
|
||||
cm = RealCM(os.path.join(tmpdir, 'cell_config.json'),
|
||||
os.path.join(tmpdir, 'data'))
|
||||
c = ServiceComposer(config_manager=cm, data_dir=tmpdir)
|
||||
self.assertEqual(c._verification_mode(), 'enforce')
|
||||
finally:
|
||||
shutil.rmtree(tmpdir)
|
||||
|
||||
def test_enforce_default_rejects_unsigned_image(self):
|
||||
"""Under the default enforce mode, an undigested image aborts install."""
|
||||
c = _verify_composer('enforce')
|
||||
c._cosign_verify = MagicMock()
|
||||
manifest = {'image': 'git.pic.ngo/roof/unsigned:latest'}
|
||||
result = c.install('svc', manifest, 'tpl')
|
||||
self.assertFalse(result['ok'])
|
||||
self.assertIn('digest', result['error'])
|
||||
c._cosign_verify.assert_not_called()
|
||||
c.up.assert_not_called()
|
||||
|
||||
def test_mode_read_failure_falls_back_to_enforce(self):
|
||||
"""Config corruption must fail closed, not weaken verification."""
|
||||
c = _verify_composer('enforce')
|
||||
c.cm.get_image_verification_mode.side_effect = RuntimeError('corrupt config')
|
||||
self.assertEqual(c._verification_mode(), 'enforce')
|
||||
|
||||
def test_enforce_default_proceeds_with_signed_digested_image(self):
|
||||
"""Under the default enforce mode, a digest-pinned signed image proceeds."""
|
||||
c = _verify_composer('enforce')
|
||||
c._cosign_verify = MagicMock(return_value={'ok': True, 'stdout': '[]', 'stderr': ''})
|
||||
manifest = {'image': _SIGNED_IMAGE}
|
||||
result = c.install('svc', manifest, 'tpl')
|
||||
self.assertTrue(result['ok'])
|
||||
c._cosign_verify.assert_called_once()
|
||||
c.up.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
|
||||
@@ -693,7 +693,7 @@ class TestInstall(unittest.TestCase):
|
||||
composer.install.assert_not_called()
|
||||
|
||||
def test_install_warn_allows_undigested_image(self):
|
||||
"""Under warn mode (default) an undigested image still installs."""
|
||||
"""Under warn mode an undigested image still installs (warn is non-default)."""
|
||||
manifest = _valid_manifest(
|
||||
id='myapp', container_name='cell-myapp',
|
||||
image='git.pic.ngo/roof/myapp:latest',
|
||||
@@ -713,6 +713,19 @@ class TestInstall(unittest.TestCase):
|
||||
self.assertTrue(result['ok'])
|
||||
composer.install.assert_called_once()
|
||||
|
||||
def test_install_enforce_default_rejects_undigested_image(self):
|
||||
"""The new default mode (enforce) rejects undigested images without explicit mode set."""
|
||||
manifest = _valid_manifest(
|
||||
id='myapp', container_name='cell-myapp',
|
||||
image='git.pic.ngo/roof/myapp:latest',
|
||||
)
|
||||
ssm, cm, _, composer = _make_ssm(manifest=manifest)
|
||||
cm.get_image_verification_mode.return_value = 'enforce'
|
||||
result = ssm.install('myapp')
|
||||
self.assertFalse(result['ok'])
|
||||
self.assertIn('digest', result['error'].lower())
|
||||
composer.install.assert_not_called()
|
||||
|
||||
def test_install_without_composer_stores_record(self):
|
||||
"""When service_composer=None, skip compose but still store the install record."""
|
||||
manifest = _valid_manifest(id='myapp', container_name='cell-myapp')
|
||||
|
||||
-138
@@ -1,138 +0,0 @@
|
||||
# Personal Internet Cell - Web UI
|
||||
|
||||
A modern React-based web interface for managing your Personal Internet Cell.
|
||||
|
||||
## Features
|
||||
|
||||
- **Dashboard**: Overview of cell status and services
|
||||
- **Peer Management**: Add, remove, and configure WireGuard peers
|
||||
- **Network Services**: DNS, DHCP, and NTP management
|
||||
- **WireGuard**: VPN configuration and status
|
||||
- **Email Services**: Postfix and Dovecot management
|
||||
- **Calendar Services**: Radicale CalDAV/CardDAV management
|
||||
- **File Storage**: WebDAV file storage management
|
||||
- **Routing**: Advanced VPN gateway and routing configuration
|
||||
- **Logs**: System logs and monitoring
|
||||
- **Settings**: Cell configuration and security settings
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **React 19**: Modern React with hooks
|
||||
- **Vite**: Fast build tool and dev server
|
||||
- **Tailwind CSS**: Utility-first CSS framework
|
||||
- **Lucide React**: Beautiful icons
|
||||
- **React Router**: Client-side routing
|
||||
- **Axios**: HTTP client for API communication
|
||||
|
||||
## Development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js 18+ and npm
|
||||
- Personal Internet Cell backend running on port 3000
|
||||
|
||||
### Setup
|
||||
|
||||
1. Install dependencies:
|
||||
```bash
|
||||
bun install
|
||||
```
|
||||
|
||||
2. Start the development server:
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
3. Open your browser to `http://localhost:5173`
|
||||
|
||||
### Development Features
|
||||
|
||||
- **Hot Reload**: Changes reflect immediately
|
||||
- **API Proxy**: Requests to `/api/*` are proxied to `http://localhost:3000`
|
||||
- **TypeScript Support**: Full TypeScript support available
|
||||
- **ESLint**: Code linting and formatting
|
||||
|
||||
## Building for Production
|
||||
|
||||
### Build
|
||||
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
This creates a `dist/` directory with optimized production files.
|
||||
|
||||
### Preview
|
||||
|
||||
```bash
|
||||
npm run preview
|
||||
```
|
||||
|
||||
This serves the built files locally for testing.
|
||||
|
||||
## API Integration
|
||||
|
||||
The Web UI communicates with the Personal Internet Cell backend API:
|
||||
|
||||
- **Base URL**: `http://localhost:3000` (development)
|
||||
- **Health Check**: `/health`
|
||||
- **API Endpoints**: `/api/*`
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Create a `.env` file to customize the API URL:
|
||||
|
||||
```env
|
||||
VITE_API_URL=http://localhost:3000
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── components/ # Reusable UI components
|
||||
│ └── Sidebar.jsx # Navigation sidebar
|
||||
├── pages/ # Page components
|
||||
│ ├── Dashboard.jsx # Main dashboard
|
||||
│ ├── Peers.jsx # Peer management
|
||||
│ ├── NetworkServices.jsx
|
||||
│ ├── WireGuard.jsx # VPN configuration
|
||||
│ ├── Email.jsx # Email services
|
||||
│ ├── Calendar.jsx # Calendar services
|
||||
│ ├── Files.jsx # File storage
|
||||
│ ├── Routing.jsx # Routing configuration
|
||||
│ ├── Logs.jsx # System logs
|
||||
│ └── Settings.jsx # Cell settings
|
||||
├── services/ # API services
|
||||
│ └── api.js # API client and endpoints
|
||||
├── App.jsx # Main app component
|
||||
├── main.jsx # App entry point
|
||||
└── index.css # Global styles
|
||||
```
|
||||
|
||||
## Styling
|
||||
|
||||
The Web UI uses Tailwind CSS with custom components:
|
||||
|
||||
- **Cards**: `.card` for content containers
|
||||
- **Buttons**: `.btn`, `.btn-primary`, `.btn-secondary`, etc.
|
||||
- **Inputs**: `.input` for form fields
|
||||
- **Status Indicators**: `.status-indicator`, `.status-online`, etc.
|
||||
|
||||
## Browser Support
|
||||
|
||||
- Chrome 90+
|
||||
- Firefox 88+
|
||||
- Safari 14+
|
||||
- Edge 90+
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Follow the existing code style
|
||||
2. Use TypeScript for new components
|
||||
3. Add tests for new features
|
||||
4. Update documentation as needed
|
||||
|
||||
## License
|
||||
|
||||
Part of the Personal Internet Cell project.
|
||||
Reference in New Issue
Block a user