10ac15d9fe
Email, calendar, and files are now optional store services, not always-on builtins. Updated README, QUICKSTART, Wiki, and service-developer-guide to reflect: dynamic nav, optional service install flow, correct egress identifiers (wireguard_ext/default vs wireguard/cell_internet), removed builtin/store distinction from manifest reference, 7 core containers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
742 lines
34 KiB
Markdown
742 lines
34 KiB
Markdown
# PIC Service Developer Guide
|
||
|
||
This guide is for developers who want to build services that integrate with Personal Internet Cell (PIC). It covers the manifest format, how PIC wires up routing, DNS, backup, and account provisioning for your service, and how to package and submit your work.
|
||
|
||
**Prerequisites:** you should be comfortable with Docker, Docker Compose, and basic Linux networking. You do not need to know Python to build a store service.
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [What a PIC service is](#1-what-a-pic-service-is)
|
||
2. [Manifest reference](#2-manifest-reference)
|
||
3. [Compose template variables](#3-compose-template-variables)
|
||
4. [Account provisioning interface](#4-account-provisioning-interface)
|
||
5. [Backup integration](#5-backup-integration)
|
||
6. [Egress routing](#6-egress-routing)
|
||
7. [Quick-start example](#7-quick-start-example)
|
||
8. [Reference implementations](#8-reference-implementations)
|
||
9. [Submitting to the store](#9-submitting-to-the-store)
|
||
|
||
---
|
||
|
||
## 1. What a PIC service is
|
||
|
||
A PIC service is a Docker container (or a set of containers) that plugs into the PIC ecosystem through a single JSON file called the **manifest**. The manifest tells PIC everything it needs to know:
|
||
|
||
- How to route HTTPS traffic to the service through Caddy
|
||
- What subdomains to expose
|
||
- Which users get accounts on the service and what credentials they receive
|
||
- Which paths to include in automated backups
|
||
- Which outbound network interfaces the service is allowed to use
|
||
|
||
All PIC services are **store services** — optional packages installed by the cell admin from the `pic-services` catalog. PIC downloads the manifest, renders a per-service Docker Compose file, and starts the containers. The core PIC stack (DNS, DHCP, NTP, WireGuard, Caddy, API, WebUI) runs independently of any installed services.
|
||
|
||
The email, calendar, and files services (in `pic-services/services/`) are the reference implementations and show the full feature set. The `ServiceRegistry` in `api/service_registry.py` is the single source of truth for all installed services. `CaddyManager`, the backup system, and the peer services endpoint all read from it rather than from hardcoded lists.
|
||
|
||
---
|
||
|
||
## 2. Manifest reference
|
||
|
||
The manifest is a JSON file with `"schema_version": 3`. Every field is described below. The `email`, `calendar`, and `files` manifests in `pic-services/services/` are the canonical reference examples.
|
||
|
||
### Top-level identity fields
|
||
|
||
| Field | Type | Required | Description |
|
||
|---|---|---|---|
|
||
| `schema_version` | integer | yes | Must be `3`. |
|
||
| `id` | string | yes | Unique service identifier, lowercase, no spaces (e.g. `"notes"`). Must match the directory name for builtins, or the store index entry for store services. |
|
||
| `name` | string | yes | Human-readable display name (e.g. `"Notes"`). |
|
||
| `description` | string | yes | One-sentence description shown in the UI. |
|
||
| `version` | string | yes | Semver string for the service package itself (e.g. `"1.0.0"`). |
|
||
| `author` | string | yes | Your name or organisation. |
|
||
| `kind` | string | yes | Must be `"store"`. |
|
||
| `min_pic_version` | string | no | Minimum PIC version required (e.g. `"1.0"`). |
|
||
|
||
```json
|
||
{
|
||
"schema_version": 3,
|
||
"id": "notes",
|
||
"name": "Notes",
|
||
"description": "Self-hosted Markdown notes with full-text search",
|
||
"version": "1.0.0",
|
||
"author": "acme",
|
||
"kind": "store",
|
||
"min_pic_version": "1.0"
|
||
}
|
||
```
|
||
|
||
### `capabilities`
|
||
|
||
A set of boolean flags that tell PIC which integrations to activate for your service.
|
||
|
||
| Field | Type | Default | Description |
|
||
|---|---|---|---|
|
||
| `has_subdomain` | bool | `false` | The service gets a subdomain and a Caddy reverse-proxy route. Requires `subdomain` and `backend`. |
|
||
| `has_accounts` | bool | `false` | The service provisions per-peer accounts. Requires `accounts`. |
|
||
| `has_admin_config` | bool | `false` | The service has admin-configurable fields. Requires `config_schema`. |
|
||
| `has_storage` | bool | `false` | The service has data worth backing up. Requires `backup`. |
|
||
| `has_egress` | bool | `false` | The admin can choose which outbound interface this service uses. Requires `egress`. |
|
||
| `has_api_hooks` | bool | `false` | Reserved for future use; set `false`. |
|
||
|
||
```json
|
||
"capabilities": {
|
||
"has_subdomain": true,
|
||
"has_accounts": true,
|
||
"has_admin_config": false,
|
||
"has_storage": true,
|
||
"has_egress": false,
|
||
"has_api_hooks": false
|
||
}
|
||
```
|
||
|
||
### `subdomain`, `extra_subdomains`, `backend`, `extra_backends`
|
||
|
||
These fields are only read when `has_subdomain` is `true`.
|
||
|
||
| Field | Type | Required | Description |
|
||
|---|---|---|---|
|
||
| `subdomain` | string | yes (if `has_subdomain`) | The primary subdomain (e.g. `"notes"`). Results in `notes.<cell-domain>`. Must not collide with reserved names: `api`, `webui`, `admin`, `www`, `ns1`, `ns2`, `git`, `registry`, `install`. |
|
||
| `extra_subdomains` | array of strings | no | Additional subdomains that point to the same backend (e.g. `["webmail"]`). |
|
||
| `backend` | string | yes (if `has_subdomain`) | The container-name:port combination that Caddy proxies to (e.g. `"cell-notes:8080"`). Uses Docker DNS on the `cell-network`. |
|
||
| `extra_backends` | object | no | Maps extra subdomain names to separate backends. Key is the subdomain string; value is the backend string. The email service uses this to send `webdav.*` to a different container than `files.*`. |
|
||
|
||
```json
|
||
"subdomain": "notes",
|
||
"extra_subdomains": [],
|
||
"backend": "cell-notes:8080"
|
||
```
|
||
|
||
**Validation at runtime:** `ServiceRegistry.get_caddy_routes()` validates all subdomain and backend values before passing them to CaddyManager or NetworkManager. Any entry whose `subdomain` does not match `^[a-z][a-z0-9-]{0,30}$`, whose `backend` does not match `^[A-Za-z0-9._-]+:\d{1,5}$`, or whose `subdomain` appears in the reserved list is silently skipped with a warning log. The same validation applies to `extra_subdomains` and `extra_backends` keys/values. For store services, this validation is also performed during installation by `ServiceStoreManager._validate_manifest()`.
|
||
|
||
### `containers`
|
||
|
||
Array of container names that belong to this service. Used by the UI and log viewer. For builtins this is informational; for store services PIC only manages the single container declared in the manifest.
|
||
|
||
```json
|
||
"containers": ["cell-notes"]
|
||
```
|
||
|
||
### `config_schema`
|
||
|
||
Defines admin-configurable fields for this service. When `has_admin_config` is `true`, the UI renders a settings form from this schema. PIC stores admin-saved values in `cell_config.json` and merges them with your `default` values at runtime. The merged result is available as the `config` key when `ServiceRegistry.get()` returns your service.
|
||
|
||
Each field is an object:
|
||
|
||
| Key | Type | Required | Description |
|
||
|---|---|---|---|
|
||
| `type` | string | yes | One of `"string"`, `"integer"`, `"boolean"`. |
|
||
| `label` | string | yes | Human-readable label for the settings form. |
|
||
| `required` | bool | no | Whether the field must have a value before the service starts. |
|
||
| `default` | any | no | Default value used when the admin has not set one. |
|
||
| `min` / `max` | integer | no (integer only) | Inclusive bounds for integer fields. |
|
||
|
||
```json
|
||
"config_schema": {
|
||
"port": {
|
||
"type": "integer",
|
||
"label": "Internal HTTP port",
|
||
"default": 8080,
|
||
"min": 1,
|
||
"max": 65535
|
||
},
|
||
"storage_path": {
|
||
"type": "string",
|
||
"label": "Data directory inside container",
|
||
"default": "/data/notes"
|
||
}
|
||
}
|
||
```
|
||
|
||
### `peer_config_template`
|
||
|
||
When a peer is provisioned on this service, PIC fills this template and returns the result to the peer as their connection info. Template substitution tokens:
|
||
|
||
| Token | Replaced with |
|
||
|---|---|
|
||
| `{domain}` | The cell's public domain (e.g. `alice.pic.ngo`) |
|
||
| `{peer.username}` | The peer's username |
|
||
| `{peer.service_credentials.<id>.<field>}` | A credential value; `<id>` is the service `id`, `<field>` matches a name in `accounts.credentials` |
|
||
| `{config.<key>}` | A value from the merged `config_schema` result |
|
||
|
||
```json
|
||
"peer_config_template": {
|
||
"url": "https://notes.{domain}/",
|
||
"username": "{peer.username}",
|
||
"password": "{peer.service_credentials.notes.password}"
|
||
}
|
||
```
|
||
|
||
### `accounts`
|
||
|
||
Required when `has_accounts` is `true`.
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `manager` | string | Set to `"http"` for store services — PIC will call your container's HTTP API for account operations (see section 4). The reference services (`email`, `calendar`, `files`) use internal manager names (`"email_manager"`, `"calendar_manager"`, `"file_manager"`). |
|
||
| `credentials` | array of strings | Names of credential fields this service issues per peer. Most services use `["password"]`. The names appear in `peer_config_template` tokens. |
|
||
|
||
```json
|
||
"accounts": {
|
||
"manager": "http",
|
||
"credentials": ["password"]
|
||
}
|
||
```
|
||
|
||
### `compose`
|
||
|
||
Unused at the manifest level. Compose configuration is provided via `compose-template.yml` in the service package (see section 3). Set to `null` in the manifest.
|
||
|
||
### `backup`
|
||
|
||
Required when `has_storage` is `true`. Tells PIC's backup system what to snapshot.
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `volumes` | array of objects | Container paths to stream out via `docker exec tar`. Each entry has three string fields: `container` (container name), `path` (absolute path inside the container), and `name` (archive filename stem). |
|
||
| `config_paths` | array of strings | Paths relative to the PIC project root on the host that contain service configuration (not user data). Copied directly into the snapshot. |
|
||
|
||
Each entry in `volumes` produces an archive at `<name>.tar.gz` inside the snapshot. For example, `"name": "maildata"` produces `maildata.tar.gz`.
|
||
|
||
```json
|
||
"backup": {
|
||
"volumes": [
|
||
{"container": "cell-notes", "path": "/data/notes", "name": "notes_data"}
|
||
],
|
||
"config_paths": ["config/notes"]
|
||
}
|
||
```
|
||
|
||
`ServiceRegistry.get_backup_plan()` aggregates these declarations across all installed services. The backup runner reads from that method rather than from any hardcoded list.
|
||
|
||
### `egress`
|
||
|
||
Required when `has_egress` is `true`. Declares which outbound network interfaces this service is permitted to use.
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `default` | string | The interface selected when the admin has not changed anything. |
|
||
| `allowed` | array of strings | The complete set of interfaces the admin can choose from. |
|
||
|
||
Valid interface identifiers: `default`, `wireguard_ext`, `openvpn`, `tor`.
|
||
|
||
```json
|
||
"egress": {
|
||
"default": "default",
|
||
"allowed": ["default", "wireguard_ext", "openvpn", "tor"]
|
||
}
|
||
```
|
||
|
||
How enforcement works is described in section 6.
|
||
|
||
### `storage`
|
||
|
||
Informational metadata used by the UI to show storage usage.
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `primary_path` | string | The path (relative to project root) that holds the bulk of user data. |
|
||
| `quota_mb` | integer or null | Storage quota in megabytes; `null` means no limit. |
|
||
|
||
```json
|
||
"storage": {
|
||
"primary_path": "data/notes",
|
||
"quota_mb": null
|
||
}
|
||
```
|
||
|
||
### Store-only manifest fields
|
||
|
||
Store services (where `kind` is `"store"`) have additional required fields that builtins do not use. These are validated by `ServiceStoreManager._validate_manifest()` before installation is permitted.
|
||
|
||
| Field | Type | Required | Description |
|
||
|---|---|---|---|
|
||
| `image` | string | yes | Docker image to pull. Must match the pattern `git.pic.ngo/roof/*`. Images from other registries are rejected. |
|
||
| `container_name` | string | yes | The name Docker gives the running container. |
|
||
| `volumes` | array | no | Named volumes to mount. Each entry must have `name` (the volume name) and `mount` (the absolute path inside the container). Mounts to `/`, `/etc`, `/var`, `/proc`, `/sys`, `/dev`, `/app`, `/run`, `/boot`, and paths that are a prefix of the PIC project root are forbidden. |
|
||
| `env` | array | no | Environment variables to pass. Each entry has `key` and `value`. Values must match `^[A-Za-z0-9._@:/+\-= ]*$`. |
|
||
| `iptables_rules` | array | no | FORWARD ACCEPT rules PIC should install in `cell-wireguard`. Each rule must have `type: "ACCEPT"`, `dest_ip: "${SERVICE_IP}"`, an integer `dest_port` (1–65535), and an optional `proto` (`"tcp"` or `"udp"`, default `"tcp"`). The literal string `${SERVICE_IP}` is replaced with the allocated container IP at install time. |
|
||
| `caddy_route` | object | no | If the service exposes a web UI, provide `subdomain` (must not be reserved; must match `^[a-z][a-z0-9-]{0,30}$`). PIC inserts the corresponding `reverse_proxy` directive into the Caddyfile. |
|
||
|
||
---
|
||
|
||
## 3. Compose template variables
|
||
|
||
This section applies only to store services. Builtins define their containers directly in `docker-compose.yml`.
|
||
|
||
When you ship a store service, you include a `compose-template.yml` alongside your `manifest.json`. `ServiceComposer.render_template()` substitutes the variables below before writing the per-service `docker-compose.yml`.
|
||
|
||
| Variable | Syntax | Value |
|
||
|---|---|---|
|
||
| `${PIC_CFG_<KEY>}` | uppercase `config_schema` key | The admin-saved value for that field, or the `default` from the schema if the admin has not set it. For example, `config_schema.port` → `${PIC_CFG_PORT}`. |
|
||
| `${PIC_SECRET_<NAME>}` | any name you choose | An auto-generated random secret produced by `secrets.token_urlsafe(24)` (~32 URL-safe base64 characters). Generated once on first install, then reused unchanged on every reconfigure. Stored per service in `data/service_secrets.json`. |
|
||
| `${PIC_DOMAIN}` | literal | Effective domain from `ConfigManager` (e.g. `alice.pic.ngo`). |
|
||
| `${PIC_CELL_NAME}` | literal | Cell name from the identity config (e.g. `alice`). |
|
||
| `${PIC_SERVICE_ID}` | literal | The `id` field from the service manifest (e.g. `notes`). |
|
||
|
||
**Volume mounts**: Because docker compose runs inside the API container but the Docker daemon runs on the host, relative volume paths in compose templates resolve relative to the compose file's directory as seen by the HOST filesystem. To avoid path resolution surprises, prefer **named volumes** for service data (Docker manages them independently). If bind mounts are required, use absolute host paths with `${PIC_PROJECT_DIR}` once that variable is implemented, or document the expected host layout clearly.
|
||
|
||
Example `compose-template.yml` for a notes service:
|
||
|
||
```yaml
|
||
services:
|
||
cell-notes:
|
||
image: git.pic.ngo/roof/pic-notes:latest
|
||
container_name: cell-notes
|
||
restart: unless-stopped
|
||
environment:
|
||
NOTES_PORT: "${PIC_CFG_PORT}"
|
||
NOTES_DOMAIN: "${PIC_DOMAIN}"
|
||
NOTES_DB_PASS: "${PIC_SECRET_DB_PASSWORD}"
|
||
volumes:
|
||
- notes-data:/data/notes
|
||
networks:
|
||
cell-network:
|
||
ipv4_address: "${SERVICE_IP}"
|
||
|
||
volumes:
|
||
notes-data:
|
||
|
||
networks:
|
||
cell-network:
|
||
external: true
|
||
```
|
||
|
||
The `SERVICE_IP` variable is the IP PIC allocated from the service pool. It is always set automatically.
|
||
|
||
---
|
||
|
||
## 4. Account provisioning interface
|
||
|
||
This section covers two related things: the `AccountManager` class that is PIC's central credential dispatcher, and the HTTP API that store services must implement to receive account operations.
|
||
|
||
### How AccountManager works
|
||
|
||
`AccountManager` (`api/account_manager.py`) is the single entry point for all account operations across every service type. It is instantiated once in `api/managers.py` and holds references to the service managers used by the reference services (`email_manager`, `calendar_manager`, `file_manager`).
|
||
|
||
When a peer account is provisioned, `AccountManager`:
|
||
|
||
1. Looks up the service in `ServiceRegistry` and reads `accounts.manager` from the manifest.
|
||
2. Dispatches to the appropriate internal manager method (for builtins) or to the service's HTTP API endpoint (for store services — not yet implemented; `"http"` manager support is planned).
|
||
3. Stores the returned credentials in `data/peer_service_credentials.json` with permissions `0o600`.
|
||
|
||
Credentials are stored in plaintext. This is intentional: the peer credentials endpoint needs to return them verbatim for one-time client configuration. The `0o600` permission matches the pattern used for WireGuard keys and `data/service_secrets.json`.
|
||
|
||
The credentials file structure is:
|
||
|
||
```json
|
||
{
|
||
"<service_id>": {
|
||
"<peer_username>": { "password": "..." }
|
||
}
|
||
}
|
||
```
|
||
|
||
Writes use a write-then-rename pattern (`tmp` → final path) with `os.fsync` to avoid partial-write corruption.
|
||
|
||
### Manifest `accounts` field
|
||
|
||
The `accounts` block in the manifest wires a service into `AccountManager`.
|
||
|
||
| Field | Type | Description |
|
||
|---|---|---|
|
||
| `manager` | string | Which underlying manager handles account operations. For builtins: `"email_manager"`, `"calendar_manager"`, or `"file_manager"`. |
|
||
| `credentials` | array of strings | Names of the credential fields this service issues per peer. Most services use `["password"]`. These names are used as token keys in `peer_config_template`. |
|
||
|
||
```json
|
||
"accounts": {
|
||
"manager": "email_manager",
|
||
"credentials": ["password"]
|
||
}
|
||
```
|
||
|
||
The `manager` value must match a key that `AccountManager` was instantiated with. If the manager name has no registered dispatch entry, `provision()` raises `ValueError` immediately.
|
||
|
||
### Provision flow
|
||
|
||
```
|
||
POST /api/services/catalog/<service_id>/accounts
|
||
Content-Type: application/json
|
||
|
||
{ "username": "alice", "password": "optional" }
|
||
```
|
||
|
||
If `password` is omitted, `AccountManager` generates one with `secrets.token_urlsafe(16)`. The response on HTTP 201 is:
|
||
|
||
```json
|
||
{ "service_id": "email", "username": "alice", "provisioned": true }
|
||
```
|
||
|
||
The password is not echoed in the response. To retrieve stored credentials for a provisioned peer, call `GET /api/services/catalog/<id>/accounts/<username>/credentials`.
|
||
|
||
Internally, `AccountManager.provision(service_id, peer_username, password)`:
|
||
|
||
1. Resolves the service and its manager via `_resolve_service()`.
|
||
2. Calls the appropriate `_provision_*` method, which delegates to the concrete manager:
|
||
- `email_manager` → `create_email_user(username, domain, password)`
|
||
- `calendar_manager` → `create_calendar_user(username, password)`
|
||
- `file_manager` → `create_user(username, password)`
|
||
3. Stores `{"password": "<value>"}` under `[service_id][peer_username]` in the credentials file.
|
||
4. Returns the credential dict to the caller.
|
||
|
||
If the underlying manager call returns `False`, `provision()` raises `RuntimeError`. The route handler maps this to HTTP 500.
|
||
|
||
For email, the domain is read from the service's merged config (`svc['config']['domain']`). If that key is absent, provisioning raises `ValueError` before calling the manager.
|
||
|
||
### Deprovision flow
|
||
|
||
```
|
||
DELETE /api/services/catalog/<service_id>/accounts/<username>
|
||
```
|
||
|
||
`AccountManager.deprovision(service_id, peer_username)`:
|
||
|
||
1. Calls the appropriate `_deprovision_*` method on the underlying manager.
|
||
2. Removes the peer's entry from the credentials file. If that leaves the service block empty, the service block itself is removed.
|
||
3. Returns `True` if the underlying call succeeded.
|
||
|
||
The route returns HTTP 200 with `{"message": "..."}` on success, or HTTP 400 if the service does not exist or does not support accounts.
|
||
|
||
**Peer deletion** calls `AccountManager.deprovision_peer(peer_username)`, which iterates over every service the peer is provisioned on and calls `deprovision()` for each. Failures on individual services are logged and skipped rather than aborting the deletion — the method returns `{service_id: bool}` for every service attempted.
|
||
|
||
### PIC admin API endpoints for account management
|
||
|
||
These endpoints are in `api/routes/services.py` and `api/routes/peers.py`.
|
||
|
||
| Method | Path | Description |
|
||
|---|---|---|
|
||
| `GET` | `/api/services/catalog/<service_id>/accounts` | Return `{"service_id": "...", "accounts": ["alice", "bob"]}` — reads directly from the credentials file. |
|
||
| `POST` | `/api/services/catalog/<service_id>/accounts` | Provision a peer account. Body: `{"username": "...", "password": "..."}` (password optional). Returns HTTP 201 with `{"service_id", "username", "provisioned": true}`. |
|
||
| `DELETE` | `/api/services/catalog/<service_id>/accounts/<username>` | Deprovision the peer's account. Returns HTTP 200 on success, HTTP 400 if the service or username is unknown. |
|
||
| `GET` | `/api/services/catalog/<service_id>/accounts/<username>/credentials` | Return stored credentials for one peer+service pair. Returns HTTP 404 if the peer is not provisioned on that service. Response: `{"service_id", "username", "password"}`. |
|
||
| `GET` | `/api/peers/<peer_name>/service-credentials` | Return filled `peer_config_template` values for all services the peer is provisioned on (see below). |
|
||
|
||
**Admin UI:** The Email, Calendar, and Files service pages in the admin dashboard each have an **Accounts** tab. From there, admins can provision and deprovision peer accounts, and reveal stored credentials for a provisioned peer. This tab calls the same API endpoints listed above.
|
||
|
||
### How `peer_config_template` connects to stored credentials
|
||
|
||
`GET /api/peers/<peer_name>/service-credentials` is the endpoint a peer device calls during first-time setup to configure email, CalDAV, and file sync clients.
|
||
|
||
The route:
|
||
|
||
1. Calls `AccountManager.get_all_credentials(peer_name)` → `{service_id: {field: value}}`.
|
||
2. For each service, calls `ServiceRegistry.get_peer_service_info(service_id, peer_name, domain, cred)`.
|
||
3. `get_peer_service_info` iterates over `peer_config_template` and replaces tokens:
|
||
- `{domain}` → effective cell domain
|
||
- `{peer.username}` → URL-percent-encoded peer username (safe='')
|
||
- `{peer.service_credentials.<service_id>.<field>}` → the value from stored credentials
|
||
- `{config.<key>}` → value from the service's merged config schema
|
||
4. Returns the filled template dict as the value for that service in the response.
|
||
|
||
Response shape:
|
||
|
||
```json
|
||
{
|
||
"peer": "alice",
|
||
"services": {
|
||
"email": {
|
||
"imap_host": "mail.alice.pic.ngo",
|
||
"username": "alice@alice.pic.ngo",
|
||
"password": "<stored>"
|
||
},
|
||
"files": {
|
||
"url": "https://files.alice.pic.ngo/dav/alice/",
|
||
"username": "alice",
|
||
"password": "<stored>"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
If a service has no `peer_config_template` in its manifest, `get_peer_service_info` returns `None` and the raw credential dict is used as the fallback.
|
||
|
||
### Container lifecycle routes
|
||
|
||
The following PIC API endpoints are available for all services (builtins and store services). These are called by the web UI and can be called directly from the PIC admin API.
|
||
|
||
| Method | Path | Description |
|
||
|---|---|---|
|
||
| `GET` | `/api/services/catalog/<id>/status` | Return container status. Builtins query the main compose stack; store services query their own compose project. Response includes a `containers` array with one entry per container. |
|
||
| `POST` | `/api/services/catalog/<id>/restart` | Restart the service containers. Builtins restart via the main compose stack; store services restart via their own compose project. |
|
||
| `POST` | `/api/services/catalog/<id>/reconfigure` | Re-render the compose file from the template and re-apply with `up -d` (rolling update). Store services only — builtins are reconfigured through their own settings routes. The request body must include a `compose_template` field containing the new template content. |
|
||
|
||
### Store service HTTP API
|
||
|
||
When `accounts.manager` is `"http"`, PIC will call your container's HTTP API for account operations. **HTTP dispatch is not yet wired up in `AccountManager`** — the current dispatch table covers only `email_manager`, `calendar_manager`, and `file_manager` (used by the reference services). Implement this interface now so your service is ready when HTTP dispatch ships.
|
||
|
||
The base path is `/service-api/accounts` on your container's internal address. There is no authentication on this API — it is reachable only from within the `cell-network` Docker network.
|
||
|
||
**Create account**
|
||
|
||
```
|
||
POST /service-api/accounts
|
||
Content-Type: application/json
|
||
|
||
{ "username": "alice", "password": "auto-generated-by-pic" }
|
||
```
|
||
|
||
PIC generates the password and passes it to your service. Return HTTP 200 with `{"ok": true}` on success. Return HTTP 400 or 409 with `{"ok": false, "error": "..."}` for expected errors (duplicate username, invalid input). Return HTTP 500 for unexpected internal errors.
|
||
|
||
**Delete account**
|
||
|
||
```
|
||
DELETE /service-api/accounts/{username}
|
||
```
|
||
|
||
Return HTTP 200 with `{"ok": true}` on success. Return HTTP 404 with `{"ok": false, "error": "not found"}` if the account does not exist.
|
||
|
||
**List accounts**
|
||
|
||
```
|
||
GET /service-api/accounts
|
||
```
|
||
|
||
Return `{"accounts": ["alice", "bob"]}` — an array of all provisioned usernames.
|
||
|
||
---
|
||
|
||
## 5. Backup integration
|
||
|
||
Declare `has_storage: true` in `capabilities` and fill in the `backup` block. PIC's `ServiceRegistry.get_backup_plan()` returns the combined backup declarations for all installed services. The backup runner reads from that method.
|
||
|
||
### Why docker exec instead of bind mounts
|
||
|
||
The API container only has access to `data/api/` on the host filesystem. Service data (mailboxes, calendar collections, file trees) lives in other containers' volumes. Rather than mount every service volume into the API container — which would require compose changes per service — PIC streams data using `docker exec <container> tar czf - <path>`. This works for any container on the Docker host regardless of how its volumes are configured.
|
||
|
||
### `volumes` entries
|
||
|
||
Each object in the `volumes` array describes one directory to capture:
|
||
|
||
| Field | Description |
|
||
|---|---|
|
||
| `container` | Name of the running container to exec into (e.g. `"cell-notes"`). |
|
||
| `path` | Absolute path inside that container to archive (e.g. `"/data/notes"`). |
|
||
| `name` | Archive filename stem. PIC saves the archive as `<name>.tar.gz` under `service_data/<service_id>/` in the backup directory. |
|
||
|
||
A service with multiple containers or multiple data directories lists one entry per directory.
|
||
|
||
**Security note:** The backup commands use `docker exec -- <container> tar -C <path> -czf - .` (note the `--` separator before the container name) to prevent option injection. The container name is also validated against `^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,63}$` before the command is run.
|
||
|
||
### `config_paths`
|
||
|
||
Paths in `config_paths` are relative to the PIC project root on the host and are copied directly into the snapshot (no docker exec). Use this for configuration files the service reads at startup, not for user data.
|
||
|
||
### Full example
|
||
|
||
```json
|
||
"backup": {
|
||
"volumes": [
|
||
{"container": "cell-notes", "path": "/data/notes", "name": "notes_data"}
|
||
],
|
||
"config_paths": ["config/notes"]
|
||
}
|
||
```
|
||
|
||
This produces one archive `notes_data.tar.gz` (streamed from the `cell-notes` container) plus a direct copy of `config/notes/` from the host.
|
||
|
||
### Restore
|
||
|
||
PIC restores each volume entry by piping the archive back via `docker exec -i -- <container> tar -C <path> -xzf -`. The `-C <path>` flag bounds extraction to the declared volume path — the same path used during backup. Archive entries are relative paths (the backup uses `tar -C <path> -czf - .`), so files land in exactly the location declared in the manifest `volumes` entry. The target container must be running at restore time.
|
||
|
||
---
|
||
|
||
## 6. Egress routing
|
||
|
||
When `has_egress` is `true`, the cell admin can assign a specific outbound interface to your service. PIC enforces the selection using `fwmark` rules and policy routing in the `cell-wireguard` container via the `ConnectivityManager`.
|
||
|
||
The valid values for `egress.allowed` and what they mean:
|
||
|
||
| Value | Path |
|
||
|---|---|
|
||
| `default` | Default route through the cell's WAN interface (no VPN). |
|
||
| `wireguard_ext` | Traffic leaves through `wg_ext0` (fwmark `0x10`, table 110). |
|
||
| `openvpn` | Traffic leaves through `tun0` (fwmark `0x20`, table 120). |
|
||
| `tor` | Traffic is redirected to the Tor transparent proxy on port 9040 (fwmark `0x30`, table 130). |
|
||
|
||
List only the interfaces that make sense for your service in `allowed`. The `default` value is used when the admin has not changed anything. Always include `default` in `allowed` so the admin has a way to use the normal path.
|
||
|
||
The egress field in the manifest tells PIC what options to present in the UI. Actual enforcement requires the cell to have the corresponding exit type configured (an OpenVPN config uploaded, a WireGuard external config active, etc.). If the chosen exit is not active, packets will be dropped by the kill-switch FORWARD rule in `cell-wireguard`.
|
||
|
||
---
|
||
|
||
## 7. Quick-start example
|
||
|
||
This section walks through a minimal working example: a static website served from Nginx with no accounts, no backup, and no egress policy.
|
||
|
||
### `manifest.json`
|
||
|
||
```json
|
||
{
|
||
"schema_version": 3,
|
||
"id": "homepage",
|
||
"name": "Homepage",
|
||
"description": "A static homepage served from your cell",
|
||
"version": "1.0.0",
|
||
"author": "acme",
|
||
"kind": "store",
|
||
"min_pic_version": "1.0",
|
||
|
||
"capabilities": {
|
||
"has_subdomain": true,
|
||
"has_accounts": false,
|
||
"has_admin_config": false,
|
||
"has_storage": false,
|
||
"has_egress": false,
|
||
"has_api_hooks": false
|
||
},
|
||
|
||
"subdomain": "home",
|
||
"extra_subdomains": [],
|
||
"backend": "cell-homepage:80",
|
||
|
||
"containers": ["cell-homepage"],
|
||
|
||
"image": "git.pic.ngo/roof/pic-homepage:latest",
|
||
"container_name": "cell-homepage",
|
||
|
||
"volumes": [
|
||
{ "name": "homepage-html", "mount": "/usr/share/nginx/html" }
|
||
],
|
||
|
||
"env": [],
|
||
|
||
"iptables_rules": [
|
||
{
|
||
"type": "ACCEPT",
|
||
"dest_ip": "${SERVICE_IP}",
|
||
"dest_port": 80,
|
||
"proto": "tcp"
|
||
}
|
||
],
|
||
|
||
"caddy_route": {
|
||
"subdomain": "home"
|
||
},
|
||
|
||
"compose": null
|
||
}
|
||
```
|
||
|
||
### What PIC does on install
|
||
|
||
1. Downloads this manifest from the store index.
|
||
2. Validates every field (image allowlist, volume safety, reserved subdomains, iptables rule format).
|
||
3. Allocates a static IP from the service pool (`172.20.0.20`–`172.20.0.254`).
|
||
4. Writes a Docker Compose override file that starts `cell-homepage` with the allocated IP on `cell-network`.
|
||
5. Runs `docker compose up -d cell-homepage`.
|
||
6. Applies the `iptables_rules` in `cell-wireguard` so peers can reach the container.
|
||
7. Regenerates the Caddyfile so `home.<cell-domain>` proxies to `cell-homepage:80`.
|
||
|
||
The result is that any WireGuard peer can reach `https://home.alice.pic.ngo/` immediately after installation.
|
||
|
||
---
|
||
|
||
## 8. Reference implementations
|
||
|
||
The `email`, `calendar`, and `files` services in `pic-services/services/` are the canonical examples of a complete store service. They demonstrate the full feature set:
|
||
|
||
| Service | Notable features demonstrated |
|
||
|---|---|
|
||
| `email` | `has_accounts`, `has_egress`, multi-container (`cell-mail` + `cell-rainloop`), `extra_backends`, custom image baking defaults via Dockerfile |
|
||
| `calendar` | `has_accounts`, CalDAV `peer_config_template`, htpasswd account provisioning |
|
||
| `files` | `has_accounts`, `has_storage`, WebDAV + Filegator `extra_backends`, `backup.volumes` with multiple entries |
|
||
|
||
When in doubt about how to structure your manifest or compose template, use these as the reference.
|
||
|
||
---
|
||
|
||
## 9. Submitting to the store
|
||
|
||
### Package format
|
||
|
||
A store service package is a ZIP archive containing:
|
||
|
||
```
|
||
homepage-1.0.0.zip
|
||
├── manifest.json (required)
|
||
├── compose-template.yml (recommended for multi-container services)
|
||
└── install.sh (optional post-install script)
|
||
```
|
||
|
||
`install.sh` is executed on the cell host after the container starts. Keep it minimal — initialise data structures, create default config files. Do not use it to install system packages or modify files outside the PIC project root.
|
||
|
||
### Store index entry
|
||
|
||
The store index at `https://git.pic.ngo/roof/pic-services/raw/branch/main/index.json` is a JSON array. Each entry looks like:
|
||
|
||
```json
|
||
{
|
||
"id": "homepage",
|
||
"name": "Homepage",
|
||
"description": "A static homepage served from your cell",
|
||
"version": "1.0.0",
|
||
"author": "acme"
|
||
}
|
||
```
|
||
|
||
PIC fetches the full manifest from `https://git.pic.ngo/roof/pic-services/raw/branch/main/services/{id}/manifest.json` when the admin clicks install.
|
||
|
||
### Submission process
|
||
|
||
1. Fork `https://git.pic.ngo/roof/pic-services`.
|
||
2. Create a directory `services/<your-id>/` and add your `manifest.json`.
|
||
3. Open a pull request against `main`.
|
||
|
||
The review checks the following before merging:
|
||
|
||
**Security**
|
||
- Image hosted on `git.pic.ngo/roof/*`. No external registries.
|
||
- No volume mounts to system paths or to the PIC project root.
|
||
- `iptables_rules` only declare `ACCEPT` rules (no DROP, no REJECT, no chain redirects).
|
||
- `env` values contain only alphanumeric characters and a small set of safe punctuation.
|
||
- `install.sh` does not call `apt`, `yum`, `curl | bash`, or modify files outside the project.
|
||
|
||
**Correctness**
|
||
- `subdomain` does not collide with the reserved list or with any existing store service.
|
||
- `backend` points to the declared `container_name`.
|
||
- If `has_accounts: true`, the container responds correctly on all three `/service-api/accounts` endpoints.
|
||
- If `has_storage: true`, every `volumes` entry names a container that is running and a path that exists inside it.
|
||
|
||
**Quality**
|
||
- `description` is one sentence, no marketing language.
|
||
- `version` is a valid semver string.
|
||
- `config_schema` labels are in plain English, sentence case.
|
||
|
||
### Versioning
|
||
|
||
Increment `version` in `manifest.json` with every change you submit. PIC does not auto-update installed services; the admin manually runs an update. When an update is available, the UI shows the version mismatch between the installed record and the store index.
|
||
|
||
---
|
||
|
||
## Appendix: manifest field quick reference
|
||
|
||
| Field | Required | Notes |
|
||
|---|---|---|
|
||
| `schema_version` | yes | Must be `3` |
|
||
| `id` | yes | |
|
||
| `name` | yes | |
|
||
| `description` | yes | |
|
||
| `version` | yes | |
|
||
| `author` | yes | |
|
||
| `kind` | yes | Must be `"store"` |
|
||
| `min_pic_version` | no | |
|
||
| `capabilities.*` | yes | All six flags must be present |
|
||
| `subdomain` | if `has_subdomain` | |
|
||
| `extra_subdomains` | no | |
|
||
| `backend` | if `has_subdomain` | |
|
||
| `extra_backends` | no | |
|
||
| `containers` | no | Informational |
|
||
| `config_schema` | if `has_admin_config` | |
|
||
| `peer_config_template` | if `has_accounts` | |
|
||
| `accounts` | if `has_accounts` | |
|
||
| `compose` | no | Always `null` — compose config goes in `compose-template.yml` |
|
||
| `backup` | if `has_storage` | |
|
||
| `egress` | if `has_egress` | |
|
||
| `storage` | if `has_storage` | |
|
||
| `image` | yes | Must match `git.pic.ngo/roof/*` |
|
||
| `container_name` | yes | Must match `^cell-[a-z0-9][a-z0-9-]{0,30}$` |
|
||
| `volumes` | no | |
|
||
| `env` | no | |
|
||
| `iptables_rules` | no | |
|
||
| `caddy_route` | no | |
|