Commit Graph

277 Commits

Author SHA1 Message Date
roof e8b8e47aa4 fix: use sudo for nft list tables — /usr/sbin not in roof user PATH
Unit Tests / test (push) Successful in 7m26s
nft lives in /usr/sbin which is absent from the non-root PATH on Debian.
The delete call already used sudo; add it to the list call too so the
session-scoped cleanup fixture doesn't crash before any test runs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 15:46:09 -04:00
roof adce219a46 fix: clean up stale wg-quick nftables tables in e2e test teardown
Unit Tests / test (push) Successful in 7m29s
wg-quick creates an nftables 'preraw' table per interface that drops
decrypted ICMP replies arriving on any other interface. If a test run
crashes before bring_down(), the table persists and silently kills pings
on subsequent runs (handshake succeeds, replies are decrypted, but the
stale table drops them before the ping process sees them).

Extend cleanup_stale_e2e_interfaces() to also delete any orphaned
wg-quick-pic-e2e-* nftables tables found on the host.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 15:35:19 -04:00
roof 65d6d07c8d fix: get_status returns actual configured WG address instead of hardcoded default
Unit Tests / test (push) Successful in 7m41s
The address field in get_status() was hardcoded to SERVER_ADDRESS
('10.0.0.1/24') regardless of what wg0.conf contains, so instances
with a non-default subnet (e.g. pic1 at 10.0.1.1/24) always reported
the wrong server IP to callers such as the e2e WG conftest fixture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 14:48:49 -04:00
roof ab6d6230dd Fix: read WG server IP and subnet from live API instead of hardcoding 10.0.0.x
Unit Tests / test (push) Successful in 7m30s
test_wg_connect_and_ping_server and the connected_peer fixture hardcoded
10.0.0.1 / 10.0.0.0/24 as the server VPN address. This breaks when the
server uses a different subnet (e.g. pic1 uses 10.0.1.1/24). Now both
read 'address' from /api/wireguard/status at session start and pass the
live server_ip / server_network through wg_server_info and connected_peer.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 14:09:48 -04:00
roof e2e9c50786 Test: skip peer-sync push test when WG tunnel between cells is not active
Unit Tests / test (push) Successful in 7m27s
The test_remote_permissions_pushed_to_cell2 test verifies that permission
changes on cell1 are pushed to cell2 via the WireGuard tunnel. When both
cells use a public endpoint (DDNS VPS) instead of LAN IPs, no tunnel is
established and the push silently fails. The test now probes cell2's API
at its WG DNS IP before asserting the push succeeded — skips gracefully
if the tunnel is down rather than failing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 12:52:03 -04:00
roof 568e4f9783 Fix: prevent wg0.conf truncation when remove_peer splits blocks
Unit Tests / test (push) Successful in 7m46s
_write_config() was stripping trailing newlines, causing the next
add_cell_peer() to create a single-newline separator between [Interface]
and [Peer] blocks instead of the required blank line. On the following
remove_peer() call, split('\n\n') treated both sections as one block,
matched the PublicKey filter, and wrote an empty string — destroying the
[Interface] section and reverting to the hardcoded SERVER_ADDRESS fallback.

Two-part fix:
1. _write_config() always ends content with a newline
2. remove_peer() normalises single-newline [Peer] headers to blank-line
   separators before splitting, and refuses to write if [Interface] would
   be lost

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 12:31:05 -04:00
roof 26576e1124 Fix: use domain_name (FQDN) in cell invite and conflict checks
Unit Tests / test (push) Successful in 7m39s
The GET /api/cells/invite endpoint was returning domain='pic.ngo' instead
of the full FQDN 'test5.pic.ngo' because it read _identity.domain rather
than _identity.domain_name.

Apply the same domain_name preference (domain_name || domain) to:
- routes/cells.py get_cell_invite() — the invite shown to connecting cells
- routes/cells.py update_cell_permissions() — Corefile DNS regeneration
- cell_link_manager.py _check_invite_conflicts() — incoming domain collision check
- cell_link_manager.py exchange_invites() — own invite construction

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 11:56:42 -04:00
roof 31f76c54fa Fix: use domain_name as service URL base and harden WG e2e tests
Unit Tests / test (push) Successful in 11m15s
API:
- _configured_domain() now prefers _identity.domain_name (full FQDN
  e.g. 'test5.pic.ngo') over domain ('pic.ngo'). Service URLs in
  /api/peer/services and /api/peer/dashboard now correctly return
  'calendar.test5.pic.ngo' instead of 'calendar.pic.ngo'.

WG e2e tests:
- test_api_domain_returns_json_not_webui: accept 3xx redirect as
  valid routing (Caddy redirects HTTP→HTTPS in pic_ngo mode).
- test_catchall_api_path_returns_json and test_catchall_root_serves_webui:
  skip when Caddy is in HTTPS-redirect mode — catch-all :80 block only
  exists in HTTP-mode cells (lan/local domain).
- test_http_api_domain_reaches_api: replace --dns-servers (requires
  c-ares) with dig + curl --host pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 08:40:59 -04:00
roof b6af71acb5 Fix: accept both VIP and Caddy IP in DNS resolution test
Unit Tests / test (push) Successful in 11m9s
Cells with wildcard zone (e.g. * -> 172.20.0.2) and cells with per-service
VIP DNS records are both valid. Accept either in the assertion so the test
passes regardless of the zone file style.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 08:29:05 -04:00
roof 352bb6bb9e Fix: use api_base fixture instead of hardcoded pic0 IP in WG domain access tests
test_peer_services_* functions hardcoded 'http://192.168.31.51:3000' as the
fallback for PIC_API_BASE, causing failures when tests run on any other host
(including pic1 itself). Use the api_base fixture, which reads PIC_HOST and
PIC_API_PORT from the environment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 08:06:29 -04:00
roof 463db029e1 Fix: expose listen_port in WG status API and add HTTPS DNAT to PostUp/PreDown
Unit Tests / test (push) Successful in 11m6s
Adds listen_port to /api/wireguard/status response so e2e test conftest
picks up the actual port (51821) instead of defaulting to 51820.

Extends PostUp/PreDown in generate_config to also DNAT and forward port
443 (HTTPS) through to cell-caddy — mirrors the ensure_service_dnat fix
so HTTPS works even after a WireGuard container restart without an API
restart. Updates _is_dnat_rule to recognize 443 rules.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 07:42:49 -04:00
roof 8da711e366 fix: DNAT and forward port 443 (HTTPS) to Caddy from WireGuard peers
Unit Tests / test (push) Successful in 11m9s
ensure_service_dnat() only wired port 80 → cell-caddy, so HTTPS was
silently dropped: no DNAT rule redirected 443 to the Caddy container,
and the FORWARD chain had no ACCEPT for dport 443. Refactored the
function to loop over both 80 and 443 so both are DNAT'd and forwarded.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 07:14:55 -04:00
roof 3e26186f85 fix: correct fake WireGuard key length and guard cell2_client teardown
Unit Tests / test (push) Successful in 11m14s
The synthetic cell fixture used a 46-char base64 key where the validator
expects exactly 43 chars before '='. The key failed format validation so
add_cell_peer returned False, making the cell connection store nothing and
all TestCellPermissionsApi tests hit 404.

The TestCellServiceAccessRestrictions and TestLiveCellConnection teardown
fixtures called _remove_connection(cell2_client, ...) without checking if
cell2_client is None (expected when no second cell is configured), causing
AttributeError on teardown.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 06:20:52 -04:00
roof f84f16fcd6 fix: add /api/network/dns/corefile endpoint and per-line iptables check
Unit Tests / test (push) Successful in 11m13s
The e2e tests were reading a stale Corefile at a hardcoded fallback path
(/home/roof/pic/config/dns/Corefile) instead of the live one written by
the API (/opt/pic/config/dns/Corefile on pic1). Adding a proper API
endpoint eliminates the path ambiguity.

The iptables test was checking whether peer_ip, DROP, and dpt:80 appeared
anywhere in the full multi-line output rather than on the same rule line,
producing false positives. Now checks per line.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 05:54:17 -04:00
roof eee0e800aa feat: add GET /api/peers/<peer_name> endpoint
Unit Tests / test (push) Successful in 11m19s
Allows fetching a single peer by name. E2E tests need this to verify
persisted peer state after PUT operations.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 05:19:10 -04:00
roof 2b29938a64 fix: set CSRF token in PicAPIClient after login
Unit Tests / test (push) Successful in 11m22s
POST requests from PicAPIClient were failing with 403 (CSRF token missing)
because the login response csrf_token was not being applied to subsequent
request headers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 05:05:08 -04:00
roof 39c59fd3ef feat: WireGuard endpoint override + fix Docker network label issue
Unit Tests / test (push) Successful in 11m14s
Endpoint override:
- Add PUT /api/wireguard/endpoint to set endpoint_override in identity
  config; GET returns detected, override, and effective endpoints
- _effective_endpoint() helper applies override in peer config generation
  (wireguard.py and peer_dashboard.py); detected IP still shown in UI
- Add Endpoint Override input in WireGuard page — solves the common case
  where auto-detected IP is a gateway/VPS but peers connect via LAN IP

Docker cell-network fix:
- Declare cell-network external in docker-compose.yml; Docker Compose v5
  enforces label ownership and rejects networks created by older versions
- Makefile start/update pre-create cell-network idempotently
- reinstall/uninstall(full) explicitly delete and recreate the network
- Fix uninstall loop path: data/api/services/ (not data/services/)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 04:51:38 -04:00
roof 1b44a18062 fix: declare cell-network external; pre-create in Makefile start/update
Unit Tests / test (push) Successful in 11m16s
Docker Compose v5 enforces label ownership on networks it creates. On
systems where cell-network was created by an older compose version (no
labels), Caddy and other services fail to start with "incorrect label"
error.

Declaring the network external in docker-compose.yml skips label
validation. The Makefile start/update targets now create the network if
it doesn't exist (idempotent). The reinstall and uninstall (full) paths
explicitly delete the network so fresh recreations are clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-06 03:13:01 -04:00
roof f3737acfa4 fix: fall back to cell effective domain when email service domain not configured
Unit Tests / test (push) Successful in 11m10s
When the email store service is installed but no explicit domain has been
set in its config, _provision_email now falls back to
config_manager.get_effective_domain() so peer account creation works
immediately without requiring a separate config step.

Also threads config_manager into AccountManager.__init__ (optional kwarg,
no existing callers break) so the fallback is available without a global
import.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 17:06:51 -04:00
roof 64dd8b8488 fix: uninstall stops optional service containers before core teardown
Unit Tests / test (push) Successful in 11m11s
Iterates data/services/*/docker-compose.yml and runs `docker compose down`
for each before stopping core containers, so stale optional-service
containers (email, calendar, files, etc.) don't leave cell-network occupied
and block a subsequent fresh install.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 15:52:49 -04:00
roof 0267dce73d feat: HTTPS cert status, IDENTITY_CHANGED wiring, remove stale ip_utils Caddyfile writes
Unit Tests / test (push) Successful in 11m18s
- CaddyManager: add refresh_cert_status() and get_cert_status_fresh() that
  open a live TLS connection to cell-caddy:443 to read cert expiry; avoids
  needing a volume mount into the API container
- CaddyManager: periodic cert refresh in health_monitor_loop (every 60 cycles)
- config.py PUT /api/ddns: publish IDENTITY_CHANGED so CaddyManager regenerates
  the Caddyfile immediately after any domain/cell_name change — previously the
  event was never fired from this route
- config.py: remove all ip_utils.write_caddyfile() calls; CaddyManager is now
  the sole authority for Caddyfile generation
- app.py: add GET /api/caddy/cert-status route
- app.py: add GET /api/egress/status and PUT /api/egress/services/<id>/exit routes
- Settings.jsx: display cert status badge (valid/expired/internal/unknown) with
  expiry date and days-remaining in the domain section
- Tests: TestRefreshCertStatus (8 tests), TestDdnsConfigUpdatesFiresIdentityChanged,
  TestCaddyCertStatusRoute added; fix expired-cert helper to set not_valid_before
  relative to expiry so it's always earlier

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 11:39:36 -04:00
roof 41d09c598b wire: AccountManager HTTP dispatch + EgressManager startup + egress API routes
Unit Tests / test (push) Successful in 11m15s
- add_peer() now calls account_manager.provision() for any installed store
  service whose manifest declares accounts.manager == 'http', enabling
  per-peer credential provisioning to third-party HTTP services
- reapply_on_startup() calls egress_manager.apply_all() so fwmark rules
  survive container restarts without manual intervention
- add GET /api/egress/status and PUT /api/egress/services/<id>/exit routes
  so the UI can read and override per-service egress policy
- tests: HTTP provision wiring (happy path + non-fatal failure), egress
  apply_all at startup (wired/unwired/failure cases)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-05 10:30:41 -04:00
roof a906c26b5d fix: resolve Caddy env vars at write time to prevent parse errors
Unit Tests / test (push) Successful in 11m25s
acme_ca and the pic_ngo DNS credentials ({$PIC_NGO_DDNS_TOKEN},
{$PIC_NGO_DDNS_API}) were written as Caddy env-var placeholders, but the
Caddy container does not inherit the API container's environment, so the
substitutions always failed — Caddy saw bare directive names with no
arguments and rejected the Caddyfile.

- _global_acme_block: only emit the acme_ca directive when ACME_CA_URL is
  actually set; omitting it makes Caddy default to Let's Encrypt production.
- _caddyfile_pic_ngo: embed the DDNS_TOTP_SECRET and DDNS_URL values directly
  into the Caddyfile at write time rather than relying on Caddy env expansion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 15:01:15 -04:00
roof e87022dc55 fix: cell-network name, install error surfacing, health history cleanup
Unit Tests / test (push) Successful in 11m22s
- docker-compose.services.yml: change external network name from
  pic_cell-network to cell-network so store-service compose files can find
  it.  The project-prefixed name was overriding the explicit name: cell-network
  fix in docker-compose.yml when both files were merged by make start.

- service_store.py: normalize docker compose stderr into the error key in
  the 400 response so the Store page shows the actual failure reason instead
  of the generic fallback message.

- app.py: skip health checks for email/calendar/files managers when those
  optional store services are not installed — prevents false Down alerts and
  unnecessary noise in health history.

- Logs.jsx: remove Email/Calendar/Files columns from the health history table;
  they are optional store services, not core builtins that should always appear.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 14:28:46 -04:00
roof 7d5c5421f1 Implement connectivity store services (wireguard-ext, openvpn-client, tor)
Unit Tests / test (push) Successful in 11m31s
- ConnectivityManager: move config dirs to data_dir/services/<id>/config so
  Docker can bind-mount them into store-service containers (Docker resolves
  bind-mount paths on the host, not inside the API container).  Add
  _migrate_legacy_configs to copy existing files from the old config_dir
  location on first boot.

- manifest_validator: add allow_host_network parameter to
  validate_rendered_compose.  When True, waives the external-network
  requirement, permits network_mode: host, and allows devices: — all needed
  by VPN/Tor containers that must share the host network namespace to create
  tun/wg interfaces.  Non-host services are unaffected.

- service_composer: read requires_host_network from the manifest and pass
  allow_host_network=True to validate_rendered_compose for connectivity
  services.

- Tests: update file-path assertions to new data_dir layout; add
  TestMigrateLegacyConfigs, TestValidateRenderedComposeHostNetwork, and
  two TestWriteCompose cases for the host-network path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 10:06:48 -04:00
roof 60601eb4af fix: give cell-network an explicit name to avoid compose project prefix
Unit Tests / test (push) Successful in 11m21s
Without name: cell-network, Docker Compose creates the network as
pic_cell-network (prefixed with the project name). Store service compose
templates declare cell-network as external: true and can't find it.
Adding name: cell-network makes the network name predictable regardless
of the Compose project name.

Existing installs need: make stop && make start to recreate the network.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 09:14:31 -04:00
roof 5ed75677c3 test: add e2e tests for service store install/uninstall flow
Unit Tests / test (push) Successful in 11m13s
Tests verify:
- /services page loads and lists all available services
- Admin can install calendar, files, email, and webmail via the store UI
- Install order respects dependencies (email before webmail)
- Uninstall flow shows confirmation dialog before removing
- Dashboard shows service links after install

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 04:51:10 -04:00
roof f7bb2cc962 fix: allow first-party store service subdomains and registry images
Unit Tests / test (push) Successful in 11m25s
Two manifest validation bugs blocked all store service installs:

1. service_store_manager.RESERVED_SUBDOMAINS included 'mail', which
   prevented the email service from using its required subdomain.
   Removed mail/calendar/files/webmail — they belong to official PIC
   store services and must be claimable by them.

2. manifest_validator required @sha256 digest pins on ALL images,
   including first-party git.pic.ngo/roof/* images that the PIC team
   builds and controls. service_store_manager._validate_manifest already
   only warned for first-party images; the secondary validator was
   stricter than intended, causing a hard reject on :latest tags.
   Aligned to warn-not-reject for first-party; malformed digests (when
   provided) are still a hard error.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 03:09:41 -04:00
roof c493630bb5 fix: Dashboard blank page — move state declarations before use
Unit Tests / test (push) Successful in 11m36s
SERVICES was computed on line 33 using activeServiceIds which was not
declared until line 36. In strict JS, const is not hoisted — this threw
a ReferenceError on mount, crashing the component and showing a blank page.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 02:44:41 -04:00
roof 0ed8669aec fix: dashboard only shows email/calendar/files if installed
Unit Tests / test (push) Successful in 11m25s
Fetches /api/services/active on load; service status cards and quick-
access links for email, calendar, files, and webmail are suppressed
until the service is installed via the Store. Core services (WireGuard,
Routing, Network) always show. Fixes #setup_complete gate on dev stack.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 01:38:16 -04:00
roof 03a67ad922 feat: add EgressManager — per-service egress enforcement via host iptables
Unit Tests / test (push) Successful in 11m20s
Routes outbound traffic from installed service containers through
alternate exits (wireguard_ext, openvpn, tor) using host-side
iptables fwmark policy-routing in a dedicated PIC_EGRESS chain.
Marks 0x110/0x120/0x130 are distinct from ConnectivityManager's
0x10/0x20/0x30. Container IPs discovered at runtime via docker
inspect. Wired into ServiceStoreManager install/remove lifecycle
and managers.py singleton. 22 new tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 00:58:47 -04:00
roof 5cbbfb41d9 feat: add HTTP dispatch to AccountManager for generic store services
Services with accounts.manager='http' now use POST/DELETE to the
service container's /service-api/accounts endpoint instead of
requiring a named Python manager. _resolve_service allows 'http'
without a registered Python object; _provision_http and
_deprovision_http handle the HTTP calls with 404-as-success on
delete. 9 new tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-30 00:46:54 -04:00
roof 1f2f9d9f6e feat: add manifest_validator.py — security chokepoint for compose and manifest validation
Unit Tests / test (push) Successful in 11m18s
Rejects privileged compose configs (network_mode:host, pid:host, ipc:host,
userns_mode:host, cap_add:ALL, string commands, missing cell-network,
reserved container names). Validates manifest schema_version=3, image
digest pinning (sha256 required, :tag-only rejected), and provision hook
format. Wired into ServiceComposer.write_compose() and
ServiceStoreManager.install() as a single enforcement point.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 18:45:45 -04:00
roof 62b31b072b feat: remove optional services step from setup wizard
Services are now installed post-setup from the Store page, so the
wizard step that let users pre-select email/calendar/files is removed.
Reduces wizard from 5 steps to 4 (Step4Services deleted, Step5Review
renamed to Step4Review). Backend drops services_enabled validation,
background install thread, and service_store_manager dependency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 18:33:43 -04:00
roof 3d594025d2 fix: remove legacy service dirs from setup_cell, update sanity_check for optional services
Unit Tests / test (push) Successful in 11m24s
setup_cell.py no longer creates mail/radicale/webdav config and data dirs —
those are managed by ServiceComposer when services are installed. Added
data/services/ for ServiceComposer. sanity_check.py now uses stdlib urllib
and discovers installed services via /api/services/active before checking
their status routes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 17:22:42 -04:00
roof 10ac15d9fe docs: Phase 7 — update docs to reflect optional services migration
Email, calendar, and files are now optional store services, not always-on
builtins. Updated README, QUICKSTART, Wiki, and service-developer-guide to
reflect: dynamic nav, optional service install flow, correct egress
identifiers (wireguard_ext/default vs wireguard/cell_internet), removed
builtin/store distinction from manifest reference, 7 core containers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 17:10:48 -04:00
roof 44d7e96f29 feat: Phase 6 — require_active_service decorator + wizard install wiring
Email/calendar/files routes now return 404 when the service is not
installed, using a require_active_service decorator that checks
ServiceRegistry. Status endpoints are exempt so health checks always work.

SetupManager.complete_setup() now accepts a service_store_manager and
installs any wizard-selected services in a background daemon thread after
setup completes. Failures are logged but do not fail the wizard.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 16:58:57 -04:00
roof a69ca1e402 feat: Phase 5 — remove legacy service blocks, one-shot container cleanup
Unit Tests / test (push) Successful in 11m20s
Email, calendar, files, webmail (rainloop), and the file manager (filegator)
are removed from the main docker-compose stack. They install as independent
per-service compose projects via ServiceComposer.

On startup, _cleanup_legacy_builtin_containers() stops and removes any of the
5 legacy containers still running from the old main stack (guarded by a
one-shot sentinel in _meta.legacy_builtins_cleaned so it never runs twice).
Per-service installs (com.docker.compose.project != 'pic') are left untouched.

Changes:
- docker-compose.yml: remove mail, radicale, webdav, rainloop, filegator blocks;
  fix dhcp + ntp to profiles: ["core","full"] so they start with --profile core
- Makefile: replace all --profile full with --profile core (6 occurrences);
  remove mailserver.env conditional from update: target
- api/legacy_cleanup.py: new module with cleanup_legacy_builtin_containers()
- api/app.py: import and call cleanup at startup before reapply_on_startup()
- tests/test_legacy_cleanup.py: 7 tests covering sentinel, absent containers,
  per-service project skip, main-stack removal, exception safety

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 15:57:45 -04:00
roof a10fe11136 feat: Phase 4 — dynamic nav + service visibility based on installed services
Unit Tests / test (push) Successful in 11m24s
Email, calendar, and files no longer appear in the nav or as usable pages
unless they are installed. The nav refreshes whenever a service is installed
or removed via the new pic-services-changed CustomEvent.

Changes:
- routes/services.py: add GET /api/services/active endpoint
- api.js: add servicesAPI.listActive()
- App.jsx: replace hardcoded coreServiceChildren with dynamic state fetched
  from /api/services/active; SERVICE_META maps ids to nav entry shapes
- ServiceNotInstalledBanner.jsx: new component — admin gets catalog link,
  peer gets "contact admin" message
- EmailPage/CalendarPage/FilesPage: show banner when service not installed
- ServicesIndex.jsx: remove CoreServiceCard + CORE_SERVICES "Built-in"
  section; rename Remove → Uninstall; dispatch pic-services-changed on
  install/uninstall success
- MyServices.jsx: conditionally render service cards based on active list;
  placeholder card when absent; page-level notice when nothing is installed
- tests/test_services_active_endpoint.py: 4 new endpoint tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 12:15:02 -04:00
roof 87c321c1c9 feat: Phase 3 — ServiceComposer deps + store install via per-service compose
Unit Tests / test (push) Successful in 11m21s
ServiceStoreManager.install() now delegates container lifecycle to
ServiceComposer (per-service docker-compose.yml) instead of appending to a
shared compose override. This eliminates IP pool allocation, compose override
rendering, and the single-stack docker exec approach.

Changes:
- service_composer.py: add _resolve_requires(), _resolve_dependents(),
  reapply_active_services() — dependency graph and startup reapply
- service_store_manager.py: rewrite install() and remove() to use
  ServiceComposer; add _fetch_template(); delete _allocate_service_ip(),
  _render_compose_override(), _write_compose_override(); remove() now guards
  against removing services that others depend on
- managers.py: pass service_composer= to ServiceStoreManager
- Tests: 13 new composer dep tests; TestInstall/TestRemove rewritten for
  the new composer-driven path; test_optional_services_feature.py updated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 09:33:02 -04:00
roof 0bfe95320b feat: Phase 2 — remove builtins layer, ServiceRegistry is installed-only
Unit Tests / test (push) Successful in 11m31s
Builtins (email/calendar/files) are no longer baked into the API image.
ServiceRegistry now only knows about installed store services. When nothing
is installed, Caddy and DNS get no service routes — no hardcoded fallback.

Changes:
- service_registry.py: remove _BUILTINS_DIR, _builtin_ids, _builtin_manifest,
  _load_manifest; get() and list_all() now delegate entirely to installed services
- caddy_manager.py: remove _build_core_service_routes(); remove hardcoded
  fallback pairs from _http01_service_pairs(); empty registry → api block only
- network_manager.py: _get_service_subdomains() returns [] when no registry
- api/services/builtins/: deleted (email, calendar, files manifests)
- Tests updated throughout: removed builtin-dependent assertions, added
  installed-service fixtures, updated fallback expectations to api-only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 08:53:44 -04:00
roof 18b50d08c1 fix: post-Phase-0 corrections — data-dir bind mounts, reserved subdomains, list_active()
Unit Tests / test (push) Successful in 11m31s
Three related fixes discovered during review of Phase 0 and Phase 1 manifests:

1. validate_rendered_compose(): add allowed_data_dir param. After ${PIC_DATA_DIR}
   substitution, compose templates produce absolute paths; without this the
   validator would reject every service install.  ServiceComposer.write_compose()
   now passes its resolved data_dir so only the designated data directory is
   exempt — /etc, /proc, docker.sock etc. still blocked.

2. _RESERVED_SUBDOMAINS: remove service-level subdomains (mail, calendar, files,
   webdav, webmail). The reserved list should protect PIC infrastructure endpoints
   (api, webui, admin) — not service subdomains that official store services
   (calendar, files, webmail) must be allowed to claim.  Aligns with the
   existing _RESERVED_SUBS in service_registry.py.

3. ServiceRegistry.list_active(): new method returning only installed store
   services (no builtins). This is the forward-looking API that Phase 2 will
   make the primary read path once builtins are deleted. Adding it now unblocks
   the QA agent's test_optional_services_feature.py which was already testing
   the expected Phase 2 behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 07:35:43 -04:00
roof c40919d374 feat: Phase 0 — manifest_validator, compose YAML safety check, cap_add allowlist, backend denylist, provision hook enforcement, size cap
Introduces api/manifest_validator.py as a single security chokepoint
imported by both ServiceComposer and ServiceStoreManager:

- validate_manifest(): rejects kind=builtin, reserved container names,
  reserved subdomains, backend denylist (localhost, cell-api, etc.),
  cap_add outside allowlist / in denylist, shell-string provision hooks,
  and env values with shell-special characters
- validate_rendered_compose(): walks the rendered YAML and rejects
  privileged:true, host network/pid/ipc/userns, absolute bind mounts,
  denied capabilities, devices key, apparmor/seccomp unconfined, and
  string-form command/entrypoint (shell-injection vector)
- validate_provision_hook(): requires argv list form, lowercase binary,
  rejects NUL bytes

ServiceStoreManager changes:
- _validate_manifest() delegates to validate_manifest() after existing checks
- _fetch_manifest() and fetch_index() now stream with a 256 KB size cap
  (prevents memory exhaustion from a malicious or compromised index)
- Digest-pin warning for images missing @sha256 (hard error for unknown
  registries, warning for git.pic.ngo/roof/* and TRUSTED_IMAGES_NO_DIGEST)

ServiceComposer changes:
- write_compose() calls validate_rendered_compose() before any disk write
  so no partial file is left if validation fails
- render_template() substitutes ${PIC_DATA_DIR} with the resolved data_dir path

102 new tests in tests/test_manifest_validator.py covering all five P0
security issues.  Existing test mocks updated to use streaming response
pattern (stream=True + raw.read) and valid compose YAML templates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 07:23:08 -04:00
roof 5e438aa991 fix: remove stray </div> in Email/Calendar/Files pages that broke vite build
Unit Tests / test (push) Successful in 11m27s
Stray closing div was left in the ternary falsy branch after AdminConfigSection
was moved outside the ternary. esbuild interpreted it as an unterminated regex.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 05:10:52 -04:00
roof c20906d6cc feat: PIC Services Architecture Phase 1 — registry-driven services ecosystem
Unit Tests / test (push) Successful in 11m30s
Implements the full Phase 1 services architecture:
- ServiceRegistry: merges built-in + installed + runtime config; drives Caddy and CoreDNS instead of hardcoded service names
- ServiceComposer: docker-compose lifecycle for third-party services
- AccountManager: per-service credential provisioning and deprovisioning per peer
- Built-in manifests (email, calendar, files) with subdomain, backup, and account hooks
- Admin UI: Accounts tab on Email, Calendar, Files pages
- Developer guide v1: manifest reference, compose variables, backup/egress integration
- 158 new tests; 1762 total passing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 05:02:26 -04:00
roof 2f5370bd98 feat: add Steps 1-4 implementation files (AccountManager, ServiceComposer, builtins, tests)
Unit Tests / test (push) Successful in 11m24s
These files were created during Steps 1-4 of the services architecture but were
never staged: AccountManager (per-service credential provisioning), ServiceComposer
(docker-compose lifecycle), built-in service manifests for email/calendar/files,
and their test suites (158 tests). Also un-tracks .coverage binaries that were
accidentally committed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 04:39:19 -04:00
roof dc7b316cbd docs: correct Step 7 developer guide to match Steps 3-6 implementation
Unit Tests / test (push) Failing after 11s
Steps 3-6 were implemented since this doc was last written. Several
technical details had drifted from the actual code:

- Provision response shape was shown as echoing the password; corrected
  to {provisioned: true} to match the security model (passwords are
  never returned after creation)
- Restore command flag corrected from -C / to -C <path>; archives use
  relative paths so the extraction target must be explicit
- Added ServiceRegistry validation chokepoint note: subdomain and
  backend are validated at registration time, before Caddyfile
  generation, not at request time
- Added Admin UI note: Accounts tab appears on service pages
- Added -- separator security note for backup command construction

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 03:10:43 -04:00
roof ad5731073d feat: Admin UI — Accounts tab on service pages (Step 6)
Unit Tests / test (push) Failing after 11s
Admins previously had no UI path to provision per-peer accounts for
email, calendar, and files: they had to hit the AccountManager API
routes directly.  This change wires those routes to a dedicated Accounts
tab on each service page so any peer can be granted or revoked service
access in two clicks.

- webui/src/services/api.js: add accountsAPI with list/provision/
  deprovision/getCredentials, pointing to
  /api/services/catalog/{serviceId}/accounts
- webui/src/components/ServiceAccountsPanel.jsx: new reusable panel;
  handles credential reveal, removal confirmation, load-error state,
  and humanized credential labels
- EmailPage, CalendarPage, FilesPage: Overview/Accounts tab nav (admin
  only); Accounts tab renders ServiceAccountsPanel; AdminConfigSection
  is hidden while on the Accounts tab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 20:29:57 -04:00
roof 16fb362df7 feat: replace hardcoded service names with ServiceRegistry-driven Caddy and CoreDNS config
Unit Tests / test (push) Failing after 11s
Previously, CaddyManager and NetworkManager contained hardcoded lists of
service names (calendar, files, mail, webdav, etc.), meaning every new
service required a code change to appear in Caddy routes and DNS records.
Now both managers accept a service_registry parameter and derive their
service lists dynamically from the registry at runtime.

- CaddyManager: new _build_registry_service_routes() and
  _http01_service_pairs() methods pull routes from the registry
- NetworkManager: new _get_service_subdomains() method returns registry
  subdomains with a hardcoded fallback when no registry is wired in;
  _build_dns_records, stale-record detection, and service name sets all
  use the registry
- managers.py: service_registry constructed before network_manager so it
  can be injected into both CaddyManager and NetworkManager
- service_registry.py: validation chokepoint in get_caddy_routes() rejects
  invalid subdomain/backend values and reserved service names
- service_store_manager.py: _validate_manifest now validates top-level
  subdomain, backend, extra_subdomains, and extra_backends fields
- tests: 24 new tests covering registry-driven routing and DNS subdomain
  generation (test_caddy_registry_integration.py)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 18:27:52 -04:00
roof 63c0dfb9d9 docs: document Services UI refactor in wiki
Unit Tests / test (push) Successful in 11m29s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 06:58:24 -04:00