roof/pic

Fork 0

Table of Contents

Admin – Monitor and Troubleshoot

Checking overall health
Common problems

API returns 428 and redirects to /setup
API returns 401 / "Not authenticated"
API returns 503 "Authentication not configured"
Forgot the admin password
Containers not starting
Port 53 already in use
WireGuard container fails to start
VPN connects but no DNS
HTTPS not working / certificate error
Domain resolves to wrong IP
Service not loading after install
Cell-to-cell traffic not routing

Opening a shell in a container
Checking WireGuard routing tables
Full reinstall

Status: Active | Owner: @roof | Applies to: main (2026-06) | Updated: 2026-06-11

Admin – Monitor and Troubleshoot

Checking overall health

# Run on: the cell server host, from /opt/pic
make status    # container status + API health check at /health

The /health endpoint is always public (no auth required). It returns a JSON object with the running state of each core component. A background thread in the API re-checks health every 30 seconds and keeps a recent history:

GET /health
GET /api/health/history
GET /api/status

Common problems

API returns 428 and redirects to /setup

The first-run wizard has not been completed. Open http://<host-ip>:8081 in a browser and finish the wizard.

API returns 401 / "Not authenticated"

Your session expired. Open http://<host-ip>:8081/login.

API returns 503 "Authentication not configured"

The auth file exists but is empty. This should not happen in normal operation, but if it does:

# Run on: the cell server host, from /opt/pic
make reset-admin-password

This prints a new random admin password.

Forgot the admin password

# Run on: the cell server host, from /opt/pic
make show-admin-password    # print the current password
make reset-admin-password   # generate and set a new random password

Containers not starting

# Run on: the cell server host, from /opt/pic
make logs
make logs-api

Look for error messages about missing config files, port conflicts, or Docker socket errors.

Port 53 already in use

On Ubuntu and Debian, systemd-resolved listens on port 53. Disable it:

# Run on: the cell server host, as root
sudo systemctl disable --now systemd-resolved
sudo rm /etc/resolv.conf
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Then run make start.

WireGuard container fails to start

The WireGuard container runs unprivileged (NET_ADMIN capability only — not full privileged mode). It requires the WireGuard kernel module on the host. Load it manually:

# Run on: the cell server host, as root
sudo modprobe wireguard

On kernels without a built-in WireGuard module, install the appropriate package for your distribution (wireguard-dkms or wireguard-linux-compat).

VPN connects but no DNS

Split-horizon DNS is not working. Check:

The WireGuard client config includes the cell's VPN IP (10.0.0.1) as the DNS server.
CoreDNS is running: make logs-dns.
The zone file for your cell domain exists: make shell-dns then check for zone files.

HTTPS not working / certificate error

Check the TLS certificate status:
```
GET /api/caddy/cert-status
```
If the cert is expired or missing, check the host clock: date. If it is wrong, fix chrony — see Admin – Configure Domains and TLS.
For lan mode, clients need to trust the internal CA. Download it from the Vault page and add it to the client's trust store.
For pic_ngo / cloudflare / duckdns, ports 80 and 443 must be reachable from the internet.

Domain resolves to wrong IP

The DDNS heartbeat runs every 5 minutes. If your IP changed recently, wait a few minutes and check again. You can force a refresh from the Network Services page, or:

GET /api/dns/refresh

Service not loading after install

Check the service's container status:

GET /api/services/catalog/<id>/status

Check the service logs from the Logs page in the UI (select the service) or via the API:

GET /api/logs?service=<service_name>

Cell-to-cell traffic not routing

Check:

The cell link is active and the WireGuard handshake is recent: check the Cell Network page.
The iptables FORWARD rules are in place: check from the Routing page or the FirewallManager log.
There is no exit-routing loop (peer A assigned to a cell-relay to cell B, which routes back to A).

Opening a shell in a container

# Run on: the cell server host, from /opt/pic
make shell-api        # Flask API container
make shell-dns        # CoreDNS container
make shell-wireguard  # WireGuard container (useful for wg show, ip rule list)
make shell-caddy      # Caddy container

Checking WireGuard routing tables

Inside the WireGuard container:

# Run on: the cell server host, from /opt/pic
make shell-wireguard
# then, inside the container:
wg show
ip rule list
ip route list table <table-number>

Connectivity routing tables start from 1000. Each named connection instance uses a dedicated table and fwmark.

Full reinstall

If the cell is in an unrecoverable state and you have a backup:

# Run on: the cell server host, from /opt/pic
make uninstall    # stop and optionally wipe config/ and data/
tar -xzf backups/cell-backup-YYYYMMDD-HHMMSS.tar.gz
make start

To start completely fresh (no restore):

# Run on: the cell server host, from /opt/pic
make reinstall    # full wipe of config/ and data/, then setup + start

⚠️ Warning: make reinstall destroys all data. Run make backup first if you have anything worth keeping.

Deeper: see User – Troubleshooting for end-user-facing problems to direct your users to.

Personal Internet Cell

Home