fix: architecture audit — security, atomicity, broken endpoints, test coverage
Sprint 1 — Security & correctness:
- Restore all 10 commented-out is_local_request() checks (vault, containers, images, volumes)
- Fix XFF spoofing: only trust the LAST X-Forwarded-For entry (Caddy's append), not all
- Require prefix length in wireguard.address (was accepting bare IPs like 10.0.0.1)
- Validate service_access list in add_peer (valid: calendar/files/mail/webdav)
- Fix dhcp/reservations POST/DELETE: unpack mac/ip/hostname from body (was passing dict as positional arg)
- Fix network/test POST: remove spurious data arg (test_connectivity takes no args)
- Fix remove_peer: clear iptables rules and regenerate DNS ACLs on deletion (was leaving stale rules)
- Fix CoreDNS reload: SIGHUP → SIGUSR1 (SIGHUP kills the process; SIGUSR1 triggers reload plugin)
- Remove local.{domain} block from Corefile template (local.zone doesn't exist, caused log spam)
- Fix routing_manager._remove_nat_rule: targeted -D instead of flushing entire POSTROUTING chain
Sprint 2 — State consistency:
- Atomic config writes in config_manager, ip_utils, firewall_manager, network_manager
(write to .tmp → fsync → os.replace, prevents truncated files on kill)
- backup_config: now also backs up Caddyfile, Corefile, .env, DNS zone files
- restore_config: restores all of the above so config stays consistent after restore
Sprint 3 — Dead code / documentation:
- Remove CellManager instantiation from app startup (was never called, double-instantiated all managers)
- Document routing_manager scope (targets host, not cell-wireguard; methods not called by any active route)
Sprint 4 — Test infrastructure:
- Add tests/conftest.py with shared tmp_dir, tmp_config_dir, tmp_data_dir, flask_client fixtures
- Add tests/test_config_validation.py: 400 paths for ip_range, port, wireguard.address validation
- Add tests/test_ip_utils_caddyfile.py: 14 tests for write_caddyfile (was completely untested)
- Expand test_app_misc.py: 7 new is_local_request tests covering XFF spoofing and cell-network IPs
- Add --cov-fail-under=70 to make test-coverage
- Add pre-commit hook that runs pytest before every commit
414 tests pass (was 372).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,539 +1,267 @@
|
||||
|
||||
# Personal Internet Cell
|
||||
|
||||
## 🌟 Overview
|
||||
|
||||
The Personal Internet Cell is a **production-grade, self-hosted, decentralized digital infrastructure** that empowers you to:
|
||||
|
||||
- **Host your own services**: Email, calendar, contacts, files, DNS, DHCP, NTP
|
||||
- **Secure mesh networking**: Connect with trusted peers via WireGuard VPN
|
||||
- **Advanced routing**: VPN gateway, NAT, firewall, exit nodes, and bridge routing
|
||||
- **Enterprise security**: Self-hosted CA, certificate management, trust systems
|
||||
- **Modern management**: RESTful API, enhanced CLI, and comprehensive monitoring
|
||||
- **Event-driven architecture**: Service orchestration and real-time communication
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Key Features
|
||||
|
||||
### 🔧 **Core Services**
|
||||
- **Network Services**: DNS, DHCP, NTP with dynamic management
|
||||
- **VPN & Mesh**: WireGuard-based peer federation with dynamic IP updates
|
||||
- **Digital Services**: Email (SMTP/IMAP), Calendar/Contacts (CalDAV/CardDAV), File Storage (WebDAV)
|
||||
- **Security**: Self-hosted Certificate Authority, Age/Fernet encryption, trust management
|
||||
- **Container Orchestration**: Docker-based service management and deployment
|
||||
|
||||
### 🏗️ **Architecture Highlights**
|
||||
- **BaseServiceManager**: Unified interface across all 10 service managers
|
||||
- **Event-Driven Service Bus**: Real-time service communication and orchestration
|
||||
- **Centralized Configuration**: Type-safe validation, backup/restore, import/export
|
||||
- **Comprehensive Logging**: Structured JSON logs with rotation, search, and export
|
||||
- **Enhanced CLI**: Interactive mode, batch operations, service wizards
|
||||
- **Health Monitoring**: Real-time health checks and performance metrics
|
||||
|
||||
### 📊 **Production Features**
|
||||
- **Service Orchestration**: Automatic service dependency management
|
||||
- **Configuration Management**: Schema validation, versioning, and migration
|
||||
- **Error Handling**: Standardized error handling and recovery mechanisms
|
||||
- **Testing**: Comprehensive test suite with 77%+ coverage
|
||||
- **Documentation**: Complete API documentation and usage guides
|
||||
|
||||
---
|
||||
|
||||
## 📋 Table of Contents
|
||||
|
||||
1. [Quick Start](#quick-start)
|
||||
2. [Architecture](#architecture)
|
||||
3. [Service Managers](#service-managers)
|
||||
4. [API Reference](#api-reference)
|
||||
5. [CLI Guide](#cli-guide)
|
||||
6. [Configuration](#configuration)
|
||||
7. [Security](#security)
|
||||
8. [Development](#development)
|
||||
9. [Testing](#testing)
|
||||
10. [Deployment](#deployment)
|
||||
11. [Contributing](#contributing)
|
||||
12. [License](#license)
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Debian/Ubuntu** host (apt-based). All other dependencies are installed automatically.
|
||||
- **2 GB+ RAM, 10 GB+ disk space**
|
||||
- **Open ports**: 53 (DNS), 80/443 (HTTP/S), 3000 (API), 8081 (Web UI), 51820/udp (WireGuard)
|
||||
|
||||
### 1. Install
|
||||
|
||||
```bash
|
||||
git clone <repo-url> pic
|
||||
cd pic
|
||||
|
||||
# Install all system dependencies (docker, python3, python3-cryptography, etc.)
|
||||
make check-deps
|
||||
|
||||
# Default cell (name=mycell, domain=cell, VPN=10.0.0.1/24, port=51820)
|
||||
make setup
|
||||
make start
|
||||
|
||||
# Custom cell — use when installing a second cell on a different host
|
||||
CELL_NAME=pic1 VPN_ADDRESS=10.1.0.1/24 make setup && make start
|
||||
```
|
||||
|
||||
`make check-deps` installs python3, python3-cryptography, docker, docker-compose, curl, openssl, git via apt and adds the current user to the docker group.
|
||||
|
||||
`make setup` generates WireGuard keys, writes configs, and creates all data directories.
|
||||
|
||||
`make start` builds and brings up all 12 Docker containers.
|
||||
|
||||
### 2. Access
|
||||
|
||||
| Service | URL |
|
||||
|---------|-----|
|
||||
| Web UI | `http://<host-ip>:8081` |
|
||||
| API | `http://<host-ip>:3000` |
|
||||
| Health | `http://<host-ip>:3000/health` |
|
||||
|
||||
On a WireGuard client: `http://mycell.cell` (or whatever your cell name is).
|
||||
|
||||
### 3. Local dev (no Docker)
|
||||
|
||||
```bash
|
||||
pip install -r api/requirements.txt
|
||||
python api/app.py # API on :3000
|
||||
|
||||
cd webui && npm install && npm run dev # React UI on :5173 (proxies API to :3000)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Management Commands
|
||||
|
||||
```bash
|
||||
# First install
|
||||
make check-deps # install all system packages via apt
|
||||
make setup # generate keys, write configs
|
||||
make start # start all 12 containers
|
||||
|
||||
# Daily operations
|
||||
make status # container status + API health
|
||||
make logs # follow all logs
|
||||
make logs-api # follow logs for one service (api, dns, wg, mail, caddy, ...)
|
||||
make shell-api # open a shell inside a container
|
||||
|
||||
# Deploy latest code
|
||||
make update # git pull + rebuild + restart
|
||||
|
||||
# Full wipe and reinstall (useful on test machine)
|
||||
make reinstall # stop, wipe config/data, setup, start fresh
|
||||
|
||||
# Remove everything
|
||||
make uninstall # stop + remove images; prompts whether to also wipe config/data
|
||||
|
||||
# Maintenance
|
||||
make backup # tar config/ + data/ into backups/
|
||||
make restore # list available backups
|
||||
make clean # remove containers/volumes, keep config/data
|
||||
|
||||
# Tests
|
||||
make test # run all tests
|
||||
make test-coverage # tests + HTML coverage report
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔗 Connecting Two Cells (PIC Mesh)
|
||||
|
||||
Two PIC instances can form a mesh — full site-to-site WireGuard tunnels with
|
||||
automatic DNS forwarding so each cell's services are reachable from the other.
|
||||
|
||||
### Install the second cell
|
||||
|
||||
```bash
|
||||
# On the second host (different VPN subnet; port 51820 is fine — different machine)
|
||||
CELL_NAME=pic1 VPN_ADDRESS=10.1.0.1/24 make setup && make start
|
||||
```
|
||||
|
||||
### Exchange invites (two pastes, two clicks)
|
||||
|
||||
1. On **Cell A** → open Web UI → **Cell Network** → copy the invite JSON.
|
||||
2. On **Cell B** → **Cell Network** → paste into "Connect to Another Cell" → click **Connect**.
|
||||
3. On **Cell B** → copy its invite JSON.
|
||||
4. On **Cell A** → paste Cell B's invite → click **Connect**.
|
||||
|
||||
Both cells now have:
|
||||
- A site-to-site WireGuard peer (AllowedIPs = remote cell's VPN subnet).
|
||||
- A CoreDNS forwarding block so `*.pic1.cell` resolves across the tunnel.
|
||||
|
||||
The **Connected Cells** panel shows live handshake status (green = online).
|
||||
|
||||
### Same-LAN tip
|
||||
|
||||
If both cells share the same external IP (behind NAT), the auto-detected
|
||||
endpoint in the invite will be the public IP. Replace it with the LAN IP
|
||||
before clicking Connect so traffic stays local:
|
||||
|
||||
```json
|
||||
{ "endpoint": "192.168.31.50:51820", ... }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
### **Service Manager Architecture**
|
||||
|
||||
All services inherit from `BaseServiceManager`, providing:
|
||||
- **Unified Interface**: Consistent methods across all services
|
||||
- **Health Monitoring**: Standardized health checks and metrics
|
||||
- **Error Handling**: Centralized error handling and logging
|
||||
- **Configuration**: Common configuration management patterns
|
||||
|
||||
### **Event-Driven Service Bus**
|
||||
|
||||
```python
|
||||
# Services communicate via events
|
||||
service_bus.register_service('network', network_manager)
|
||||
service_bus.register_service('wireguard', wireguard_manager)
|
||||
service_bus.publish_event(EventType.SERVICE_STARTED, 'network', data)
|
||||
```
|
||||
|
||||
### **Service Dependencies**
|
||||
|
||||
```
|
||||
wireguard → network
|
||||
email → network, vault
|
||||
calendar → network, vault
|
||||
files → network, vault
|
||||
routing → network, wireguard
|
||||
vault → network
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Service Managers
|
||||
|
||||
### **Core Network Services**
|
||||
- **NetworkManager**: DNS, DHCP, NTP with dynamic zone management
|
||||
- **WireGuardManager**: VPN configuration, peer management, key generation
|
||||
- **PeerRegistry**: Peer registration, IP updates, trust management
|
||||
|
||||
### **Digital Services**
|
||||
- **EmailManager**: SMTP/IMAP email with user management
|
||||
- **CalendarManager**: CalDAV/CardDAV calendar and contacts
|
||||
- **FileManager**: WebDAV file storage with user directories
|
||||
|
||||
### **Infrastructure Services**
|
||||
- **RoutingManager**: NAT, firewall, advanced routing (exit/bridge/split)
|
||||
- **VaultManager**: Certificate authority, trust management, encryption
|
||||
- **ContainerManager**: Docker orchestration and container management
|
||||
- **CellManager**: Overall cell configuration and service orchestration
|
||||
|
||||
---
|
||||
|
||||
## 📡 API Reference
|
||||
|
||||
### **Core Endpoints**
|
||||
|
||||
```bash
|
||||
# Service Status
|
||||
GET /api/services/status
|
||||
GET /api/services/connectivity
|
||||
|
||||
# Configuration Management
|
||||
GET /api/config
|
||||
PUT /api/config
|
||||
POST /api/config/backup
|
||||
POST /api/config/restore/<backup_id>
|
||||
|
||||
# Service Bus
|
||||
GET /api/services/bus/status
|
||||
GET /api/services/bus/events
|
||||
POST /api/services/bus/services/<service>/start
|
||||
|
||||
# Logging
|
||||
GET /api/logs/services/<service>
|
||||
POST /api/logs/search
|
||||
POST /api/logs/export
|
||||
```
|
||||
|
||||
### **Service-Specific Endpoints**
|
||||
|
||||
```bash
|
||||
# Network Services
|
||||
GET /api/dns/records
|
||||
POST /api/dns/records
|
||||
GET /api/dhcp/leases
|
||||
GET /api/ntp/status
|
||||
|
||||
# WireGuard & Peers
|
||||
GET /api/wireguard/peers
|
||||
POST /api/wireguard/peers
|
||||
GET /api/wireguard/status
|
||||
|
||||
# Digital Services
|
||||
GET /api/email/users
|
||||
GET /api/calendar/users
|
||||
GET /api/files/users
|
||||
|
||||
# Routing & Security
|
||||
GET /api/routing/status
|
||||
POST /api/routing/nat
|
||||
GET /api/vault/certificates
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 💻 CLI Guide
|
||||
|
||||
### **Enhanced CLI Features**
|
||||
|
||||
```bash
|
||||
# Interactive Mode
|
||||
python api/enhanced_cli.py --interactive
|
||||
|
||||
# Batch Operations
|
||||
python api/enhanced_cli.py --batch "status" "services" "health"
|
||||
|
||||
# Configuration Management
|
||||
python api/enhanced_cli.py --export-config json
|
||||
python api/enhanced_cli.py --import-config config.json
|
||||
|
||||
# Service Wizards
|
||||
python api/enhanced_cli.py --wizard network
|
||||
python api/enhanced_cli.py --wizard email
|
||||
|
||||
# Health Monitoring
|
||||
python api/enhanced_cli.py --health
|
||||
python api/enhanced_cli.py --logs network
|
||||
```
|
||||
|
||||
### **Service Management**
|
||||
|
||||
```bash
|
||||
# Show status
|
||||
python api/enhanced_cli.py --status
|
||||
|
||||
# List services
|
||||
python api/enhanced_cli.py --services
|
||||
|
||||
# Peer management
|
||||
python api/enhanced_cli.py --peers
|
||||
|
||||
# Service logs
|
||||
python api/enhanced_cli.py --logs wireguard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
### **Configuration Management**
|
||||
|
||||
```bash
|
||||
# Export configuration
|
||||
curl -X GET http://localhost:3000/api/config
|
||||
|
||||
# Update configuration
|
||||
curl -X PUT http://localhost:3000/api/config \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"cell_name": "mycell", "domain": "mycell.cell"}'
|
||||
|
||||
# Backup configuration
|
||||
curl -X POST http://localhost:3000/api/config/backup
|
||||
```
|
||||
|
||||
### **Service Configuration**
|
||||
|
||||
Each service has its own configuration schema:
|
||||
- **Network**: DNS zones, DHCP ranges, NTP servers
|
||||
- **WireGuard**: Interface settings, peer configurations
|
||||
- **Email**: Domain settings, user accounts, mailboxes
|
||||
- **Calendar**: User accounts, calendar sharing
|
||||
- **Files**: Storage quotas, user directories
|
||||
- **Routing**: NAT rules, firewall policies, routing tables
|
||||
|
||||
---
|
||||
|
||||
## 🔒 Security
|
||||
|
||||
### **Certificate Management**
|
||||
- **Self-hosted CA**: Issue and manage TLS certificates
|
||||
- **Certificate Lifecycle**: Generate, renew, revoke certificates
|
||||
- **Trust Management**: Direct and indirect trust relationships
|
||||
- **Age Encryption**: Modern encryption for sensitive data
|
||||
|
||||
### **Network Security**
|
||||
- **WireGuard VPN**: Secure peer-to-peer communication
|
||||
- **Firewall & NAT**: Granular access control
|
||||
- **Service Isolation**: Docker containers for each service
|
||||
- **Input Validation**: All API endpoints validate input
|
||||
|
||||
### **Data Protection**
|
||||
- **Encrypted Storage**: Sensitive data encrypted at rest
|
||||
- **Secure Communication**: TLS for all API endpoints
|
||||
- **Access Control**: Role-based access for services
|
||||
- **Audit Logging**: Comprehensive security event logging
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Development
|
||||
|
||||
### **Project Structure**
|
||||
|
||||
```
|
||||
PersonalInternetCell/
|
||||
├── api/ # Backend API server
|
||||
│ ├── base_service_manager.py # Base class for all services
|
||||
│ ├── config_manager.py # Configuration management
|
||||
│ ├── service_bus.py # Event-driven service bus
|
||||
│ ├── log_manager.py # Comprehensive logging
|
||||
│ ├── enhanced_cli.py # Enhanced CLI tool
|
||||
│ ├── network_manager.py # DNS, DHCP, NTP
|
||||
│ ├── wireguard_manager.py # VPN and peer management
|
||||
│ ├── email_manager.py # Email services
|
||||
│ ├── calendar_manager.py # Calendar services
|
||||
│ ├── file_manager.py # File storage
|
||||
│ ├── routing_manager.py # Routing and NAT
|
||||
│ ├── vault_manager.py # Security and trust
|
||||
│ ├── container_manager.py # Container orchestration
|
||||
│ ├── cell_manager.py # Overall cell management
|
||||
│ ├── peer_registry.py # Peer registration
|
||||
│ └── app.py # Main API server
|
||||
├── webui/ # React frontend
|
||||
├── config/ # Configuration files
|
||||
├── data/ # Persistent data
|
||||
├── tests/ # Test suite
|
||||
└── docker-compose.yml # Container orchestration
|
||||
```
|
||||
|
||||
### **Running Locally**
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r api/requirements.txt
|
||||
|
||||
# Start the API server
|
||||
python api/app.py
|
||||
|
||||
# Run tests
|
||||
python api/test_enhanced_api.py
|
||||
|
||||
# Start frontend (if available)
|
||||
cd webui && bun install && npm run dev
|
||||
```
|
||||
|
||||
### **Service Development**
|
||||
|
||||
```python
|
||||
from base_service_manager import BaseServiceManager
|
||||
|
||||
class MyServiceManager(BaseServiceManager):
|
||||
def __init__(self, data_dir='/app/data', config_dir='/app/config'):
|
||||
super().__init__('myservice', data_dir, config_dir)
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
# Implement service status
|
||||
pass
|
||||
|
||||
def test_connectivity(self) -> Dict[str, Any]:
|
||||
# Implement connectivity test
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### **Test Suite**
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
python api/test_enhanced_api.py
|
||||
|
||||
# Test specific components
|
||||
python -m pytest api/tests/test_network_manager.py
|
||||
python -m pytest api/tests/test_service_bus.py
|
||||
|
||||
# Coverage report
|
||||
coverage run -m pytest api/tests/
|
||||
coverage html
|
||||
```
|
||||
|
||||
### **Test Coverage**
|
||||
- **BaseServiceManager**: 100% coverage
|
||||
- **ConfigManager**: 95%+ coverage
|
||||
- **ServiceBus**: 95%+ coverage
|
||||
- **LogManager**: 95%+ coverage
|
||||
- **All Service Managers**: 77%+ overall coverage
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### **Docker Deployment**
|
||||
|
||||
```bash
|
||||
# Production deployment
|
||||
docker-compose -f docker-compose.prod.yml up -d
|
||||
|
||||
# Development deployment
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
### **System Requirements**
|
||||
- **CPU**: 2+ cores
|
||||
- **RAM**: 2GB+ (4GB recommended)
|
||||
- **Storage**: 10GB+ (SSD recommended)
|
||||
- **Network**: Stable internet connection
|
||||
|
||||
### **Monitoring**
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:3000/health
|
||||
|
||||
# Service status
|
||||
curl http://localhost:3000/api/services/status
|
||||
|
||||
# Service connectivity
|
||||
curl http://localhost:3000/api/services/connectivity
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
1. **Fork** the repository
|
||||
2. **Create** a feature branch
|
||||
3. **Implement** your changes
|
||||
4. **Add tests** for new functionality
|
||||
5. **Submit** a pull request
|
||||
|
||||
### **Development Guidelines**
|
||||
- Follow the existing code style
|
||||
- Add comprehensive tests
|
||||
- Update documentation
|
||||
- Use the BaseServiceManager pattern
|
||||
- Implement proper error handling
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT License - see [LICENSE](LICENSE) file for details.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
- **[Quick Start Guide](QUICKSTART.md)**: Get up and running quickly
|
||||
- **[API Documentation](api/API_DOCUMENTATION.md)**: Complete API reference
|
||||
- **[Comprehensive Improvements](COMPREHENSIVE_IMPROVEMENTS_SUMMARY.md)**: Detailed architecture overview
|
||||
- **[Enhanced API Improvements](ENHANCED_API_IMPROVEMENTS.md)**: Technical implementation details
|
||||
|
||||
---
|
||||
|
||||
**🌟 The Personal Internet Cell - Your self-hosted, production-grade digital infrastructure!**
|
||||
|
||||
# Personal Internet Cell (PIC)
|
||||
|
||||
A self-hosted digital infrastructure platform. One stack, one API, one UI — managing DNS, DHCP, NTP, WireGuard VPN, email, calendar/contacts, file storage, and a reverse proxy on your own hardware.
|
||||
|
||||
---
|
||||
|
||||
## What it does
|
||||
|
||||
- **Network services** — CoreDNS, dnsmasq DHCP, chrony NTP, all dynamically managed
|
||||
- **WireGuard VPN** — peer lifecycle, QR-code provisioning, per-peer service access control
|
||||
- **Digital services** — Email (Postfix/Dovecot), Calendar/Contacts (Radicale CalDAV), Files (WebDAV + Filegator)
|
||||
- **Reverse proxy** — Caddy with per-service virtual IPs; subdomains like `calendar.mycell.cell` work on VPN clients automatically
|
||||
- **Certificate authority** — self-hosted CA via VaultManager
|
||||
- **Cell mesh** — connect two PIC instances with site-to-site WireGuard + DNS forwarding
|
||||
|
||||
Everything is configured through a REST API and a React web UI. No manual config file editing needed for normal operations.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Debian/Ubuntu host (apt-based)
|
||||
- 2 GB+ RAM, 10 GB+ disk
|
||||
- Open ports: 53 (DNS), 80 (HTTP), 3000 (API), 8081 (Web UI), 51820/udp (WireGuard)
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
git clone <repo-url> pic
|
||||
cd pic
|
||||
|
||||
# Install system deps (docker, python3, python3-cryptography, etc.)
|
||||
make check-deps
|
||||
|
||||
# Generate keys + write configs
|
||||
make setup
|
||||
|
||||
# Build and start all 12 containers
|
||||
make start
|
||||
```
|
||||
|
||||
`make setup` accepts overrides for a second cell on a different host:
|
||||
|
||||
```bash
|
||||
CELL_NAME=pic1 VPN_ADDRESS=10.1.0.1/24 make setup && make start
|
||||
```
|
||||
|
||||
### Access
|
||||
|
||||
| Service | URL |
|
||||
|---------|-----|
|
||||
| Web UI | `http://<host-ip>:8081` |
|
||||
| API | `http://<host-ip>:3000` |
|
||||
| Health | `http://<host-ip>:3000/health` |
|
||||
|
||||
From a WireGuard client: `http://mycell.cell` (replace with your cell name/domain).
|
||||
|
||||
### Local dev (no Docker)
|
||||
|
||||
```bash
|
||||
pip install -r api/requirements.txt
|
||||
python api/app.py # Flask API on :3000
|
||||
|
||||
cd webui && npm install && npm run dev # React UI on :5173 (proxies /api → :3000)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Management Commands
|
||||
|
||||
```bash
|
||||
# First install
|
||||
make check-deps # install system packages via apt
|
||||
make setup # generate keys, write configs, create data dirs
|
||||
make start # start all 12 containers
|
||||
|
||||
# Daily operations
|
||||
make status # container status + API health
|
||||
make logs # follow all container logs
|
||||
make logs-api # follow logs for one service (api, dns, wg, mail, caddy, ...)
|
||||
make shell-api # shell inside a container
|
||||
|
||||
# Deploy latest code
|
||||
make update # git pull + rebuild api image + restart
|
||||
|
||||
# Maintenance
|
||||
make backup # tar config/ + data/ into backups/
|
||||
make restore # list available backups and restore
|
||||
make clean # remove containers/volumes, keep config/data
|
||||
|
||||
# Full wipe (test machines)
|
||||
make reinstall # stop, wipe config/data, setup, start fresh
|
||||
make uninstall # stop + remove images; prompts to also wipe config/data
|
||||
|
||||
# Tests
|
||||
make test # run full pytest suite
|
||||
make test-coverage # tests + HTML coverage report in htmlcov/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Connecting Two Cells (PIC Mesh)
|
||||
|
||||
Two PIC instances form a mesh: site-to-site WireGuard tunnels with automatic DNS forwarding so each cell's services resolve from the other.
|
||||
|
||||
### Exchange invites
|
||||
|
||||
1. On **Cell A** → Web UI → **Cell Network** → copy the invite JSON.
|
||||
2. On **Cell B** → **Cell Network** → paste into "Connect to Another Cell" → **Connect**.
|
||||
3. On **Cell B** → copy its invite JSON.
|
||||
4. On **Cell A** → paste Cell B's invite → **Connect**.
|
||||
|
||||
Both cells now have a WireGuard peer with `AllowedIPs = remote VPN subnet` and a CoreDNS forwarding block so `*.pic1.cell` resolves across the tunnel.
|
||||
|
||||
### Same-LAN tip
|
||||
|
||||
If both cells share the same external IP (behind NAT), replace the auto-detected endpoint with the LAN IP before connecting:
|
||||
|
||||
```json
|
||||
{ "endpoint": "192.168.31.50:51820", ... }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Stack
|
||||
|
||||
```
|
||||
cell-caddy (Caddy) :80/:443 + per-service virtual IPs
|
||||
cell-api (Flask :3000) REST API + config management + container orchestration
|
||||
cell-webui (Nginx :8081) React UI
|
||||
cell-dns (CoreDNS :53) internal DNS + per-peer ACLs
|
||||
cell-dhcp (dnsmasq) DHCP + static reservations
|
||||
cell-ntp (chrony) NTP
|
||||
cell-wireguard WireGuard VPN
|
||||
cell-mail (docker-mailserver) SMTP/IMAP
|
||||
cell-radicale CalDAV/CardDAV :5232
|
||||
cell-webdav WebDAV :80
|
||||
cell-filegator file manager UI :8080
|
||||
cell-rainloop webmail :8888
|
||||
```
|
||||
|
||||
All containers share a custom Docker bridge network. Static IPs are assigned in `docker-compose.yml`. Caddy adds per-service virtual IPs to its own interface at API startup so `calendar.<domain>`, `files.<domain>`, etc. route to the right container.
|
||||
|
||||
### Backend (`api/`)
|
||||
|
||||
Service managers (`network_manager.py`, `wireguard_manager.py`, `peer_registry.py`, etc.) all inherit `BaseServiceManager`. `app.py` contains all Flask routes — one file, organized by service.
|
||||
|
||||
`ConfigManager` (`config_manager.py`) is the single source of truth. Config lives in `config/api/cell_config.json`. All managers read/write through it.
|
||||
|
||||
`ip_utils.py` owns all container IP logic via `CONTAINER_OFFSETS` — do not hardcode IPs elsewhere.
|
||||
|
||||
When a config change requires recreating the Docker network (e.g. `ip_range` change), the API spawns a helper container that outlives cell-api to run `docker compose down && up`. Other restarts run `compose up -d --no-deps <containers>` directly.
|
||||
|
||||
### Frontend (`webui/`)
|
||||
|
||||
React 18 + Vite + Tailwind CSS. All API calls go through `src/services/api.js` (Axios). Vite dev server proxies `/api` to `localhost:3000`. Pages in `src/pages/`, shared components in `src/components/`.
|
||||
|
||||
### Project layout
|
||||
|
||||
```
|
||||
pic/
|
||||
├── api/ # Flask API + all service managers
|
||||
│ ├── app.py # all routes (~2700 lines)
|
||||
│ ├── config_manager.py # unified config CRUD
|
||||
│ ├── ip_utils.py # IP/CIDR helpers + Caddyfile generator
|
||||
│ ├── firewall_manager.py # iptables (via cell-wireguard) + Corefile
|
||||
│ ├── network_manager.py # DNS zones, DHCP, NTP
|
||||
│ ├── wireguard_manager.py
|
||||
│ ├── peer_registry.py
|
||||
│ ├── vault_manager.py
|
||||
│ ├── email_manager.py
|
||||
│ ├── calendar_manager.py
|
||||
│ ├── file_manager.py
|
||||
│ └── container_manager.py
|
||||
├── webui/ # React frontend
|
||||
├── config/ # Config files (bind-mounted into containers)
|
||||
│ ├── api/cell_config.json ← live config
|
||||
│ ├── caddy/Caddyfile
|
||||
│ ├── dns/Corefile
|
||||
│ └── ...
|
||||
├── data/ # Persistent data (git-ignored)
|
||||
├── tests/ # pytest suite (372 tests, 27 files)
|
||||
├── docker-compose.yml
|
||||
└── Makefile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Config
|
||||
|
||||
```
|
||||
GET /api/config full config + service IPs
|
||||
PUT /api/config update identity or service config
|
||||
GET /api/config/pending pending restart info
|
||||
POST /api/config/apply apply pending restart
|
||||
POST /api/config/backup create backup
|
||||
POST /api/config/restore/<backup_id> restore from backup
|
||||
```
|
||||
|
||||
### Network
|
||||
|
||||
```
|
||||
GET /api/dns/records
|
||||
POST /api/dns/records
|
||||
GET /api/dhcp/leases
|
||||
GET /api/dhcp/reservations
|
||||
POST /api/dhcp/reservations
|
||||
```
|
||||
|
||||
### WireGuard & Peers
|
||||
|
||||
```
|
||||
GET /api/wireguard/status
|
||||
GET /api/wireguard/peers
|
||||
POST /api/wireguard/peers
|
||||
GET /api/peers
|
||||
POST /api/peers
|
||||
PUT /api/peers/<name>
|
||||
DELETE /api/peers/<name>
|
||||
GET /api/peers/<name>/config peer config + QR code
|
||||
```
|
||||
|
||||
### Containers & Health
|
||||
|
||||
```
|
||||
GET /api/containers
|
||||
POST /api/containers/<name>/restart
|
||||
GET /health
|
||||
GET /api/services/status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
make test # run full suite
|
||||
make test-coverage # coverage report in htmlcov/
|
||||
pytest tests/test_<module>.py # single file
|
||||
pytest tests/ -k "test_name" # single test
|
||||
```
|
||||
|
||||
Tests live in `tests/` and use `unittest.TestCase` collected by pytest. External system calls (Docker, iptables, file writes) are mocked with `unittest.mock.patch`.
|
||||
|
||||
Known coverage gaps: `write_caddyfile`, `POST /api/config/apply` (helper container path), `PUT /api/config` 400 validation paths. These are the highest-risk untested paths.
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
- The API is access-controlled by `is_local_request()` — it checks whether the request comes from a local/loopback/cell-network IP. Sensitive endpoints (containers, vault) are restricted to local access only.
|
||||
- All per-peer service access is enforced via iptables rules inside `cell-wireguard` and CoreDNS ACL blocks.
|
||||
- The Docker socket is mounted into `cell-api` for container management — treat network access to port 3000 as privileged.
|
||||
- `ip_range` must be an RFC-1918 CIDR (10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16). The API and UI both validate this.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
|
||||
Reference in New Issue
Block a user