Update all documentation to reflect current architecture
Unit Tests / test (push) Failing after 8m47s
Unit Tests / test (push) Failing after 8m47s
README, QUICKSTART, and Wiki were pre-wizard, pre-auth, pre-DDNS, and pre-service-store. Full rewrite covering: - First-run wizard replaces manual make setup + .env identity config - Session-based auth (admin/peer roles, CSRF protection) - DDNS: pic.ngo registration with TOTP, provider abstraction - Service store: install/remove optional services from manifest index - Cell-to-cell networking and peer-sync protocol - Extended connectivity: WG external, OpenVPN, Tor exit routing - Caddy HTTPS: Let's Encrypt (DNS-01/HTTP-01) or internal CA - Current container list, port bindings, and security model - Accurate make targets (ddns-update, reset-admin-password, etc.) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,535 +1,343 @@
|
||||
# Personal Internet Cell – Project Wiki
|
||||
|
||||
## 🌟 Overview
|
||||
|
||||
Personal Internet Cell is a **production-grade, self-hosted, decentralized digital infrastructure** solution designed to provide individuals with full control over their digital services and data. The project has evolved from a phase-based implementation to a **unified, enterprise-ready system** with modern architecture, comprehensive testing, and production-grade features.
|
||||
|
||||
## 📋 Table of Contents
|
||||
|
||||
1. [Project Goals](#project-goals)
|
||||
2. [Architecture & Components](#architecture--components)
|
||||
3. [Service Manager Architecture](#service-manager-architecture)
|
||||
4. [Core Services](#core-services)
|
||||
5. [API Reference](#api-reference)
|
||||
6. [Enhanced CLI](#enhanced-cli)
|
||||
7. [Security Model](#security-model)
|
||||
8. [Testing & Quality Assurance](#testing--quality-assurance)
|
||||
9. [Usage Examples](#usage-examples)
|
||||
10. [Development & Deployment](#development--deployment)
|
||||
11. [Future Enhancements](#future-enhancements)
|
||||
12. [Project Status](#project-status)
|
||||
|
||||
## 🎯 Project Goals
|
||||
|
||||
- **Self-Hosted**: Run your own digital services (email, calendar, files, VPN, etc.) on your hardware
|
||||
- **Decentralized**: Peer-to-peer networking and trust, no central authority
|
||||
- **Production-Grade**: Enterprise-ready architecture with comprehensive monitoring
|
||||
- **Secure**: Modern cryptography, certificate management, and encrypted storage
|
||||
- **User-Friendly**: Professional CLI and API for easy management
|
||||
- **Extensible**: Modular architecture for future services and integrations
|
||||
- **Event-Driven**: Real-time service communication and orchestration
|
||||
|
||||
## 🏗️ Architecture & Components
|
||||
|
||||
### **Modern Architecture Stack**
|
||||
|
||||
- **Backend**: Python (Flask) with production-grade service managers
|
||||
- **Service Architecture**: BaseServiceManager pattern with unified interfaces
|
||||
- **Event System**: Service bus for real-time communication and orchestration
|
||||
- **Configuration**: Centralized configuration management with validation
|
||||
- **Logging**: Structured JSON logging with rotation and search
|
||||
- **Containerization**: Docker-based deployment and service isolation
|
||||
- **API**: RESTful endpoints with comprehensive documentation
|
||||
|
||||
### **Core Architecture Components**
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Personal Internet Cell │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Enhanced CLI │ Web UI │ REST API │ Service Bus │ Logging │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Service Managers │
|
||||
│ Network │ WireGuard │ Email │ Calendar │ Files │ Routing │
|
||||
│ Vault │ Container │ Cell │ Peer │ │ │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Core Infrastructure │
|
||||
│ DNS │ DHCP │ NTP │ VPN │ CA │ Encryption │ Trust │ Storage │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 🔧 Service Manager Architecture
|
||||
|
||||
### **BaseServiceManager Pattern**
|
||||
|
||||
All services inherit from `BaseServiceManager`, providing:
|
||||
|
||||
```python
|
||||
class BaseServiceManager(ABC):
|
||||
def __init__(self, service_name: str, data_dir: str, config_dir: str)
|
||||
|
||||
@abstractmethod
|
||||
def get_status(self) -> Dict[str, Any]
|
||||
|
||||
@abstractmethod
|
||||
def test_connectivity(self) -> Dict[str, Any]
|
||||
|
||||
# Common methods
|
||||
def get_logs(self, lines: int = 50) -> List[str]
|
||||
def restart_service(self) -> bool
|
||||
def get_config(self) -> Dict[str, Any]
|
||||
def update_config(self, config: Dict[str, Any]) -> bool
|
||||
def health_check(self) -> Dict[str, Any]
|
||||
def handle_error(self, error: Exception, context: str) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
### **Service Bus Integration**
|
||||
|
||||
```python
|
||||
# Event-driven service communication
|
||||
service_bus.register_service('network', network_manager)
|
||||
service_bus.register_service('wireguard', wireguard_manager)
|
||||
service_bus.publish_event(EventType.SERVICE_STARTED, 'network', data)
|
||||
|
||||
# Service dependencies
|
||||
service_dependencies = {
|
||||
'wireguard': ['network'],
|
||||
'email': ['network', 'vault'],
|
||||
'calendar': ['network', 'vault'],
|
||||
'files': ['network', 'vault'],
|
||||
'routing': ['network', 'wireguard'],
|
||||
'vault': ['network']
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Core Services
|
||||
|
||||
### **Network Services**
|
||||
- **NetworkManager**: DNS, DHCP, NTP with dynamic management
|
||||
- Dynamic zone file generation
|
||||
- DHCP lease monitoring
|
||||
- Network connectivity testing
|
||||
- Service health monitoring
|
||||
|
||||
### **VPN & Mesh Networking**
|
||||
- **WireGuardManager**: WireGuard VPN configuration and peer management
|
||||
- Key generation and management
|
||||
- Peer configuration
|
||||
- Connectivity testing
|
||||
- Dynamic IP updates
|
||||
|
||||
- **PeerRegistry**: Peer registration and trust management
|
||||
- Peer lifecycle management
|
||||
- Trust relationship tracking
|
||||
- Data integrity validation
|
||||
- Peer statistics
|
||||
|
||||
### **Digital Services**
|
||||
- **EmailManager**: SMTP/IMAP email services
|
||||
- User account management
|
||||
- Mailbox configuration
|
||||
- Service connectivity testing
|
||||
- Email delivery monitoring
|
||||
|
||||
- **CalendarManager**: CalDAV/CardDAV calendar and contacts
|
||||
- User and calendar management
|
||||
- Event synchronization
|
||||
- Service health monitoring
|
||||
- Connectivity testing
|
||||
|
||||
- **FileManager**: WebDAV file storage
|
||||
- User directory management
|
||||
- Storage quota monitoring
|
||||
- File system access testing
|
||||
- Backup and restore capabilities
|
||||
|
||||
### **Infrastructure Services**
|
||||
- **RoutingManager**: Advanced routing and NAT
|
||||
- NAT rule management
|
||||
- Firewall configuration
|
||||
- Exit node routing
|
||||
- Bridge and split routing
|
||||
- Connectivity testing
|
||||
|
||||
- **VaultManager**: Security and trust management
|
||||
- Self-hosted Certificate Authority
|
||||
- Certificate lifecycle management
|
||||
- Age/Fernet encryption
|
||||
- Trust relationship management
|
||||
- Cryptographic verification
|
||||
|
||||
- **ContainerManager**: Docker orchestration
|
||||
- Container lifecycle management
|
||||
- Image and volume management
|
||||
- Docker daemon connectivity
|
||||
- Service isolation
|
||||
|
||||
- **CellManager**: Overall cell orchestration
|
||||
- Service coordination
|
||||
- Health monitoring
|
||||
- Configuration management
|
||||
- Peer management
|
||||
|
||||
## 📡 API Reference
|
||||
|
||||
### **Core API Endpoints**
|
||||
|
||||
```bash
|
||||
# Service Status and Health
|
||||
GET /api/services/status # All services status
|
||||
GET /api/services/connectivity # Service connectivity tests
|
||||
GET /health # API health check
|
||||
|
||||
# Configuration Management
|
||||
GET /api/config # Get configuration
|
||||
PUT /api/config # Update configuration
|
||||
POST /api/config/backup # Create backup
|
||||
GET /api/config/backups # List backups
|
||||
POST /api/config/restore/<id> # Restore backup
|
||||
GET /api/config/export # Export configuration
|
||||
POST /api/config/import # Import configuration
|
||||
|
||||
# Service Bus
|
||||
GET /api/services/bus/status # Service bus status
|
||||
GET /api/services/bus/events # Event history
|
||||
POST /api/services/bus/services/<service>/start
|
||||
POST /api/services/bus/services/<service>/stop
|
||||
POST /api/services/bus/services/<service>/restart
|
||||
|
||||
# Logging
|
||||
GET /api/logs/services/<service> # Service logs
|
||||
POST /api/logs/search # Log search
|
||||
POST /api/logs/export # Log export
|
||||
GET /api/logs/statistics # Log statistics
|
||||
POST /api/logs/rotate # Log rotation
|
||||
```
|
||||
|
||||
### **Service-Specific Endpoints**
|
||||
|
||||
```bash
|
||||
# Network Services
|
||||
GET /api/dns/records # DNS records
|
||||
POST /api/dns/records # Add DNS record
|
||||
DELETE /api/dns/records # Remove DNS record
|
||||
GET /api/dhcp/leases # DHCP leases
|
||||
POST /api/dhcp/reservations # Add DHCP reservation
|
||||
GET /api/ntp/status # NTP status
|
||||
GET /api/network/info # Network information
|
||||
POST /api/network/test # Network connectivity test
|
||||
|
||||
# WireGuard & Peers
|
||||
GET /api/wireguard/keys # WireGuard keys
|
||||
POST /api/wireguard/keys/peer # Generate peer keys
|
||||
GET /api/wireguard/config # WireGuard configuration
|
||||
GET /api/wireguard/peers # List peers
|
||||
POST /api/wireguard/peers # Add peer
|
||||
DELETE /api/wireguard/peers # Remove peer
|
||||
GET /api/wireguard/status # WireGuard status
|
||||
POST /api/wireguard/connectivity # Connectivity test
|
||||
PUT /api/wireguard/peers/ip # Update peer IP
|
||||
|
||||
# Digital Services
|
||||
GET /api/email/users # Email users
|
||||
POST /api/email/users # Add email user
|
||||
DELETE /api/email/users/<user> # Remove email user
|
||||
GET /api/email/status # Email service status
|
||||
GET /api/email/connectivity # Email connectivity
|
||||
POST /api/email/send # Send email
|
||||
GET /api/email/mailbox/<user> # User mailbox
|
||||
|
||||
GET /api/calendar/users # Calendar users
|
||||
POST /api/calendar/users # Add calendar user
|
||||
DELETE /api/calendar/users/<user> # Remove calendar user
|
||||
POST /api/calendar/calendars # Create calendar
|
||||
POST /api/calendar/events # Add event
|
||||
GET /api/calendar/events/<user>/<calendar> # List events
|
||||
GET /api/calendar/status # Calendar service status
|
||||
GET /api/calendar/connectivity # Calendar connectivity
|
||||
|
||||
GET /api/files/users # File users
|
||||
POST /api/files/users # Add file user
|
||||
DELETE /api/files/users/<user> # Remove file user
|
||||
POST /api/files/folders # Create folder
|
||||
DELETE /api/files/folders/<user>/<path> # Remove folder
|
||||
POST /api/files/upload/<user> # Upload file
|
||||
GET /api/files/download/<user>/<path> # Download file
|
||||
DELETE /api/files/delete/<user>/<path> # Delete file
|
||||
GET /api/files/list/<user> # List files
|
||||
GET /api/files/status # File service status
|
||||
GET /api/files/connectivity # File connectivity
|
||||
|
||||
# Routing & Security
|
||||
GET /api/routing/status # Routing status
|
||||
POST /api/routing/nat # Add NAT rule
|
||||
DELETE /api/routing/nat/<id> # Remove NAT rule
|
||||
POST /api/routing/peers # Add peer route
|
||||
DELETE /api/routing/peers/<peer> # Remove peer route
|
||||
POST /api/routing/exit-nodes # Add exit node
|
||||
POST /api/routing/bridge # Add bridge route
|
||||
POST /api/routing/split # Add split route
|
||||
POST /api/routing/firewall # Add firewall rule
|
||||
POST /api/routing/connectivity # Routing connectivity test
|
||||
GET /api/routing/logs # Routing logs
|
||||
GET /api/routing/nat # List NAT rules
|
||||
GET /api/routing/peers # List peer routes
|
||||
GET /api/routing/firewall # List firewall rules
|
||||
|
||||
GET /api/vault/status # Vault status
|
||||
GET /api/vault/certificates # List certificates
|
||||
POST /api/vault/certificates # Generate certificate
|
||||
DELETE /api/vault/certificates/<name> # Revoke certificate
|
||||
GET /api/vault/ca/certificate # CA certificate
|
||||
GET /api/vault/age/public-key # Age public key
|
||||
GET /api/vault/trust/keys # Trusted keys
|
||||
POST /api/vault/trust/keys # Add trusted key
|
||||
DELETE /api/vault/trust/keys/<name> # Remove trusted key
|
||||
POST /api/vault/trust/verify # Verify trust
|
||||
GET /api/vault/trust/chains # Trust chains
|
||||
```
|
||||
|
||||
## 💻 Enhanced CLI
|
||||
|
||||
### **CLI Features**
|
||||
|
||||
```bash
|
||||
# Interactive mode with tab completion
|
||||
python api/enhanced_cli.py --interactive
|
||||
|
||||
# Batch operations
|
||||
python api/enhanced_cli.py --batch "status" "services" "health"
|
||||
|
||||
# Configuration management
|
||||
python api/enhanced_cli.py --export-config json
|
||||
python api/enhanced_cli.py --import-config config.json
|
||||
|
||||
# Service wizards
|
||||
python api/enhanced_cli.py --wizard network
|
||||
python api/enhanced_cli.py --wizard email
|
||||
|
||||
# Health monitoring
|
||||
python api/enhanced_cli.py --health
|
||||
python api/enhanced_cli.py --logs network
|
||||
|
||||
# Service status
|
||||
python api/enhanced_cli.py --status
|
||||
python api/enhanced_cli.py --services
|
||||
python api/enhanced_cli.py --peers
|
||||
```
|
||||
|
||||
### **CLI Capabilities**
|
||||
- **Interactive Mode**: Tab completion, command history, help system
|
||||
- **Batch Operations**: Execute multiple commands in sequence
|
||||
- **Configuration Wizards**: Guided setup for complex services
|
||||
- **Real-time Monitoring**: Live status updates and health checks
|
||||
- **Log Management**: View, search, and export service logs
|
||||
- **Service Management**: Start, stop, restart, and configure services
|
||||
|
||||
## 🔒 Security Model
|
||||
|
||||
### **Certificate Management**
|
||||
- **Self-hosted CA**: Issue and manage TLS certificates for all services
|
||||
- **Certificate Lifecycle**: Generate, renew, revoke, and monitor certificates
|
||||
- **Trust Management**: Direct, indirect, and verified trust relationships
|
||||
- **Age Encryption**: Modern encryption for sensitive data and keys
|
||||
|
||||
### **Network Security**
|
||||
- **WireGuard VPN**: Secure peer-to-peer communication with key rotation
|
||||
- **Firewall & NAT**: Granular control over network access and routing
|
||||
- **Service Isolation**: Docker containers for each service
|
||||
- **Input Validation**: All API endpoints validate and sanitize input
|
||||
|
||||
### **Data Protection**
|
||||
- **Encrypted Storage**: Sensitive data encrypted at rest using Age/Fernet
|
||||
- **Secure Communication**: TLS for all API endpoints and service communication
|
||||
- **Access Control**: Role-based access for services and API endpoints
|
||||
- **Audit Logging**: Comprehensive security event logging and monitoring
|
||||
|
||||
## 🧪 Testing & Quality Assurance
|
||||
|
||||
### **Test Coverage**
|
||||
- **BaseServiceManager**: 100% coverage
|
||||
- **ConfigManager**: 95%+ coverage
|
||||
- **ServiceBus**: 95%+ coverage
|
||||
- **LogManager**: 95%+ coverage
|
||||
- **All Service Managers**: 77%+ overall coverage
|
||||
- **API Endpoints**: 100% endpoint coverage
|
||||
|
||||
### **Test Types**
|
||||
- **Unit Tests**: Individual component testing
|
||||
- **Integration Tests**: Service interaction testing
|
||||
- **API Tests**: Endpoint functionality testing
|
||||
- **Error Handling**: Exception and edge case testing
|
||||
- **Performance Tests**: Load and stress testing
|
||||
|
||||
### **Testing Commands**
|
||||
```bash
|
||||
# Run all tests
|
||||
python api/test_enhanced_api.py
|
||||
|
||||
# Run specific test suites
|
||||
python -m pytest api/tests/test_network_manager.py
|
||||
python -m pytest api/tests/test_service_bus.py
|
||||
|
||||
# Generate coverage report
|
||||
coverage run -m pytest api/tests/
|
||||
coverage html
|
||||
```
|
||||
|
||||
## 📝 Usage Examples
|
||||
|
||||
### **Add DNS Record**
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/dns/records \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "www",
|
||||
"type": "A",
|
||||
"value": "192.168.1.100",
|
||||
"ttl": 300
|
||||
}'
|
||||
```
|
||||
|
||||
### **Register Peer**
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/wireguard/peers \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "bob",
|
||||
"ip": "203.0.113.22",
|
||||
"public_key": "peer_public_key_here",
|
||||
"allowed_networks": ["10.0.0.0/24"]
|
||||
}'
|
||||
```
|
||||
|
||||
### **Generate Certificate**
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/vault/certificates \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"common_name": "myapp.example.com",
|
||||
"domains": ["myapp.example.com", "www.myapp.example.com"],
|
||||
"days": 365
|
||||
}'
|
||||
```
|
||||
|
||||
### **Configure NAT Rule**
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/routing/nat \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source_network": "10.0.0.0/24",
|
||||
"target_interface": "eth0",
|
||||
"nat_type": "MASQUERADE",
|
||||
"protocol": "ALL"
|
||||
}'
|
||||
```
|
||||
|
||||
## 🛠️ Development & Deployment
|
||||
|
||||
### **Development Setup**
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install -r api/requirements.txt
|
||||
|
||||
# Start development server
|
||||
python api/app.py
|
||||
|
||||
# Run tests
|
||||
python api/test_enhanced_api.py
|
||||
|
||||
# Start frontend (if available)
|
||||
cd webui && bun install && npm run dev
|
||||
```
|
||||
|
||||
### **Production Deployment**
|
||||
```bash
|
||||
# Docker deployment
|
||||
docker-compose up --build -d
|
||||
|
||||
# Health check
|
||||
curl http://localhost:3000/health
|
||||
|
||||
# Service status
|
||||
curl http://localhost:3000/api/services/status
|
||||
```
|
||||
|
||||
### **Service Development**
|
||||
```python
|
||||
from base_service_manager import BaseServiceManager
|
||||
|
||||
class MyServiceManager(BaseServiceManager):
|
||||
def __init__(self, data_dir='/app/data', config_dir='/app/config'):
|
||||
super().__init__('myservice', data_dir, config_dir)
|
||||
|
||||
def get_status(self) -> Dict[str, Any]:
|
||||
# Implement service status
|
||||
return {
|
||||
'running': True,
|
||||
'status': 'online',
|
||||
'timestamp': datetime.utcnow().isoformat()
|
||||
}
|
||||
|
||||
def test_connectivity(self) -> Dict[str, Any]:
|
||||
# Implement connectivity test
|
||||
return {
|
||||
'success': True,
|
||||
'message': 'Service connectivity working',
|
||||
'timestamp': datetime.utcnow().isoformat()
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Future Enhancements
|
||||
|
||||
### **Planned Features**
|
||||
- **Certificate Auto-renewal**: Automatic certificate renewal and monitoring
|
||||
- **Web of Trust Models**: Advanced trust relationship management
|
||||
- **Certificate Transparency**: CT log integration and monitoring
|
||||
- **Hardware Security Module (HSM)**: HSM integration for key management
|
||||
- **WebSocket Updates**: Real-time service status updates
|
||||
- **Advanced Monitoring**: Metrics collection and alerting systems
|
||||
- **Mobile App**: Mobile application for remote management
|
||||
- **Plugin System**: Extensible architecture for custom services
|
||||
|
||||
### **Architecture Improvements**
|
||||
- **Service Discovery**: Dynamic service registration and discovery
|
||||
- **Load Balancing**: Multi-instance service deployment
|
||||
- **Advanced Caching**: Redis-based caching for performance
|
||||
- **Message Queues**: RabbitMQ/Kafka for reliable messaging
|
||||
- **Distributed Tracing**: OpenTelemetry integration
|
||||
- **Configuration Management**: GitOps-style configuration management
|
||||
|
||||
## 📊 Project Status
|
||||
|
||||
### **✅ Completed Features**
|
||||
- **Production-Grade Architecture**: BaseServiceManager pattern implemented
|
||||
- **Event-Driven Communication**: Service bus with real-time events
|
||||
- **Centralized Configuration**: Type-safe configuration with validation
|
||||
- **Comprehensive Logging**: Structured logging with search and export
|
||||
- **Enhanced CLI**: Interactive CLI with batch operations
|
||||
- **Health Monitoring**: Real-time health checks across all services
|
||||
- **Security Framework**: Self-hosted CA, encryption, and trust management
|
||||
- **Complete API**: RESTful API with comprehensive documentation
|
||||
- **Testing Framework**: Comprehensive test suite with high coverage
|
||||
|
||||
### **🎯 Current Status**
|
||||
- **All Services**: 10 service managers fully implemented and integrated
|
||||
- **API Server**: Running on port 3000 with all endpoints functional
|
||||
- **CLI Tool**: Enhanced CLI with all features working
|
||||
- **Test Coverage**: 77%+ overall coverage with comprehensive testing
|
||||
- **Documentation**: Complete documentation for all components
|
||||
- **Production Ready**: Suitable for personal and small business deployment
|
||||
|
||||
### **🌟 Key Achievements**
|
||||
- **Unified Architecture**: All services follow the same patterns and interfaces
|
||||
- **Event-Driven Design**: Services communicate and orchestrate automatically
|
||||
- **Configuration Management**: Centralized, validated configuration system
|
||||
- **Comprehensive Logging**: Production-grade logging with advanced features
|
||||
- **Enhanced CLI**: Professional command-line interface for management
|
||||
- **Health Monitoring**: Real-time monitoring and alerting capabilities
|
||||
- **Security Framework**: Enterprise-grade security with modern cryptography
|
||||
- **Complete Testing**: Comprehensive test suite ensuring reliability
|
||||
|
||||
---
|
||||
|
||||
**The Personal Internet Cell empowers users with full control over their digital infrastructure, combining privacy, security, and usability in a single, production-ready, self-hosted platform.** 🌟
|
||||
|
||||
# Personal Internet Cell – Project Wiki
|
||||
|
||||
## Overview
|
||||
|
||||
Personal Internet Cell (PIC) is a self-hosted digital infrastructure platform. It runs DNS, DHCP, NTP, WireGuard VPN, email, calendar/contacts, file storage, HTTPS reverse proxy, a certificate authority, and optional services — all managed from a single REST API and React web UI.
|
||||
|
||||
The goal is to give a person full ownership of their core internet services on their own hardware, without relying on cloud providers.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture](#architecture)
|
||||
2. [Service Managers](#service-managers)
|
||||
3. [First-Run Wizard](#first-run-wizard)
|
||||
4. [Authentication](#authentication)
|
||||
5. [API Reference](#api-reference)
|
||||
6. [DDNS](#ddns)
|
||||
7. [Service Store](#service-store)
|
||||
8. [Cell-to-Cell Networking](#cell-to-cell-networking)
|
||||
9. [Extended Connectivity](#extended-connectivity)
|
||||
10. [Security Model](#security-model)
|
||||
11. [Testing](#testing)
|
||||
12. [Development](#development)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser / WireGuard peer
|
||||
└── Caddy (:80/:443) reverse proxy, TLS termination
|
||||
└── React SPA (:8081) Vite + Tailwind (Nginx in container)
|
||||
└── Flask API (:3000) REST API, bound to 127.0.0.1
|
||||
├── NetworkManager CoreDNS, dnsmasq, chrony
|
||||
├── WireGuardManager WireGuard VPN peer lifecycle
|
||||
├── PeerRegistry peer registration and trust
|
||||
├── EmailManager Postfix + Dovecot
|
||||
├── CalendarManager Radicale CalDAV/CardDAV
|
||||
├── FileManager WebDAV + Filegator
|
||||
├── RoutingManager iptables NAT and routing
|
||||
├── FirewallManager iptables firewall rules
|
||||
├── VaultManager internal CA, cert lifecycle, Age encryption
|
||||
├── ContainerManager Docker SDK
|
||||
├── CellLinkManager cell-to-cell WireGuard links
|
||||
├── ConnectivityManager exit routing (WG ext, OpenVPN, Tor)
|
||||
├── DDNSManager dynamic DNS heartbeat
|
||||
├── ServiceStoreManager optional service install/remove
|
||||
├── CaddyManager Caddyfile generation and reload
|
||||
├── AuthManager session auth, RBAC
|
||||
└── SetupManager first-run wizard state
|
||||
```
|
||||
|
||||
All 12 service containers run on a Docker bridge network (`cell-network`, `172.20.0.0/16` default). Static IPs per container are defined in `docker-compose.yml`.
|
||||
|
||||
Runtime configuration lives in `config/api/cell_config.json`, managed by `ConfigManager`. All service managers read and write through `ConfigManager`, which validates and backs up automatically.
|
||||
|
||||
---
|
||||
|
||||
## Service Managers
|
||||
|
||||
All managers inherit `BaseServiceManager` (`api/base_service_manager.py`), which provides:
|
||||
- `get_status()` — current running state
|
||||
- `get_config()` / `update_config()` — config read/write
|
||||
- `test_connectivity()` — reachability check
|
||||
- `get_logs()` — last N lines from the service log
|
||||
- `restart_service()` — container restart via Docker SDK
|
||||
|
||||
The `ServiceBus` (`api/service_bus.py`) handles pub/sub events between managers (e.g., `CONFIG_CHANGED`, `SERVICE_STARTED`). Dependencies are declared in the bus (wireguard depends on network; email depends on network and vault).
|
||||
|
||||
### Manager summary
|
||||
|
||||
| Manager | Responsibilities |
|
||||
|---|---|
|
||||
| `NetworkManager` | CoreDNS zone files, dnsmasq DHCP config and lease monitoring, chrony NTP |
|
||||
| `WireGuardManager` | Key generation, `wg0.conf` generation, peer add/remove, route sync |
|
||||
| `PeerRegistry` | Peer registration, trust tracking, peer statistics |
|
||||
| `EmailManager` | docker-mailserver accounts, mailbox config, alias management |
|
||||
| `CalendarManager` | Radicale user/calendar/contacts lifecycle |
|
||||
| `FileManager` | WebDAV user directories, Filegator access |
|
||||
| `RoutingManager` | NAT rules, per-peer routing policy, fwmark-based exit routing |
|
||||
| `FirewallManager` | iptables INPUT/FORWARD/OUTPUT rule management |
|
||||
| `VaultManager` | Internal CA (self-signed root), TLS cert issue/revoke, Age public key |
|
||||
| `ContainerManager` | Docker container/image/volume management via SDK |
|
||||
| `CellLinkManager` | Site-to-site WireGuard links to other PIC cells, peer-sync protocol |
|
||||
| `ConnectivityManager` | Per-peer exit routing via WireGuard external, OpenVPN, or Tor |
|
||||
| `DDNSManager` | Public IP heartbeat, provider abstraction (pic_ngo, cloudflare, duckdns, noip, freedns) |
|
||||
| `ServiceStoreManager` | Fetch manifest index, install/remove optional services |
|
||||
| `CaddyManager` | Caddyfile generation, reload-on-change |
|
||||
| `AuthManager` | bcrypt password store, session management, admin/peer RBAC |
|
||||
| `SetupManager` | First-run wizard state, setup-complete flag |
|
||||
|
||||
---
|
||||
|
||||
## First-Run Wizard
|
||||
|
||||
On first start, `SetupManager.is_setup_complete()` returns `False`. The `enforce_setup` before-request hook returns HTTP 428 for all `/api/*` requests except `/api/setup/*` and `/health`, redirecting clients to `/setup`.
|
||||
|
||||
The wizard collects:
|
||||
- **Cell name** — used for hostnames and DDNS subdomain (e.g. `myhome` → `myhome.pic.ngo`)
|
||||
- **Domain mode** — determines TLS certificate source: `lan` (internal CA), `pic_ngo`, `cloudflare`, `duckdns`, `http01`
|
||||
- **Timezone**
|
||||
- **Initial services to enable**
|
||||
- **Admin password** — minimum 12 characters
|
||||
|
||||
On completion:
|
||||
1. Admin account is created in `data/auth_users.json`
|
||||
2. Cell identity is written to `config/api/cell_config.json`
|
||||
3. Caddy config is generated
|
||||
4. If domain mode is `pic_ngo`, the cell registers `<name>.pic.ngo` with the DDNS service
|
||||
|
||||
Wizard endpoints: `GET/POST /api/setup/step`, `GET /api/setup/status`, `POST /api/setup/complete`.
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
`AuthManager` stores bcrypt-hashed credentials in `data/auth_users.json`. Two roles:
|
||||
|
||||
| Role | Access |
|
||||
|---|---|
|
||||
| `admin` | All `/api/*` endpoints except `/api/peer/*` |
|
||||
| `peer` | `/api/peer/*` only (peer dashboard, key exchange) |
|
||||
|
||||
Session auth flow:
|
||||
- `POST /api/auth/login` — creates a Flask session
|
||||
- `GET /api/auth/me` — current session info
|
||||
- `POST /api/auth/logout` — clears session
|
||||
- `POST /api/auth/change-password` — change own password
|
||||
- `POST /api/auth/admin/reset-password` — admin resets another user's password
|
||||
|
||||
CSRF protection: all `POST`, `PUT`, `DELETE`, `PATCH` on `/api/*` (except `/api/auth/*` and `/api/setup/*`) require the `X-CSRF-Token` header matching the session token, obtained via `GET /api/auth/csrf-token`.
|
||||
|
||||
Cell-to-cell peer-sync endpoints (`/api/cells/peer-sync/*`) use source-IP + WireGuard public key auth, not session cookies.
|
||||
|
||||
Auth enforcement is active once any user exists in the store. If the store is empty (fresh install before wizard), all requests bypass auth — `enforce_setup` already blocks them with 428.
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
**Base URL:** `http://localhost:3000`
|
||||
**Auth:** session cookie (`X-CSRF-Token` header required for mutations)
|
||||
|
||||
### Core
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| GET | `/health` | Health check (always public) |
|
||||
| GET | `/api/status` | All-service status summary |
|
||||
| GET | `/api/config` | Full cell config |
|
||||
| PUT | `/api/config` | Update cell config |
|
||||
| GET | `/api/health/history` | Recent health check history |
|
||||
|
||||
### Auth (`/api/auth/`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| POST | `/api/auth/login` | Create session |
|
||||
| POST | `/api/auth/logout` | Destroy session |
|
||||
| GET | `/api/auth/me` | Current user info |
|
||||
| GET | `/api/auth/csrf-token` | Get CSRF token |
|
||||
| POST | `/api/auth/change-password` | Change own password |
|
||||
| POST | `/api/auth/admin/reset-password` | Admin: reset another user's password |
|
||||
| GET | `/api/auth/users` | Admin: list users |
|
||||
|
||||
### Setup (`/api/setup/`)
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/setup/status` | Setup complete flag + current step |
|
||||
| GET | `/api/setup/step` | Current wizard step data |
|
||||
| POST | `/api/setup/step` | Submit current step |
|
||||
| POST | `/api/setup/complete` | Finalize setup |
|
||||
|
||||
### Network Services (`/api/dns/`, `/api/dhcp/`, `/api/ntp/`, `/api/network/`)
|
||||
|
||||
DNS records, DHCP leases and reservations, NTP status, network connectivity test.
|
||||
|
||||
### WireGuard (`/api/wireguard/`, `/api/peers/`)
|
||||
|
||||
Peer add/remove, key generation, QR code export, per-peer routing policy, WireGuard status.
|
||||
|
||||
### Email (`/api/email/`)
|
||||
|
||||
User account management, mailbox config, alias management, connectivity test.
|
||||
|
||||
### Calendar (`/api/calendar/`)
|
||||
|
||||
User, calendar, and contacts (CardDAV) management.
|
||||
|
||||
### Files (`/api/files/`)
|
||||
|
||||
WebDAV user management, file upload/download/delete, folder management.
|
||||
|
||||
### Routing (`/api/routing/`)
|
||||
|
||||
NAT rules, peer routes, exit node configuration.
|
||||
|
||||
### Vault (`/api/vault/`)
|
||||
|
||||
Certificate issue/revoke, CA certificate, trust key management, Age public key.
|
||||
|
||||
### Containers (`/api/containers/`)
|
||||
|
||||
List, start, stop, inspect containers; manage images and volumes.
|
||||
|
||||
### Cell Network (`/api/cells/`)
|
||||
|
||||
List connected cells, add/remove cell links, peer-sync.
|
||||
|
||||
### Connectivity (`/api/connectivity/`)
|
||||
|
||||
List exit nodes, configure WireGuard external / OpenVPN / Tor exits, assign per-peer exit policy.
|
||||
|
||||
### Service Store (`/api/store/`)
|
||||
|
||||
List available services, install, remove.
|
||||
|
||||
### Logs (`/api/logs/`)
|
||||
|
||||
Per-service log retrieval, log search, log statistics.
|
||||
|
||||
---
|
||||
|
||||
## DDNS
|
||||
|
||||
`DDNSManager` maintains a `<cell-name>.pic.ngo` DNS A record pointing at the cell's public IP. A background thread runs every 5 minutes and calls `provider.update(token, ip)` only when the IP changes.
|
||||
|
||||
Registration happens during the setup wizard (if domain mode is `pic_ngo`) via `provider.register(name, ip)`, which returns a bearer token stored in `data/api/.ddns_token`.
|
||||
|
||||
DDNS config lives in `cell_config.json` under the top-level `ddns` key:
|
||||
|
||||
```json
|
||||
{
|
||||
"ddns": {
|
||||
"provider": "pic_ngo",
|
||||
"api_base_url": "https://ddns.pic.ngo",
|
||||
"totp_secret": "<base32 secret>"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Registration requires a time-based OTP (`X-Register-OTP` header) derived from the shared `REGISTER_TOTP_SECRET` on the DDNS server. This prevents unauthorized subdomain registration.
|
||||
|
||||
Supported providers: `pic_ngo`, `cloudflare`, `duckdns`, `noip`, `freedns`.
|
||||
|
||||
---
|
||||
|
||||
## Service Store
|
||||
|
||||
`ServiceStoreManager` fetches a manifest index from `http://git.pic.ngo/roof/pic-services/raw/branch/main/index.json`. Each manifest declares:
|
||||
- Container image
|
||||
- Caddy routes (added to the Caddyfile)
|
||||
- iptables rules
|
||||
- Environment variables
|
||||
- Health check endpoint
|
||||
|
||||
`POST /api/store/install` pulls the image, writes the Caddy route, applies iptables rules, and starts the container. `POST /api/store/remove` reverses this.
|
||||
|
||||
---
|
||||
|
||||
## Cell-to-Cell Networking
|
||||
|
||||
`CellLinkManager` manages WireGuard site-to-site tunnels between PIC cells. Each link is a WireGuard peer configured with a dedicated `/32` address and allowed-IPs covering the remote cell's subnet.
|
||||
|
||||
The peer-sync protocol (`/api/cells/peer-sync/`) exchanges public keys and allowed networks between cells using source-IP + WireGuard public key authentication (no session required).
|
||||
|
||||
Access control is per-service (calendar, files, mail, WebDAV) and enforced at the iptables level.
|
||||
|
||||
---
|
||||
|
||||
## Extended Connectivity
|
||||
|
||||
`ConnectivityManager` provides per-peer exit routing: traffic from a specific WireGuard peer can be routed through an alternate exit instead of the cell's default gateway.
|
||||
|
||||
Supported exits:
|
||||
- **WireGuard external** — another WireGuard endpoint (e.g. a VPS)
|
||||
- **OpenVPN** — OpenVPN client running in a container
|
||||
- **Tor** — Tor SOCKS proxy with transparent redirection
|
||||
|
||||
Routing uses fwmark and `ip rule` / `ip route` in separate routing tables. Configuration is via `PUT /api/connectivity/peers/<peer_name>/exit`.
|
||||
|
||||
---
|
||||
|
||||
## Security Model
|
||||
|
||||
- **No open ports for the API** — Flask API binds to `127.0.0.1:3000` only; Caddy proxies HTTPS requests to it.
|
||||
- **Session auth** — bcrypt passwords, Flask server-side sessions, CSRF double-submit.
|
||||
- **Setup wizard gate** — all `/api/*` requests return 428 until setup is complete.
|
||||
- **Role separation** — admin cannot access peer endpoints; peer cannot access admin endpoints.
|
||||
- **HTTPS everywhere** — Caddy handles TLS; internal services are reached via reverse proxy paths.
|
||||
- **Internal CA** — VaultManager issues certificates for services that don't use Let's Encrypt.
|
||||
- **Docker socket isolation** — the Docker socket is mounted only into `cell-api`; other containers have no Docker access.
|
||||
- **iptables firewall** — FirewallManager manages INPUT/FORWARD rules; WireGuard peer isolation is enforced at the packet level.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
make test # unit tests (pytest, ~1500 functions)
|
||||
make test-coverage # coverage report in htmlcov/
|
||||
```
|
||||
|
||||
Test layout:
|
||||
- `tests/` — unit and endpoint tests; no running services required
|
||||
- `tests/integration/` — require a running PIC stack
|
||||
- `tests/e2e/` — Playwright UI tests and WireGuard integration tests
|
||||
|
||||
CI: Gitea Actions runs `pytest tests/ --ignore=tests/e2e --ignore=tests/integration` on every push.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Full stack in Docker
|
||||
make start
|
||||
make stop
|
||||
make logs
|
||||
|
||||
# Flask API without Docker (port 3000)
|
||||
pip install -r api/requirements.txt
|
||||
python api/app.py
|
||||
|
||||
# React UI dev server (port 5173, proxies /api → :3000)
|
||||
cd webui && npm install && npm run dev
|
||||
|
||||
# Rebuild containers after code change
|
||||
make build-api
|
||||
make build-webui
|
||||
```
|
||||
|
||||
Key files:
|
||||
- `api/app.py` — Flask app, blueprint registration, before-request hooks, health monitor thread
|
||||
- `api/managers.py` — singleton instantiation of all service managers
|
||||
- `api/base_service_manager.py` — abstract base class all managers implement
|
||||
- `api/config_manager.py` — `cell_config.json` read/write/validate/backup
|
||||
- `api/service_bus.py` — pub/sub event system
|
||||
- `webui/src/services/api.js` — Axios API client used by all UI pages
|
||||
- `docker-compose.yml` — container definitions and network topology
|
||||
- `Makefile` — all operational commands
|
||||
|
||||
Reference in New Issue
Block a user