pic/README.md

# Personal Internet Cell

## 🌟 Overview

The Personal Internet Cell is a **production-grade, self-hosted, decentralized digital infrastructure** that empowers you to:

- **Host your own services**: Email, calendar, contacts, files, DNS, DHCP, NTP
- **Secure mesh networking**: Connect with trusted peers via WireGuard VPN
- **Advanced routing**: VPN gateway, NAT, firewall, exit nodes, and bridge routing
- **Enterprise security**: Self-hosted CA, certificate management, trust systems
- **Modern management**: RESTful API, enhanced CLI, and comprehensive monitoring
- **Event-driven architecture**: Service orchestration and real-time communication

---

## 🚀 Key Features

### 🔧 **Core Services**
- **Network Services**: DNS, DHCP, NTP with dynamic management
- **VPN & Mesh**: WireGuard-based peer federation with dynamic IP updates
- **Digital Services**: Email (SMTP/IMAP), Calendar/Contacts (CalDAV/CardDAV), File Storage (WebDAV)
- **Security**: Self-hosted Certificate Authority, Age/Fernet encryption, trust management
- **Container Orchestration**: Docker-based service management and deployment

### 🏗️ **Architecture Highlights**
- **BaseServiceManager**: Unified interface across all 10 service managers
- **Event-Driven Service Bus**: Real-time service communication and orchestration
- **Centralized Configuration**: Type-safe validation, backup/restore, import/export
- **Comprehensive Logging**: Structured JSON logs with rotation, search, and export
- **Enhanced CLI**: Interactive mode, batch operations, service wizards
- **Health Monitoring**: Real-time health checks and performance metrics

### 📊 **Production Features**
- **Service Orchestration**: Automatic service dependency management
- **Configuration Management**: Schema validation, versioning, and migration
- **Error Handling**: Standardized error handling and recovery mechanisms
- **Testing**: Comprehensive test suite with 77%+ coverage
- **Documentation**: Complete API documentation and usage guides

---

## 📋 Table of Contents

1. [Quick Start](#quick-start)
2. [Architecture](#architecture)
3. [Service Managers](#service-managers)
4. [API Reference](#api-reference)
5. [CLI Guide](#cli-guide)
6. [Configuration](#configuration)
7. [Security](#security)
8. [Development](#development)
9. [Testing](#testing)
10. [Deployment](#deployment)
11. [Contributing](#contributing)
12. [License](#license)

---

## 🚀 Quick Start

### Prerequisites

- **Debian/Ubuntu** host (apt-based). All other dependencies are installed automatically.
- **2 GB+ RAM, 10 GB+ disk space**
- **Open ports**: 53 (DNS), 80/443 (HTTP/S), 3000 (API), 8081 (Web UI), 51820/udp (WireGuard)

### 1. Install

```bash
git clone <repo-url> pic
cd pic

# Install all system dependencies (docker, python3, python3-cryptography, etc.)
make check-deps

# Default cell (name=mycell, domain=cell, VPN=10.0.0.1/24, port=51820)
make setup
make start

# Custom cell — use when installing a second cell on a different host
CELL_NAME=pic1 VPN_ADDRESS=10.1.0.1/24 make setup && make start
```

`make check-deps` installs python3, python3-cryptography, docker, docker-compose, curl, openssl, git via apt and adds the current user to the docker group.

`make setup` generates WireGuard keys, writes configs, and creates all data directories.

`make start` builds and brings up all 12 Docker containers.

### 2. Access

| Service | URL |
|---------|-----|
| Web UI | `http://<host-ip>:8081` |
| API | `http://<host-ip>:3000` |
| Health | `http://<host-ip>:3000/health` |

On a WireGuard client: `http://mycell.cell` (or whatever your cell name is).

### 3. Local dev (no Docker)

```bash
pip install -r api/requirements.txt
python api/app.py                          # API on :3000

cd webui && npm install && npm run dev     # React UI on :5173 (proxies API to :3000)
```

---

## 🛠️ Management Commands

```bash
# First install
make check-deps          # install all system packages via apt
make setup               # generate keys, write configs
make start               # start all 12 containers

# Daily operations
make status              # container status + API health
make logs                # follow all logs
make logs-api            # follow logs for one service (api, dns, wg, mail, caddy, ...)
make shell-api           # open a shell inside a container

# Deploy latest code
make update              # git pull + rebuild + restart

# Full wipe and reinstall (useful on test machine)
make reinstall           # stop, wipe config/data, setup, start fresh

# Remove everything
make uninstall           # stop containers, remove images and all data

# Maintenance
make backup              # tar config/ + data/ into backups/
make restore             # list available backups
make clean               # remove containers/volumes, keep config/data

# Tests
make test                # run all tests
make test-coverage       # tests + HTML coverage report
```

---

## 🔗 Connecting Two Cells (PIC Mesh)

Two PIC instances can form a mesh — full site-to-site WireGuard tunnels with
automatic DNS forwarding so each cell's services are reachable from the other.

### Install the second cell

```bash
# On the second host (different VPN subnet; port 51820 is fine — different machine)
CELL_NAME=pic1 VPN_ADDRESS=10.1.0.1/24 make setup && make start
```

### Exchange invites (two pastes, two clicks)

1. On **Cell A** → open Web UI → **Cell Network** → copy the invite JSON.
2. On **Cell B** → **Cell Network** → paste into "Connect to Another Cell" → click **Connect**.
3. On **Cell B** → copy its invite JSON.
4. On **Cell A** → paste Cell B's invite → click **Connect**.

Both cells now have:
- A site-to-site WireGuard peer (AllowedIPs = remote cell's VPN subnet).
- A CoreDNS forwarding block so `*.pic1.cell` resolves across the tunnel.

The **Connected Cells** panel shows live handshake status (green = online).

### Same-LAN tip

If both cells share the same external IP (behind NAT), the auto-detected
endpoint in the invite will be the public IP. Replace it with the LAN IP
before clicking Connect so traffic stays local:

```json
{ "endpoint": "192.168.31.50:51820", ... }
```

---

## 🏗️ Architecture

### **Service Manager Architecture**

All services inherit from `BaseServiceManager`, providing:
- **Unified Interface**: Consistent methods across all services
- **Health Monitoring**: Standardized health checks and metrics
- **Error Handling**: Centralized error handling and logging
- **Configuration**: Common configuration management patterns

### **Event-Driven Service Bus**

```python
# Services communicate via events
service_bus.register_service('network', network_manager)
service_bus.register_service('wireguard', wireguard_manager)
service_bus.publish_event(EventType.SERVICE_STARTED, 'network', data)
```

### **Service Dependencies**

```
wireguard → network
email → network, vault
calendar → network, vault
files → network, vault
routing → network, wireguard
vault → network
```

---

## 🔧 Service Managers

### **Core Network Services**
- **NetworkManager**: DNS, DHCP, NTP with dynamic zone management
- **WireGuardManager**: VPN configuration, peer management, key generation
- **PeerRegistry**: Peer registration, IP updates, trust management

### **Digital Services**
- **EmailManager**: SMTP/IMAP email with user management
- **CalendarManager**: CalDAV/CardDAV calendar and contacts
- **FileManager**: WebDAV file storage with user directories

### **Infrastructure Services**
- **RoutingManager**: NAT, firewall, advanced routing (exit/bridge/split)
- **VaultManager**: Certificate authority, trust management, encryption
- **ContainerManager**: Docker orchestration and container management
- **CellManager**: Overall cell configuration and service orchestration

---

## 📡 API Reference

### **Core Endpoints**

```bash
# Service Status
GET /api/services/status
GET /api/services/connectivity

# Configuration Management
GET /api/config
PUT /api/config
POST /api/config/backup
POST /api/config/restore/<backup_id>

# Service Bus
GET /api/services/bus/status
GET /api/services/bus/events
POST /api/services/bus/services/<service>/start

# Logging
GET /api/logs/services/<service>
POST /api/logs/search
POST /api/logs/export
```

### **Service-Specific Endpoints**

```bash
# Network Services
GET /api/dns/records
POST /api/dns/records
GET /api/dhcp/leases
GET /api/ntp/status

# WireGuard & Peers
GET /api/wireguard/peers
POST /api/wireguard/peers
GET /api/wireguard/status

# Digital Services
GET /api/email/users
GET /api/calendar/users
GET /api/files/users

# Routing & Security
GET /api/routing/status
POST /api/routing/nat
GET /api/vault/certificates
```

---

## 💻 CLI Guide

### **Enhanced CLI Features**

```bash
# Interactive Mode
python api/enhanced_cli.py --interactive

# Batch Operations
python api/enhanced_cli.py --batch "status" "services" "health"

# Configuration Management
python api/enhanced_cli.py --export-config json
python api/enhanced_cli.py --import-config config.json

# Service Wizards
python api/enhanced_cli.py --wizard network
python api/enhanced_cli.py --wizard email

# Health Monitoring
python api/enhanced_cli.py --health
python api/enhanced_cli.py --logs network
```

### **Service Management**

```bash
# Show status
python api/enhanced_cli.py --status

# List services
python api/enhanced_cli.py --services

# Peer management
python api/enhanced_cli.py --peers

# Service logs
python api/enhanced_cli.py --logs wireguard
```

---

## ⚙️ Configuration

### **Configuration Management**

```bash
# Export configuration
curl -X GET http://localhost:3000/api/config

# Update configuration
curl -X PUT http://localhost:3000/api/config \
  -H "Content-Type: application/json" \
  -d '{"cell_name": "mycell", "domain": "mycell.cell"}'

# Backup configuration
curl -X POST http://localhost:3000/api/config/backup
```

### **Service Configuration**

Each service has its own configuration schema:
- **Network**: DNS zones, DHCP ranges, NTP servers
- **WireGuard**: Interface settings, peer configurations
- **Email**: Domain settings, user accounts, mailboxes
- **Calendar**: User accounts, calendar sharing
- **Files**: Storage quotas, user directories
- **Routing**: NAT rules, firewall policies, routing tables

---

## 🔒 Security

### **Certificate Management**
- **Self-hosted CA**: Issue and manage TLS certificates
- **Certificate Lifecycle**: Generate, renew, revoke certificates
- **Trust Management**: Direct and indirect trust relationships
- **Age Encryption**: Modern encryption for sensitive data

### **Network Security**
- **WireGuard VPN**: Secure peer-to-peer communication
- **Firewall & NAT**: Granular access control
- **Service Isolation**: Docker containers for each service
- **Input Validation**: All API endpoints validate input

### **Data Protection**
- **Encrypted Storage**: Sensitive data encrypted at rest
- **Secure Communication**: TLS for all API endpoints
- **Access Control**: Role-based access for services
- **Audit Logging**: Comprehensive security event logging

---

## 🛠️ Development

### **Project Structure**

```
PersonalInternetCell/
├── api/                    # Backend API server
│   ├── base_service_manager.py    # Base class for all services
│   ├── config_manager.py          # Configuration management
│   ├── service_bus.py             # Event-driven service bus
│   ├── log_manager.py             # Comprehensive logging
│   ├── enhanced_cli.py            # Enhanced CLI tool
│   ├── network_manager.py         # DNS, DHCP, NTP
│   ├── wireguard_manager.py       # VPN and peer management
│   ├── email_manager.py           # Email services
│   ├── calendar_manager.py        # Calendar services
│   ├── file_manager.py            # File storage
│   ├── routing_manager.py         # Routing and NAT
│   ├── vault_manager.py           # Security and trust
│   ├── container_manager.py       # Container orchestration
│   ├── cell_manager.py            # Overall cell management
│   ├── peer_registry.py           # Peer registration
│   └── app.py                     # Main API server
├── webui/                  # React frontend
├── config/                 # Configuration files
├── data/                   # Persistent data
├── tests/                  # Test suite
└── docker-compose.yml      # Container orchestration
```

### **Running Locally**

```bash
# Install dependencies
pip install -r api/requirements.txt

# Start the API server
python api/app.py

# Run tests
python api/test_enhanced_api.py

# Start frontend (if available)
cd webui && bun install && npm run dev
```

### **Service Development**

```python
from base_service_manager import BaseServiceManager

class MyServiceManager(BaseServiceManager):
    def __init__(self, data_dir='/app/data', config_dir='/app/config'):
        super().__init__('myservice', data_dir, config_dir)
    
    def get_status(self) -> Dict[str, Any]:
        # Implement service status
        pass
    
    def test_connectivity(self) -> Dict[str, Any]:
        # Implement connectivity test
        pass
```

---

## 🧪 Testing

### **Test Suite**

```bash
# Run all tests
python api/test_enhanced_api.py

# Test specific components
python -m pytest api/tests/test_network_manager.py
python -m pytest api/tests/test_service_bus.py

# Coverage report
coverage run -m pytest api/tests/
coverage html
```

### **Test Coverage**
- **BaseServiceManager**: 100% coverage
- **ConfigManager**: 95%+ coverage
- **ServiceBus**: 95%+ coverage
- **LogManager**: 95%+ coverage
- **All Service Managers**: 77%+ overall coverage

---

## 🚀 Deployment

### **Docker Deployment**

```bash
# Production deployment
docker-compose -f docker-compose.prod.yml up -d

# Development deployment
docker-compose up --build
```

### **System Requirements**
- **CPU**: 2+ cores
- **RAM**: 2GB+ (4GB recommended)
- **Storage**: 10GB+ (SSD recommended)
- **Network**: Stable internet connection

### **Monitoring**

```bash
# Health check
curl http://localhost:3000/health

# Service status
curl http://localhost:3000/api/services/status

# Service connectivity
curl http://localhost:3000/api/services/connectivity
```

---

## 🤝 Contributing

1. **Fork** the repository
2. **Create** a feature branch
3. **Implement** your changes
4. **Add tests** for new functionality
5. **Submit** a pull request

### **Development Guidelines**
- Follow the existing code style
- Add comprehensive tests
- Update documentation
- Use the BaseServiceManager pattern
- Implement proper error handling

---

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

---

## 📚 Documentation

- **[Quick Start Guide](QUICKSTART.md)**: Get up and running quickly
- **[API Documentation](api/API_DOCUMENTATION.md)**: Complete API reference
- **[Comprehensive Improvements](COMPREHENSIVE_IMPROVEMENTS_SUMMARY.md)**: Detailed architecture overview
- **[Enhanced API Improvements](ENHANCED_API_IMPROVEMENTS.md)**: Technical implementation details

---

**🌟 The Personal Internet Cell - Your self-hosted, production-grade digital infrastructure!**