pic/Personal Internet Cell – Project Wiki.md

# Personal Internet Cell – Project Wiki

## 🌟 Overview

Personal Internet Cell is a **production-grade, self-hosted, decentralized digital infrastructure** solution designed to provide individuals with full control over their digital services and data. The project has evolved from a phase-based implementation to a **unified, enterprise-ready system** with modern architecture, comprehensive testing, and production-grade features.

## 📋 Table of Contents

1. [Project Goals](#project-goals)
2. [Architecture & Components](#architecture--components)
3. [Service Manager Architecture](#service-manager-architecture)
4. [Core Services](#core-services)
5. [API Reference](#api-reference)
6. [Enhanced CLI](#enhanced-cli)
7. [Security Model](#security-model)
8. [Testing & Quality Assurance](#testing--quality-assurance)
9. [Usage Examples](#usage-examples)
10. [Development & Deployment](#development--deployment)
11. [Future Enhancements](#future-enhancements)
12. [Project Status](#project-status)

## 🎯 Project Goals

- **Self-Hosted**: Run your own digital services (email, calendar, files, VPN, etc.) on your hardware
- **Decentralized**: Peer-to-peer networking and trust, no central authority
- **Production-Grade**: Enterprise-ready architecture with comprehensive monitoring
- **Secure**: Modern cryptography, certificate management, and encrypted storage
- **User-Friendly**: Professional CLI and API for easy management
- **Extensible**: Modular architecture for future services and integrations
- **Event-Driven**: Real-time service communication and orchestration

## 🏗️ Architecture & Components

### **Modern Architecture Stack**

- **Backend**: Python (Flask) with production-grade service managers
- **Service Architecture**: BaseServiceManager pattern with unified interfaces
- **Event System**: Service bus for real-time communication and orchestration
- **Configuration**: Centralized configuration management with validation
- **Logging**: Structured JSON logging with rotation and search
- **Containerization**: Docker-based deployment and service isolation
- **API**: RESTful endpoints with comprehensive documentation

### **Core Architecture Components**

```
┌─────────────────────────────────────────────────────────────┐
│                    Personal Internet Cell                   │
├─────────────────────────────────────────────────────────────┤
│  Enhanced CLI │ Web UI │ REST API │ Service Bus │ Logging   │
├─────────────────────────────────────────────────────────────┤
│                    Service Managers                         │
│  Network │ WireGuard │ Email │ Calendar │ Files │ Routing   │
│  Vault   │ Container │ Cell  │ Peer     │       │           │
├─────────────────────────────────────────────────────────────┤
│                    Core Infrastructure                      │
│  DNS │ DHCP │ NTP │ VPN │ CA │ Encryption │ Trust │ Storage │
└─────────────────────────────────────────────────────────────┘
```

## 🔧 Service Manager Architecture

### **BaseServiceManager Pattern**

All services inherit from `BaseServiceManager`, providing:

```python
class BaseServiceManager(ABC):
    def __init__(self, service_name: str, data_dir: str, config_dir: str)
    
    @abstractmethod
    def get_status(self) -> Dict[str, Any]
    
    @abstractmethod
    def test_connectivity(self) -> Dict[str, Any]
    
    # Common methods
    def get_logs(self, lines: int = 50) -> List[str]
    def restart_service(self) -> bool
    def get_config(self) -> Dict[str, Any]
    def update_config(self, config: Dict[str, Any]) -> bool
    def health_check(self) -> Dict[str, Any]
    def handle_error(self, error: Exception, context: str) -> Dict[str, Any]
```

### **Service Bus Integration**

```python
# Event-driven service communication
service_bus.register_service('network', network_manager)
service_bus.register_service('wireguard', wireguard_manager)
service_bus.publish_event(EventType.SERVICE_STARTED, 'network', data)

# Service dependencies
service_dependencies = {
    'wireguard': ['network'],
    'email': ['network', 'vault'],
    'calendar': ['network', 'vault'],
    'files': ['network', 'vault'],
    'routing': ['network', 'wireguard'],
    'vault': ['network']
}
```

## 🔧 Core Services

### **Network Services**
- **NetworkManager**: DNS, DHCP, NTP with dynamic management
  - Dynamic zone file generation
  - DHCP lease monitoring
  - Network connectivity testing
  - Service health monitoring

### **VPN & Mesh Networking**
- **WireGuardManager**: WireGuard VPN configuration and peer management
  - Key generation and management
  - Peer configuration
  - Connectivity testing
  - Dynamic IP updates

- **PeerRegistry**: Peer registration and trust management
  - Peer lifecycle management
  - Trust relationship tracking
  - Data integrity validation
  - Peer statistics

### **Digital Services**
- **EmailManager**: SMTP/IMAP email services
  - User account management
  - Mailbox configuration
  - Service connectivity testing
  - Email delivery monitoring

- **CalendarManager**: CalDAV/CardDAV calendar and contacts
  - User and calendar management
  - Event synchronization
  - Service health monitoring
  - Connectivity testing

- **FileManager**: WebDAV file storage
  - User directory management
  - Storage quota monitoring
  - File system access testing
  - Backup and restore capabilities

### **Infrastructure Services**
- **RoutingManager**: Advanced routing and NAT
  - NAT rule management
  - Firewall configuration
  - Exit node routing
  - Bridge and split routing
  - Connectivity testing

- **VaultManager**: Security and trust management
  - Self-hosted Certificate Authority
  - Certificate lifecycle management
  - Age/Fernet encryption
  - Trust relationship management
  - Cryptographic verification

- **ContainerManager**: Docker orchestration
  - Container lifecycle management
  - Image and volume management
  - Docker daemon connectivity
  - Service isolation

- **CellManager**: Overall cell orchestration
  - Service coordination
  - Health monitoring
  - Configuration management
  - Peer management

## 📡 API Reference

### **Core API Endpoints**

```bash
# Service Status and Health
GET /api/services/status          # All services status
GET /api/services/connectivity    # Service connectivity tests
GET /health                       # API health check

# Configuration Management
GET /api/config                   # Get configuration
PUT /api/config                   # Update configuration
POST /api/config/backup           # Create backup
GET /api/config/backups           # List backups
POST /api/config/restore/<id>     # Restore backup
GET /api/config/export            # Export configuration
POST /api/config/import           # Import configuration

# Service Bus
GET /api/services/bus/status      # Service bus status
GET /api/services/bus/events      # Event history
POST /api/services/bus/services/<service>/start
POST /api/services/bus/services/<service>/stop
POST /api/services/bus/services/<service>/restart

# Logging
GET /api/logs/services/<service>  # Service logs
POST /api/logs/search             # Log search
POST /api/logs/export             # Log export
GET /api/logs/statistics          # Log statistics
POST /api/logs/rotate             # Log rotation
```

### **Service-Specific Endpoints**

```bash
# Network Services
GET /api/dns/records              # DNS records
POST /api/dns/records             # Add DNS record
DELETE /api/dns/records           # Remove DNS record
GET /api/dhcp/leases              # DHCP leases
POST /api/dhcp/reservations       # Add DHCP reservation
GET /api/ntp/status               # NTP status
GET /api/network/info             # Network information
POST /api/network/test            # Network connectivity test

# WireGuard & Peers
GET /api/wireguard/keys           # WireGuard keys
POST /api/wireguard/keys/peer     # Generate peer keys
GET /api/wireguard/config         # WireGuard configuration
GET /api/wireguard/peers          # List peers
POST /api/wireguard/peers         # Add peer
DELETE /api/wireguard/peers       # Remove peer
GET /api/wireguard/status         # WireGuard status
POST /api/wireguard/connectivity  # Connectivity test
PUT /api/wireguard/peers/ip       # Update peer IP

# Digital Services
GET /api/email/users              # Email users
POST /api/email/users             # Add email user
DELETE /api/email/users/<user>    # Remove email user
GET /api/email/status             # Email service status
GET /api/email/connectivity       # Email connectivity
POST /api/email/send              # Send email
GET /api/email/mailbox/<user>     # User mailbox

GET /api/calendar/users           # Calendar users
POST /api/calendar/users          # Add calendar user
DELETE /api/calendar/users/<user> # Remove calendar user
POST /api/calendar/calendars      # Create calendar
POST /api/calendar/events         # Add event
GET /api/calendar/events/<user>/<calendar> # List events
GET /api/calendar/status          # Calendar service status
GET /api/calendar/connectivity    # Calendar connectivity

GET /api/files/users              # File users
POST /api/files/users             # Add file user
DELETE /api/files/users/<user>    # Remove file user
POST /api/files/folders           # Create folder
DELETE /api/files/folders/<user>/<path> # Remove folder
POST /api/files/upload/<user>     # Upload file
GET /api/files/download/<user>/<path> # Download file
DELETE /api/files/delete/<user>/<path> # Delete file
GET /api/files/list/<user>        # List files
GET /api/files/status             # File service status
GET /api/files/connectivity       # File connectivity

# Routing & Security
GET /api/routing/status           # Routing status
POST /api/routing/nat             # Add NAT rule
DELETE /api/routing/nat/<id>      # Remove NAT rule
POST /api/routing/peers           # Add peer route
DELETE /api/routing/peers/<peer>  # Remove peer route
POST /api/routing/exit-nodes      # Add exit node
POST /api/routing/bridge          # Add bridge route
POST /api/routing/split           # Add split route
POST /api/routing/firewall        # Add firewall rule
POST /api/routing/connectivity    # Routing connectivity test
GET /api/routing/logs             # Routing logs
GET /api/routing/nat              # List NAT rules
GET /api/routing/peers            # List peer routes
GET /api/routing/firewall         # List firewall rules

GET /api/vault/status             # Vault status
GET /api/vault/certificates       # List certificates
POST /api/vault/certificates      # Generate certificate
DELETE /api/vault/certificates/<name> # Revoke certificate
GET /api/vault/ca/certificate     # CA certificate
GET /api/vault/age/public-key     # Age public key
GET /api/vault/trust/keys         # Trusted keys
POST /api/vault/trust/keys        # Add trusted key
DELETE /api/vault/trust/keys/<name> # Remove trusted key
POST /api/vault/trust/verify      # Verify trust
GET /api/vault/trust/chains       # Trust chains
```

## 💻 Enhanced CLI

### **CLI Features**

```bash
# Interactive mode with tab completion
python api/enhanced_cli.py --interactive

# Batch operations
python api/enhanced_cli.py --batch "status" "services" "health"

# Configuration management
python api/enhanced_cli.py --export-config json
python api/enhanced_cli.py --import-config config.json

# Service wizards
python api/enhanced_cli.py --wizard network
python api/enhanced_cli.py --wizard email

# Health monitoring
python api/enhanced_cli.py --health
python api/enhanced_cli.py --logs network

# Service status
python api/enhanced_cli.py --status
python api/enhanced_cli.py --services
python api/enhanced_cli.py --peers
```

### **CLI Capabilities**
- **Interactive Mode**: Tab completion, command history, help system
- **Batch Operations**: Execute multiple commands in sequence
- **Configuration Wizards**: Guided setup for complex services
- **Real-time Monitoring**: Live status updates and health checks
- **Log Management**: View, search, and export service logs
- **Service Management**: Start, stop, restart, and configure services

## 🔒 Security Model

### **Certificate Management**
- **Self-hosted CA**: Issue and manage TLS certificates for all services
- **Certificate Lifecycle**: Generate, renew, revoke, and monitor certificates
- **Trust Management**: Direct, indirect, and verified trust relationships
- **Age Encryption**: Modern encryption for sensitive data and keys

### **Network Security**
- **WireGuard VPN**: Secure peer-to-peer communication with key rotation
- **Firewall & NAT**: Granular control over network access and routing
- **Service Isolation**: Docker containers for each service
- **Input Validation**: All API endpoints validate and sanitize input

### **Data Protection**
- **Encrypted Storage**: Sensitive data encrypted at rest using Age/Fernet
- **Secure Communication**: TLS for all API endpoints and service communication
- **Access Control**: Role-based access for services and API endpoints
- **Audit Logging**: Comprehensive security event logging and monitoring

## 🧪 Testing & Quality Assurance

### **Test Coverage**
- **BaseServiceManager**: 100% coverage
- **ConfigManager**: 95%+ coverage
- **ServiceBus**: 95%+ coverage
- **LogManager**: 95%+ coverage
- **All Service Managers**: 77%+ overall coverage
- **API Endpoints**: 100% endpoint coverage

### **Test Types**
- **Unit Tests**: Individual component testing
- **Integration Tests**: Service interaction testing
- **API Tests**: Endpoint functionality testing
- **Error Handling**: Exception and edge case testing
- **Performance Tests**: Load and stress testing

### **Testing Commands**
```bash
# Run all tests
python api/test_enhanced_api.py

# Run specific test suites
python -m pytest api/tests/test_network_manager.py
python -m pytest api/tests/test_service_bus.py

# Generate coverage report
coverage run -m pytest api/tests/
coverage html
```

## 📝 Usage Examples

### **Add DNS Record**
```bash
curl -X POST http://localhost:3000/api/dns/records \
  -H "Content-Type: application/json" \
  -d '{
    "name": "www",
    "type": "A",
    "value": "192.168.1.100",
    "ttl": 300
  }'
```

### **Register Peer**
```bash
curl -X POST http://localhost:3000/api/wireguard/peers \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bob",
    "ip": "203.0.113.22",
    "public_key": "peer_public_key_here",
    "allowed_networks": ["10.0.0.0/24"]
  }'
```

### **Generate Certificate**
```bash
curl -X POST http://localhost:3000/api/vault/certificates \
  -H "Content-Type: application/json" \
  -d '{
    "common_name": "myapp.example.com",
    "domains": ["myapp.example.com", "www.myapp.example.com"],
    "days": 365
  }'
```

### **Configure NAT Rule**
```bash
curl -X POST http://localhost:3000/api/routing/nat \
  -H "Content-Type: application/json" \
  -d '{
    "source_network": "10.0.0.0/24",
    "target_interface": "eth0",
    "nat_type": "MASQUERADE",
    "protocol": "ALL"
  }'
```

## 🛠️ Development & Deployment

### **Development Setup**
```bash
# Install dependencies
pip install -r api/requirements.txt

# Start development server
python api/app.py

# Run tests
python api/test_enhanced_api.py

# Start frontend (if available)
cd webui && bun install && npm run dev
```

### **Production Deployment**
```bash
# Docker deployment
docker-compose up --build -d

# Health check
curl http://localhost:3000/health

# Service status
curl http://localhost:3000/api/services/status
```

### **Service Development**
```python
from base_service_manager import BaseServiceManager

class MyServiceManager(BaseServiceManager):
    def __init__(self, data_dir='/app/data', config_dir='/app/config'):
        super().__init__('myservice', data_dir, config_dir)
    
    def get_status(self) -> Dict[str, Any]:
        # Implement service status
        return {
            'running': True,
            'status': 'online',
            'timestamp': datetime.utcnow().isoformat()
        }
    
    def test_connectivity(self) -> Dict[str, Any]:
        # Implement connectivity test
        return {
            'success': True,
            'message': 'Service connectivity working',
            'timestamp': datetime.utcnow().isoformat()
        }
```

## 🚀 Future Enhancements

### **Planned Features**
- **Certificate Auto-renewal**: Automatic certificate renewal and monitoring
- **Web of Trust Models**: Advanced trust relationship management
- **Certificate Transparency**: CT log integration and monitoring
- **Hardware Security Module (HSM)**: HSM integration for key management
- **WebSocket Updates**: Real-time service status updates
- **Advanced Monitoring**: Metrics collection and alerting systems
- **Mobile App**: Mobile application for remote management
- **Plugin System**: Extensible architecture for custom services

### **Architecture Improvements**
- **Service Discovery**: Dynamic service registration and discovery
- **Load Balancing**: Multi-instance service deployment
- **Advanced Caching**: Redis-based caching for performance
- **Message Queues**: RabbitMQ/Kafka for reliable messaging
- **Distributed Tracing**: OpenTelemetry integration
- **Configuration Management**: GitOps-style configuration management

## 📊 Project Status

### **✅ Completed Features**
- **Production-Grade Architecture**: BaseServiceManager pattern implemented
- **Event-Driven Communication**: Service bus with real-time events
- **Centralized Configuration**: Type-safe configuration with validation
- **Comprehensive Logging**: Structured logging with search and export
- **Enhanced CLI**: Interactive CLI with batch operations
- **Health Monitoring**: Real-time health checks across all services
- **Security Framework**: Self-hosted CA, encryption, and trust management
- **Complete API**: RESTful API with comprehensive documentation
- **Testing Framework**: Comprehensive test suite with high coverage

### **🎯 Current Status**
- **All Services**: 10 service managers fully implemented and integrated
- **API Server**: Running on port 3000 with all endpoints functional
- **CLI Tool**: Enhanced CLI with all features working
- **Test Coverage**: 77%+ overall coverage with comprehensive testing
- **Documentation**: Complete documentation for all components
- **Production Ready**: Suitable for personal and small business deployment

### **🌟 Key Achievements**
- **Unified Architecture**: All services follow the same patterns and interfaces
- **Event-Driven Design**: Services communicate and orchestrate automatically
- **Configuration Management**: Centralized, validated configuration system
- **Comprehensive Logging**: Production-grade logging with advanced features
- **Enhanced CLI**: Professional command-line interface for management
- **Health Monitoring**: Real-time monitoring and alerting capabilities
- **Security Framework**: Enterprise-grade security with modern cryptography
- **Complete Testing**: Comprehensive test suite ensuring reliability

---

**The Personal Internet Cell empowers users with full control over their digital infrastructure, combining privacy, security, and usability in a single, production-ready, self-hosted platform.** 🌟