init

2025-09-12 23:04:52 +03:00
commit 2277b11563
127 changed files with 23640 additions and 0 deletions
@@ -0,0 +1,535 @@
+# Personal Internet Cell – Project Wiki
+
+## 🌟 Overview
+
+Personal Internet Cell is a **production-grade, self-hosted, decentralized digital infrastructure** solution designed to provide individuals with full control over their digital services and data. The project has evolved from a phase-based implementation to a **unified, enterprise-ready system** with modern architecture, comprehensive testing, and production-grade features.
+
+## 📋 Table of Contents
+
+1. [Project Goals](#project-goals)
+2. [Architecture & Components](#architecture--components)
+3. [Service Manager Architecture](#service-manager-architecture)
+4. [Core Services](#core-services)
+5. [API Reference](#api-reference)
+6. [Enhanced CLI](#enhanced-cli)
+7. [Security Model](#security-model)
+8. [Testing & Quality Assurance](#testing--quality-assurance)
+9. [Usage Examples](#usage-examples)
+10. [Development & Deployment](#development--deployment)
+11. [Future Enhancements](#future-enhancements)
+12. [Project Status](#project-status)
+
+## 🎯 Project Goals
+
+- **Self-Hosted**: Run your own digital services (email, calendar, files, VPN, etc.) on your hardware
+- **Decentralized**: Peer-to-peer networking and trust, no central authority
+- **Production-Grade**: Enterprise-ready architecture with comprehensive monitoring
+- **Secure**: Modern cryptography, certificate management, and encrypted storage
+- **User-Friendly**: Professional CLI and API for easy management
+- **Extensible**: Modular architecture for future services and integrations
+- **Event-Driven**: Real-time service communication and orchestration
+
+## 🏗️ Architecture & Components
+
+### **Modern Architecture Stack**
+
+- **Backend**: Python (Flask) with production-grade service managers
+- **Service Architecture**: BaseServiceManager pattern with unified interfaces
+- **Event System**: Service bus for real-time communication and orchestration
+- **Configuration**: Centralized configuration management with validation
+- **Logging**: Structured JSON logging with rotation and search
+- **Containerization**: Docker-based deployment and service isolation
+- **API**: RESTful endpoints with comprehensive documentation
+
+### **Core Architecture Components**
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Personal Internet Cell                   │
+├─────────────────────────────────────────────────────────────┤
+│  Enhanced CLI │ Web UI │ REST API │ Service Bus │ Logging   │
+├─────────────────────────────────────────────────────────────┤
+│                    Service Managers                         │
+│  Network │ WireGuard │ Email │ Calendar │ Files │ Routing   │
+│  Vault   │ Container │ Cell  │ Peer     │       │           │
+├─────────────────────────────────────────────────────────────┤
+│                    Core Infrastructure                      │
+│  DNS │ DHCP │ NTP │ VPN │ CA │ Encryption │ Trust │ Storage │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## 🔧 Service Manager Architecture
+
+### **BaseServiceManager Pattern**
+
+All services inherit from `BaseServiceManager`, providing:
+
+```python
+class BaseServiceManager(ABC):
+    def __init__(self, service_name: str, data_dir: str, config_dir: str)
+    
+    @abstractmethod
+    def get_status(self) -> Dict[str, Any]
+    
+    @abstractmethod
+    def test_connectivity(self) -> Dict[str, Any]
+    
+    # Common methods
+    def get_logs(self, lines: int = 50) -> List[str]
+    def restart_service(self) -> bool
+    def get_config(self) -> Dict[str, Any]
+    def update_config(self, config: Dict[str, Any]) -> bool
+    def health_check(self) -> Dict[str, Any]
+    def handle_error(self, error: Exception, context: str) -> Dict[str, Any]
+```
+
+### **Service Bus Integration**
+
+```python
+# Event-driven service communication
+service_bus.register_service('network', network_manager)
+service_bus.register_service('wireguard', wireguard_manager)
+service_bus.publish_event(EventType.SERVICE_STARTED, 'network', data)
+
+# Service dependencies
+service_dependencies = {
+    'wireguard': ['network'],
+    'email': ['network', 'vault'],
+    'calendar': ['network', 'vault'],
+    'files': ['network', 'vault'],
+    'routing': ['network', 'wireguard'],
+    'vault': ['network']
+}
+```
+
+## 🔧 Core Services
+
+### **Network Services**
+- **NetworkManager**: DNS, DHCP, NTP with dynamic management
+  - Dynamic zone file generation
+  - DHCP lease monitoring
+  - Network connectivity testing
+  - Service health monitoring
+
+### **VPN & Mesh Networking**
+- **WireGuardManager**: WireGuard VPN configuration and peer management
+  - Key generation and management
+  - Peer configuration
+  - Connectivity testing
+  - Dynamic IP updates
+
+- **PeerRegistry**: Peer registration and trust management
+  - Peer lifecycle management
+  - Trust relationship tracking
+  - Data integrity validation
+  - Peer statistics
+
+### **Digital Services**
+- **EmailManager**: SMTP/IMAP email services
+  - User account management
+  - Mailbox configuration
+  - Service connectivity testing
+  - Email delivery monitoring
+
+- **CalendarManager**: CalDAV/CardDAV calendar and contacts
+  - User and calendar management
+  - Event synchronization
+  - Service health monitoring
+  - Connectivity testing
+
+- **FileManager**: WebDAV file storage
+  - User directory management
+  - Storage quota monitoring
+  - File system access testing
+  - Backup and restore capabilities
+
+### **Infrastructure Services**
+- **RoutingManager**: Advanced routing and NAT
+  - NAT rule management
+  - Firewall configuration
+  - Exit node routing
+  - Bridge and split routing
+  - Connectivity testing
+
+- **VaultManager**: Security and trust management
+  - Self-hosted Certificate Authority
+  - Certificate lifecycle management
+  - Age/Fernet encryption
+  - Trust relationship management
+  - Cryptographic verification
+
+- **ContainerManager**: Docker orchestration
+  - Container lifecycle management
+  - Image and volume management
+  - Docker daemon connectivity
+  - Service isolation
+
+- **CellManager**: Overall cell orchestration
+  - Service coordination
+  - Health monitoring
+  - Configuration management
+  - Peer management
+
+## 📡 API Reference
+
+### **Core API Endpoints**
+
+```bash
+# Service Status and Health
+GET /api/services/status          # All services status
+GET /api/services/connectivity    # Service connectivity tests
+GET /health                       # API health check
+
+# Configuration Management
+GET /api/config                   # Get configuration
+PUT /api/config                   # Update configuration
+POST /api/config/backup           # Create backup
+GET /api/config/backups           # List backups
+POST /api/config/restore/<id>     # Restore backup
+GET /api/config/export            # Export configuration
+POST /api/config/import           # Import configuration
+
+# Service Bus
+GET /api/services/bus/status      # Service bus status
+GET /api/services/bus/events      # Event history
+POST /api/services/bus/services/<service>/start
+POST /api/services/bus/services/<service>/stop
+POST /api/services/bus/services/<service>/restart
+
+# Logging
+GET /api/logs/services/<service>  # Service logs
+POST /api/logs/search             # Log search
+POST /api/logs/export             # Log export
+GET /api/logs/statistics          # Log statistics
+POST /api/logs/rotate             # Log rotation
+```
+
+### **Service-Specific Endpoints**
+
+```bash
+# Network Services
+GET /api/dns/records              # DNS records
+POST /api/dns/records             # Add DNS record
+DELETE /api/dns/records           # Remove DNS record
+GET /api/dhcp/leases              # DHCP leases
+POST /api/dhcp/reservations       # Add DHCP reservation
+GET /api/ntp/status               # NTP status
+GET /api/network/info             # Network information
+POST /api/network/test            # Network connectivity test
+
+# WireGuard & Peers
+GET /api/wireguard/keys           # WireGuard keys
+POST /api/wireguard/keys/peer     # Generate peer keys
+GET /api/wireguard/config         # WireGuard configuration
+GET /api/wireguard/peers          # List peers
+POST /api/wireguard/peers         # Add peer
+DELETE /api/wireguard/peers       # Remove peer
+GET /api/wireguard/status         # WireGuard status
+POST /api/wireguard/connectivity  # Connectivity test
+PUT /api/wireguard/peers/ip       # Update peer IP
+
+# Digital Services
+GET /api/email/users              # Email users
+POST /api/email/users             # Add email user
+DELETE /api/email/users/<user>    # Remove email user
+GET /api/email/status             # Email service status
+GET /api/email/connectivity       # Email connectivity
+POST /api/email/send              # Send email
+GET /api/email/mailbox/<user>     # User mailbox
+
+GET /api/calendar/users           # Calendar users
+POST /api/calendar/users          # Add calendar user
+DELETE /api/calendar/users/<user> # Remove calendar user
+POST /api/calendar/calendars      # Create calendar
+POST /api/calendar/events         # Add event
+GET /api/calendar/events/<user>/<calendar> # List events
+GET /api/calendar/status          # Calendar service status
+GET /api/calendar/connectivity    # Calendar connectivity
+
+GET /api/files/users              # File users
+POST /api/files/users             # Add file user
+DELETE /api/files/users/<user>    # Remove file user
+POST /api/files/folders           # Create folder
+DELETE /api/files/folders/<user>/<path> # Remove folder
+POST /api/files/upload/<user>     # Upload file
+GET /api/files/download/<user>/<path> # Download file
+DELETE /api/files/delete/<user>/<path> # Delete file
+GET /api/files/list/<user>        # List files
+GET /api/files/status             # File service status
+GET /api/files/connectivity       # File connectivity
+
+# Routing & Security
+GET /api/routing/status           # Routing status
+POST /api/routing/nat             # Add NAT rule
+DELETE /api/routing/nat/<id>      # Remove NAT rule
+POST /api/routing/peers           # Add peer route
+DELETE /api/routing/peers/<peer>  # Remove peer route
+POST /api/routing/exit-nodes      # Add exit node
+POST /api/routing/bridge          # Add bridge route
+POST /api/routing/split           # Add split route
+POST /api/routing/firewall        # Add firewall rule
+POST /api/routing/connectivity    # Routing connectivity test
+GET /api/routing/logs             # Routing logs
+GET /api/routing/nat              # List NAT rules
+GET /api/routing/peers            # List peer routes
+GET /api/routing/firewall         # List firewall rules
+
+GET /api/vault/status             # Vault status
+GET /api/vault/certificates       # List certificates
+POST /api/vault/certificates      # Generate certificate
+DELETE /api/vault/certificates/<name> # Revoke certificate
+GET /api/vault/ca/certificate     # CA certificate
+GET /api/vault/age/public-key     # Age public key
+GET /api/vault/trust/keys         # Trusted keys
+POST /api/vault/trust/keys        # Add trusted key
+DELETE /api/vault/trust/keys/<name> # Remove trusted key
+POST /api/vault/trust/verify      # Verify trust
+GET /api/vault/trust/chains       # Trust chains
+```
+
+## 💻 Enhanced CLI
+
+### **CLI Features**
+
+```bash
+# Interactive mode with tab completion
+python api/enhanced_cli.py --interactive
+
+# Batch operations
+python api/enhanced_cli.py --batch "status" "services" "health"
+
+# Configuration management
+python api/enhanced_cli.py --export-config json
+python api/enhanced_cli.py --import-config config.json
+
+# Service wizards
+python api/enhanced_cli.py --wizard network
+python api/enhanced_cli.py --wizard email
+
+# Health monitoring
+python api/enhanced_cli.py --health
+python api/enhanced_cli.py --logs network
+
+# Service status
+python api/enhanced_cli.py --status
+python api/enhanced_cli.py --services
+python api/enhanced_cli.py --peers
+```
+
+### **CLI Capabilities**
+- **Interactive Mode**: Tab completion, command history, help system
+- **Batch Operations**: Execute multiple commands in sequence
+- **Configuration Wizards**: Guided setup for complex services
+- **Real-time Monitoring**: Live status updates and health checks
+- **Log Management**: View, search, and export service logs
+- **Service Management**: Start, stop, restart, and configure services
+
+## 🔒 Security Model
+
+### **Certificate Management**
+- **Self-hosted CA**: Issue and manage TLS certificates for all services
+- **Certificate Lifecycle**: Generate, renew, revoke, and monitor certificates
+- **Trust Management**: Direct, indirect, and verified trust relationships
+- **Age Encryption**: Modern encryption for sensitive data and keys
+
+### **Network Security**
+- **WireGuard VPN**: Secure peer-to-peer communication with key rotation
+- **Firewall & NAT**: Granular control over network access and routing
+- **Service Isolation**: Docker containers for each service
+- **Input Validation**: All API endpoints validate and sanitize input
+
+### **Data Protection**
+- **Encrypted Storage**: Sensitive data encrypted at rest using Age/Fernet
+- **Secure Communication**: TLS for all API endpoints and service communication
+- **Access Control**: Role-based access for services and API endpoints
+- **Audit Logging**: Comprehensive security event logging and monitoring
+
+## 🧪 Testing & Quality Assurance
+
+### **Test Coverage**
+- **BaseServiceManager**: 100% coverage
+- **ConfigManager**: 95%+ coverage
+- **ServiceBus**: 95%+ coverage
+- **LogManager**: 95%+ coverage
+- **All Service Managers**: 77%+ overall coverage
+- **API Endpoints**: 100% endpoint coverage
+
+### **Test Types**
+- **Unit Tests**: Individual component testing
+- **Integration Tests**: Service interaction testing
+- **API Tests**: Endpoint functionality testing
+- **Error Handling**: Exception and edge case testing
+- **Performance Tests**: Load and stress testing
+
+### **Testing Commands**
+```bash
+# Run all tests
+python api/test_enhanced_api.py
+
+# Run specific test suites
+python -m pytest api/tests/test_network_manager.py
+python -m pytest api/tests/test_service_bus.py
+
+# Generate coverage report
+coverage run -m pytest api/tests/
+coverage html
+```
+
+## 📝 Usage Examples
+
+### **Add DNS Record**
+```bash
+curl -X POST http://localhost:3000/api/dns/records \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "www",
+    "type": "A",
+    "value": "192.168.1.100",
+    "ttl": 300
+  }'
+```
+
+### **Register Peer**
+```bash
+curl -X POST http://localhost:3000/api/wireguard/peers \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "bob",
+    "ip": "203.0.113.22",
+    "public_key": "peer_public_key_here",
+    "allowed_networks": ["10.0.0.0/24"]
+  }'
+```
+
+### **Generate Certificate**
+```bash
+curl -X POST http://localhost:3000/api/vault/certificates \
+  -H "Content-Type: application/json" \
+  -d '{
+    "common_name": "myapp.example.com",
+    "domains": ["myapp.example.com", "www.myapp.example.com"],
+    "days": 365
+  }'
+```
+
+### **Configure NAT Rule**
+```bash
+curl -X POST http://localhost:3000/api/routing/nat \
+  -H "Content-Type: application/json" \
+  -d '{
+    "source_network": "10.0.0.0/24",
+    "target_interface": "eth0",
+    "nat_type": "MASQUERADE",
+    "protocol": "ALL"
+  }'
+```
+
+## 🛠️ Development & Deployment
+
+### **Development Setup**
+```bash
+# Install dependencies
+pip install -r api/requirements.txt
+
+# Start development server
+python api/app.py
+
+# Run tests
+python api/test_enhanced_api.py
+
+# Start frontend (if available)
+cd webui && npm install && npm run dev
+```
+
+### **Production Deployment**
+```bash
+# Docker deployment
+docker-compose up --build -d
+
+# Health check
+curl http://localhost:3000/health
+
+# Service status
+curl http://localhost:3000/api/services/status
+```
+
+### **Service Development**
+```python
+from base_service_manager import BaseServiceManager
+
+class MyServiceManager(BaseServiceManager):
+    def __init__(self, data_dir='/app/data', config_dir='/app/config'):
+        super().__init__('myservice', data_dir, config_dir)
+    
+    def get_status(self) -> Dict[str, Any]:
+        # Implement service status
+        return {
+            'running': True,
+            'status': 'online',
+            'timestamp': datetime.utcnow().isoformat()
+        }
+    
+    def test_connectivity(self) -> Dict[str, Any]:
+        # Implement connectivity test
+        return {
+            'success': True,
+            'message': 'Service connectivity working',
+            'timestamp': datetime.utcnow().isoformat()
+        }
+```
+
+## 🚀 Future Enhancements
+
+### **Planned Features**
+- **Certificate Auto-renewal**: Automatic certificate renewal and monitoring
+- **Web of Trust Models**: Advanced trust relationship management
+- **Certificate Transparency**: CT log integration and monitoring
+- **Hardware Security Module (HSM)**: HSM integration for key management
+- **WebSocket Updates**: Real-time service status updates
+- **Advanced Monitoring**: Metrics collection and alerting systems
+- **Mobile App**: Mobile application for remote management
+- **Plugin System**: Extensible architecture for custom services
+
+### **Architecture Improvements**
+- **Service Discovery**: Dynamic service registration and discovery
+- **Load Balancing**: Multi-instance service deployment
+- **Advanced Caching**: Redis-based caching for performance
+- **Message Queues**: RabbitMQ/Kafka for reliable messaging
+- **Distributed Tracing**: OpenTelemetry integration
+- **Configuration Management**: GitOps-style configuration management
+
+## 📊 Project Status
+
+### **✅ Completed Features**
+- **Production-Grade Architecture**: BaseServiceManager pattern implemented
+- **Event-Driven Communication**: Service bus with real-time events
+- **Centralized Configuration**: Type-safe configuration with validation
+- **Comprehensive Logging**: Structured logging with search and export
+- **Enhanced CLI**: Interactive CLI with batch operations
+- **Health Monitoring**: Real-time health checks across all services
+- **Security Framework**: Self-hosted CA, encryption, and trust management
+- **Complete API**: RESTful API with comprehensive documentation
+- **Testing Framework**: Comprehensive test suite with high coverage
+
+### **🎯 Current Status**
+- **All Services**: 10 service managers fully implemented and integrated
+- **API Server**: Running on port 3000 with all endpoints functional
+- **CLI Tool**: Enhanced CLI with all features working
+- **Test Coverage**: 77%+ overall coverage with comprehensive testing
+- **Documentation**: Complete documentation for all components
+- **Production Ready**: Suitable for personal and small business deployment
+
+### **🌟 Key Achievements**
+- **Unified Architecture**: All services follow the same patterns and interfaces
+- **Event-Driven Design**: Services communicate and orchestrate automatically
+- **Configuration Management**: Centralized, validated configuration system
+- **Comprehensive Logging**: Production-grade logging with advanced features
+- **Enhanced CLI**: Professional command-line interface for management
+- **Health Monitoring**: Real-time monitoring and alerting capabilities
+- **Security Framework**: Enterprise-grade security with modern cryptography
+- **Complete Testing**: Comprehensive test suite ensuring reliability
+
+---
+
+**The Personal Internet Cell empowers users with full control over their digital infrastructure, combining privacy, security, and usability in a single, production-ready, self-hosted platform.** 🌟
+