Migration/pnpm-workspace

MAKEFILE_UPDATES.md

@@ -0,0 +1,121 @@
+# Makefile Updates Summary
+
+## Overview
+Updated the root-level Makefile based on analysis of the apps directory structure, incorporating service-specific commands and management capabilities from the individual applications.
+
+## Analysis Summary
+
+### Apps Directory Structure Analyzed
+- **Core Services**: jane-orchestrator (FastAPI), jane-ui (Next.js)
+- **AI Services**: cyber-llm-server, mtg-llm-server, infrastructure-llm-server (all Python/FastAPI)
+- **Security Services**: 17 services including nemesis, helk, misp, maltrail, ghostwriter, etc.
+- **Productivity Services**: 10 services including actual, mealie, firecrawl, goose, inspector, etc.
+- **TCG Services**: 4 services including commander-spellbook components
+
+### Key Patterns Discovered
+1. **NX Integration**: Most services use NX for build/serve/test orchestration via project.json
+2. **Docker Compose**: Many services have sophisticated Docker orchestration (nemesis, helk, misp)
+3. **Python Services**: Mix of FastAPI, Django, and custom Python applications
+4. **Node.js Services**: Next.js, Express, and custom TypeScript applications
+5. **Requirements Management**: Various requirements.txt and pyproject.toml files across services
+
+## New Commands Added
+
+### Enhanced Installation
+- **`install`**: Updated to include all Python requirements files and Node.js dependencies
+  - Automatically finds and installs from all requirements.txt files
+  - Runs npm install in all package.json directories
+
+### Service Management
+- **`serve-jane`**: Start core Jane UI and Orchestrator
+- **`serve-security`**: Start all security domain services
+- **`serve-productivity`**: Start all productivity domain services  
+- **`serve-tcg`**: Start all TCG domain services
+- **`status`**: Show running service status
+- **`health-check`**: Check service health endpoints
+
+### Docker Service Management
+- **`docker-serve-nemesis`**: Start Nemesis service stack
+- **`docker-serve-helk`**: Start HELK service
+- **`docker-serve-misp`**: Start MISP service
+
+### Domain-Specific Testing
+- **`test-security`**: Test all security services
+- **`test-productivity`**: Test all productivity services
+- **`test-tcg`**: Test all TCG services
+- **`test-python`**: Test all Python services with pytest
+
+### Language-Specific Linting
+- **`lint-python`**: Lint Python code with ruff and black
+- **`lint-typescript`**: Lint TypeScript services
+
+### Development Support
+- **`setup-dev`**: Complete development environment setup
+- **`docs-build`**: Build documentation (supports Sphinx makefiles found)
+- **`clean-services`**: Clean service-specific artifacts
+
+### Enhanced Help
+- Reorganized help output into logical categories
+- Added descriptions for all new commands
+- Grouped commands by function (Basic, Service Management, Docker, Testing, etc.)
+
+## Files Analyzed
+
+### Makefiles Found
+- `apps/security/ghostwriter-service/DOCS/Makefile` - Sphinx documentation
+- `apps/security/meterpreter-service/Makefile` - Complex C/C++ build system
+- `apps/security/misp-service/src/app/Makefile` - PHP linting and testing
+- Multiple smaller component Makefiles in various services
+
+### Configuration Files
+- 50+ project.json files with NX build configurations
+- 30+ requirements.txt files with Python dependencies
+- 20+ pyproject.toml files with modern Python project configs
+- 10+ docker-compose.yml files for service orchestration
+
+### Key Dependencies Patterns
+- **FastAPI + Uvicorn**: Core API services
+- **Next.js + React**: Frontend applications
+- **Django**: Complex web applications (Ghostwriter)
+- **Docker Compose**: Multi-service orchestration
+- **PostgreSQL**: Primary database for most services
+- **Redis**: Caching and session management
+
+## Benefits of Updates
+
+1. **Domain Organization**: Commands organized by functional domains (security, productivity, TCG)
+2. **Service Discovery**: Easy way to start related services together
+3. **Development Workflow**: Streamlined setup and testing processes
+4. **Health Monitoring**: Built-in health checking and status reporting
+5. **Language-Specific Operations**: Tailored linting and testing by technology stack
+6. **Docker Integration**: Service-specific Docker orchestration commands
+
+## Usage Examples
+
+```bash
+# Setup complete development environment
+make setup-dev
+
+# Start all core Jane services
+make serve-jane
+
+# Start security analysis stack
+make serve-security
+
+# Test all Python services
+make test-python
+
+# Check service health
+make health-check
+
+# Clean everything
+make clean
+```
+
+## Next Steps
+
+1. **Service Health Endpoints**: Ensure all services implement `/health` endpoints
+2. **Environment Configuration**: Standardize environment variable management
+3. **Logging Integration**: Add centralized logging commands
+4. **Monitoring**: Add performance monitoring commands
+5. **Database Management**: Add database migration and backup commands

Makefile

@@ -1,46 +1,85 @@
 .PHONY: help install build test lint clean deploy docker-build docker-up docker-down
+.PHONY: serve-jane serve-security serve-productivity serve-tcg
+.PHONY: docker-serve-nemesis docker-serve-helk docker-serve-misp
+.PHONY: test-security test-productivity test-tcg test-python
+.PHONY: lint-python lint-typescript docs-build health-check status
+.PHONY: setup-dev clean-services
 
 # Default target
 help:
 	@echo "Jane Monorepo - Available Commands:"
 	@echo ""
+	@echo "BASIC OPERATIONS:"
 	@echo "  make install          - Install all dependencies (Node.js and Python)"
 	@echo "  make build            - Build all applications"
 	@echo "  make test             - Run all tests"
 	@echo "  make lint             - Lint all code"
 	@echo "  make clean            - Clean build artifacts"
 	@echo "  make deploy           - Deploy applications"
+	@echo ""
+	@echo "SERVICE MANAGEMENT:"
+	@echo "  make serve-jane       - Start Jane UI and Orchestrator"
+	@echo "  make serve-security   - Start all security services"
+	@echo "  make serve-productivity - Start all productivity services"
+	@echo "  make serve-tcg        - Start all TCG services"
+	@echo "  make status           - Show service status"
+	@echo "  make health-check     - Check service health"
+	@echo ""
+	@echo "DOCKER OPERATIONS:"
 	@echo "  make docker-build     - Build all Docker images"
 	@echo "  make docker-up        - Start Docker Compose services"
 	@echo "  make docker-down      - Stop Docker Compose services"
+	@echo "  make docker-serve-nemesis - Start Nemesis service stack"
+	@echo "  make docker-serve-helk    - Start HELK service"
+	@echo "  make docker-serve-misp    - Start MISP service"
+	@echo ""
+	@echo "TESTING BY DOMAIN:"
+	@echo "  make test-security    - Test security services"
+	@echo "  make test-productivity - Test productivity services"
+	@echo "  make test-tcg         - Test TCG services"
+	@echo "  make test-python      - Test Python services"
+	@echo ""
+	@echo "LINTING BY LANGUAGE:"
+	@echo "  make lint-python      - Lint Python code"
+	@echo "  make lint-typescript  - Lint TypeScript code"
+	@echo ""
+	@echo "DEVELOPMENT:"
+	@echo "  make setup-dev        - Setup complete development environment"
+	@echo "  make docs-build       - Build documentation"
+	@echo "  make clean-services   - Clean service-specific artifacts"
+	@echo ""
+	@echo "AFFECTED OPERATIONS:"
 	@echo "  make affected-build   - Build affected projects"
 	@echo "  make affected-test    - Test affected projects"
 	@echo "  make affected-lint    - Lint affected projects"
 	@echo ""
 
 # Install dependencies
 install:
-	npm install
+	pnpm install
 	@echo "Installing Python dependencies for apps..."
-	pip install -r apps/jane-orchestrator/requirements.txt
-	pip install -r apps/cyber-llm-server/requirements.txt
-	pip install -r apps/mtg-llm-server/requirements.txt
-	pip install -r apps/code-llm-server/requirements.txt
+	pip install -r apps/core/jane-orchestrator/requirements.txt
+	pip install -r apps/ai/cyber-llm-server/requirements.txt
+	pip install -r apps/ai/mtg-llm-server/requirements.txt
+	pip install -r apps/ai/infrastructure-llm-server/requirements.txt
+	@echo "Installing additional service dependencies..."
+	find apps/ -name "requirements.txt" -not -path "*/node_modules/*" -exec pip install -r {} \;
+	@echo "Node.js dependencies managed by pnpm workspace"
 
 # Build all applications
 build:
-	npx nx run-many --target=build --all
+	pnpm nx run-many --target=build --all
 
 # Run all tests
 test:
-	npx nx run-many --target=test --all
+	pnpm nx run-many --target=test --all
 
 # Lint all code
 lint:
-	npx nx run-many --target=lint --all
+	pnpm nx run-many --target=lint --all
 
 # Clean build artifacts
-clean:
+clean: clean-services
 	rm -rf dist/
 	rm -rf .next/
 	rm -rf build/
@@ -50,7 +89,7 @@ clean:
 
 # Deploy applications
 deploy:
-	npx nx run-many --target=deploy --all
+	pnpm nx run-many --target=deploy --all
 
 # Build Docker images
 docker-build:
@@ -74,16 +113,96 @@ staging:
 
 # Affected commands
 affected-build:
-	npx nx affected --target=build
+	pnpm nx affected --target=build
 
 affected-test:
-	npx nx affected --target=test
+	pnpm nx affected --target=test
 
 affected-lint:
-	npx nx affected --target=lint
+	pnpm nx affected --target=lint
 
 affected-deploy:
-	npx nx affected --target=deploy
+	pnpm nx affected --target=deploy
+
+# Service-specific commands
+serve-jane:
+	pnpm nx serve jane-ui & pnpm nx serve jane-orchestrator
+
+serve-security:
+	@echo "Starting security services..."
+	pnpm nx serve misp-service &
+	pnpm nx serve nemesis-service &
+	pnpm nx serve helk-service &
+	pnpm nx serve maltrail-service &
+	pnpm nx serve ghostwriter-service
+
+serve-productivity:
+	@echo "Starting productivity services..."
+	pnpm nx serve actual-service &
+	pnpm nx serve mealie-service &
+	pnpm nx serve firecrawl-service &
+	pnpm nx serve goose-service &
+	pnpm nx serve inspector-service
+
+serve-tcg:
+	@echo "Starting TCG services..."
+	pnpm nx serve commander-spellbook-site &
+	pnpm nx serve commander-spellbook-backend
+
+# Docker service management
+docker-serve-nemesis:
+	docker-compose -f apps/security/nemesis-service/compose.yaml up -d
+
+docker-serve-helk:
+	pnpm nx docker:start helk-service
+
+docker-serve-misp:
+	docker-compose -f apps/security/misp-service/infrastructure/docker/docker-compose.yml up -d
+
+# Service testing
+test-security:
+	pnpm nx run-many --target=test --projects=nemesis-*,helk-service,misp-service,maltrail-service,ghostwriter-service
+
+test-productivity:
+	pnpm nx run-many --target=test --projects=actual-service,mealie-service,firecrawl-service,goose-service,inspector-service
+
+test-tcg:
+	pnpm nx run-many --target=test --projects=commander-spellbook-*
+
+test-python:
+	@echo "Running Python service tests..."
+	cd apps/core/jane-orchestrator && pytest tests/ -v
+	find apps/ -name "pytest.ini" -o -name "pyproject.toml" -path "*/test*" -execdir python -m pytest \;
+
+# Service linting
+lint-python:
+	@echo "Linting Python services..."
+	find apps/ -name "*.py" -not -path "*/node_modules/*" -not -path "./.venv/*" -exec ruff check {} +
+	find apps/ -name "*.py" -not -path "*/node_modules/*" -not -path "./.venv/*" -exec black --check {} +
+
+lint-typescript:
+	@echo "Linting TypeScript services..."
+	pnpm nx run-many --target=lint --projects=jane-ui,actual-service,firecrawl-service,goose-service,inspector-service
+
+# Documentation generation
+docs-build:
+	@echo "Building documentation..."
+	make -C apps/security/ghostwriter-service/DOCS html
+	find apps/ -name "docs" -type d -exec make -C {} html \; 2>/dev/null || true
+
+# Health checks
+health-check:
+	@echo "Checking service health..."
+	curl -f http://localhost:8000/health || echo "Jane Orchestrator not responding"
+	curl -f http://localhost:3000/api/health || echo "Jane UI not responding"
+	curl -f http://localhost:8080/health || echo "Ghostwriter not responding"
+
+# Service status
+status:
+	@echo "Service Status:"
+	@echo "==============="
+	pnpm nx run-many --target=status --all 2>/dev/null || echo "Status commands not available for all services"
+	docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -E "(jane|nemesis|helk|misp|actual|mealie)" || echo "No containerized services running"
 
 # Kubernetes commands
 k8s-apply-dev:
@@ -107,5 +226,22 @@ terraform-apply:
 
 # Format code
 format:
-	npx prettier --write .
+	pnpm prettier --write .
 	find . -name "*.py" -not -path "./node_modules/*" -not -path "./.venv/*" -exec black {} +
+
+# Development environment setup
+setup-dev:
+	@echo "Setting up development environment..."
+	make install
+	make docker-build
+	@echo "Development environment ready!"
+
+# Clean service-specific artifacts
+clean-services:
+	@echo "Cleaning service artifacts..."
+	find apps/ -name "dist" -type d -exec rm -rf {} + 2>/dev/null || true
+	find apps/ -name "build" -type d -exec rm -rf {} + 2>/dev/null || true
+	find apps/ -name ".next" -type d -exec rm -rf {} + 2>/dev/null || true
+	find apps/ -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
+	find apps/ -name "*.pyc" -delete 2>/dev/null || true
+	find apps/ -name "node_modules/.cache" -type d -exec rm -rf {} + 2>/dev/null || true