Getting Started with AlephAuto
AlephAuto is a job queue framework for automation pipelines. This dashboard provides real-time monitoring of:
- Pipeline Status: Track all active pipelines and their health
- Job Queue: Monitor active and queued jobs with capacity tracking
- Real-time Updates: WebSocket connection provides live status updates
- Activity Feed: Recent events, completions, and errors
Quick Actions
- View pipeline details by clicking on any pipeline card
- Check job queue capacity and active jobs
- Monitor recent activity in the activity feed
- Switch documentation tabs for detailed information
Available Pipelines
🔍 Duplicate Detection Pipeline
Purpose: Scans repositories for duplicate code using AST-grep and similarity algorithms
Schedule: Daily at 2 AM (configurable via CRON_SCHEDULE)
Key Features:
- 7-stage processing pipeline (JS → Python)
- Multi-layer similarity detection
- Redis caching with Git commit-based invalidation
- HTML/JSON/Markdown report generation
📄 Schema Enhancement Pipeline
Purpose: Injects Schema.org structured data into README files
Schedule: Daily at 3 AM (DOC_CRON_SCHEDULE)
Key Features:
- Automatic Schema.org JSON-LD generation
- SEO impact scoring
- Git workflow with auto-PR creation
📦 Repomix Generator Pipeline
Purpose: Generates repomix output files for repositories
Schedule: Daily at 2 AM (CRON_SCHEDULE)
Key Features:
- Automated repomix file generation
- Configurable timeout and buffer size
- Ignore pattern support
📊 Git Activity Pipeline
Purpose: Generates weekly/monthly git activity reports
Schedule: Sunday at 8 PM (GIT_CRON_SCHEDULE)
Key Features:
- Commit statistics and visualizations
- Author contribution tracking
- Markdown report generation
🔌 Plugin Management Pipeline
Purpose: Audits Claude Code plugins for duplicates and issues
Schedule: Monday at 9 AM (PLUGIN_CRON_SCHEDULE)
Key Features:
- Duplicate plugin detection
- Version conflict identification
- Cleanup recommendations
📝 Gitignore Management Pipeline
Purpose: Batch updates .gitignore files across repositories with repomix-output.xml entry
Schedule: Daily at 4 AM (GITIGNORE_CRON_SCHEDULE)
Key Features:
- Automatic .gitignore file creation and updates
- Dry-run mode for preview
- Repository scanning with exclusion rules
- Integration with AlephAuto job queue
🏥 Claude Health Pipeline
Purpose: Monitors Claude Code environment health and configuration status
Schedule: Daily at 6 AM (HEALTH_CRON_SCHEDULE)
Key Features:
- Environment variable validation
- Dependency health checks
- Configuration integrity verification
- Health report generation
🧪 Test Refactor Pipeline
Purpose: Scans test suites and generates modular utility files
Schedule: Sunday at 4 AM (TEST_REFACTOR_CRON)
Key Features:
- Test pattern detection (renderWaitFor, form interactions)
- Hardcoded string extraction
- Duplicate assertion identification
- Utility file generation with recommendations
🐛 Bugfix Audit Pipeline
Purpose: Audits repositories for bugfix opportunities and technical debt
Schedule: On-demand (manual trigger)
Key Features:
- Sentry error aggregation
- Prioritized bugfix recommendations
- Technical debt analysis
- Integration with issue tracking
API Reference
All endpoints return JSON. Authentication is required for most endpoints via X-API-Key header.
System Status
GET
/health
Health check endpoint - returns server status
🔓 No auth required
GET
/api/status
System status with ALL pipelines from database, job queue stats, and activity feed
🔓 No auth required
Response Example
{
"timestamp": "2025-11-24T18:12:45.749Z",
"pipelines": [
{
"id": "duplicate-detection",
"name": "Duplicate Detection",
"status": "idle",
"completedJobs": 10,
"failedJobs": 2,
"lastRun": "2025-11-24T02:00:00.000Z",
"nextRun": null
},
// ... all 8 pipelines from database
],
"queue": {
"active": 0,
"queued": 0,
"capacity": 0
},
"recentActivity": [ /* 20 recent events */ ]
}
✨ Updated: Now returns all pipelines from database (8+) instead of just duplicate-detection
GET
/ws/status
WebSocket connection status and client count
🔓 No auth required
Pipeline Management
GET
/api/sidequest/pipeline-runners/:pipelineId/jobs
Fetch job history for a specific pipeline with pagination
🔓 No auth required
Query Parameters
status (optional): Filter by status (queued, running, completed, failed)limit (optional): Max jobs per page (default: 10, max: 100)offset (optional): Pagination offset (default: 0)tab (optional): UI tab context (recent, failed, all)
Response Example
{
"pipelineId": "repomix",
"jobs": [ /* array of job objects */ ],
"total": 201,
"hasMore": true,
"timestamp": "2025-11-24T18:12:45.749Z"
}
✨ Updated: total now shows actual database count (e.g., 201) instead of page size (e.g., 50)
POST
/api/sidequest/pipeline-runners/:pipelineId/trigger
Manually trigger a pipeline job
🔓 No auth required
Request Body
{
"parameters": {
"repositoryPath": "/path/to/repo"
// or
"repositoryPaths": ["/path/1", "/path/2"]
}
}
Scan Operations
POST
/api/scans
Trigger a new repository scan (duplicate detection)
🔓 No auth required
Request Body
{
"scanType": "intra-project",
"repositoryPath": "/path/to/repo"
}
// or
{
"scanType": "inter-project",
"repositoryPaths": ["/path/1", "/path/2"]
}
GET
/api/scans/:scanId
Get scan results by ID
🔓 No auth required
Reports
GET
/api/reports
List available scan reports with filtering and pagination
🔓 No auth required
Query Parameters
limit (optional): Max reports to return (default: 20)format (optional): Filter by format (html, markdown, json)type (optional): Filter by type (summary, full)
GET
/api/reports/:filename
Get a specific report file (HTML, JSON, or Markdown)
🔓 No auth required
🔒 Directory traversal protection enabled (rejects "../" in filenames)
DELETE
/api/reports/:filename
Delete a report file
🔓 No auth required
WebSocket Events
Connect to ws://localhost:8080 for real-time updates
job:created - New job added to queue with job detailsjob:started - Job execution started with timestampjob:completed - Job finished successfully with resultsjob:failed - Job execution failed with error detailspipeline:status - Pipeline status update (idle/running)activity - Activity feed event (scan complete, error, etc.)
Available Pipeline IDs
duplicate-detection - Duplicate code detectionrepomix - Repomix file generationgit-activity - Git activity reportsclaude-health - Claude Code health monitoringgitignore-manager - Gitignore batch updatesplugin-manager - Plugin audit and cleanupschema-enhancement - Schema.org structured data injectiontest-refactor - Test suite refactoring analysisbugfix-audit - Bugfix opportunity identificationrepo-cleanup - Repository cleanup automation
System Architecture
AlephAuto Job Queue Framework
┌─────────────────────────────────────┐
│ SidequestServer (Base) │
│ - Job queue management │
│ - Concurrency control │
│ - Event emission │
│ - Sentry integration │
│ - Retry logic & circuit breaker │
└─────────────────────────────────────┘
▲ extends
┌─────────┴──────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┐
│ │ │ │ │ │ │
DuplicateDetection Repomix SchemaEnhancement GitActivity PluginManager Gitignore TestRefactor
Worker Worker Worker Worker Worker Worker Worker
Duplicate Detection Pipeline (7 Stages)
Stage 1-2 (JavaScript): Repository Scanner → AST-Grep Detector
↓ JSON via stdin/stdout
Stage 3-7 (Python): Block Extraction → Semantic Annotation
→ Duplicate Grouping → Suggestions → Reports
Key Components
- API Server: Express REST API with WebSocket support
- Job Queue: AlephAuto framework with concurrency control
- Redis Cache: Git commit-based result caching (30-day TTL)
- Sentry: Error tracking and performance monitoring
- Event Broadcasting: WebSocket real-time updates
Pipeline Data Flow Documentation
Comprehensive data flow diagrams and architectural documentation for all 9 AlephAuto pipelines.
Loading pipeline data flow documentation...