gavindiaz/NexQuant
1
0
Fork 0
mirror of https://github.com/NicolasBohn/NexQuant.git synced 2026-08-05 03:07:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add parallel run system with API key distribution

Browse Source 
- Add predix_parallel.py: Run multiple factor experiments concurrently
  * python predix_parallel.py --runs 5 --api-keys 2 -m openrouter
  * Round-robin API key distribution across available keys
  * Rich live dashboard with per-run status, elapsed time, exit codes
  * Graceful shutdown (Ctrl+C kills all children cleanly)

- Add --run-id parameter to predix.py for isolated single runs
  * Separate log files: fin_quant_run{N}.log
  * Separate results: results/runs/run{N}/
  * Separate workspace: RD-Agent_workspace_run{N}/
  * Separate databases per run

- Modify CoSTEER and FactorRunner for PARALLEL_RUN_ID isolation
  * _save_intermediate_results uses run-specific directories
  * _save_result_to_database and _write_run_log isolated per run
  * _ensure_results_dirs creates run-specific paths

- Reduce max_loop from 10 to 3 for faster iterations
- Add docs/parallel_runs.md with full documentation

Tests: 103 passed
This commit is contained in:
TPTBusiness
2026-04-04 09:39:12 +02:00
parent 54073da2b0
commit 8b7eb87546
8 changed files with 1062 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/parallel_runs.md
+238
View File
@@ -0,0 +1,238 @@
# Predix Parallel Run System
## Overview
The Parallel Run System enables concurrent execution of 5+ factor generation experiments with automatic API key distribution and complete isolation between runs.
## Architecture
### Components
| File | Purpose |
|------|---------|
| `predix.py` | Extended with `--run-id` parameter for isolated single runs |
| `predix_parallel.py` | Parallel runner manager with Rich live dashboard |
| `factor_runner.py` | Modified to use `PARALLEL_RUN_ID` for path isolation |
| `CoSTEER/__init__.py` | Modified to use `PARALLEL_RUN_ID` for intermediate results |
### Directory Structure (Per Run)
```
results/
├── db/ # Shared database
├── runs/
│ ├── run1/ # Run #1 isolated results
│ │ ├── factors/ # Factor JSON files
│ │ ├── logs/ # Run-specific logs
│ │ ├── db/ # Run-specific database
│ │ └── costeer/ # CoSTEER intermediate results
│ ├── run2/ # Run #2 isolated results
│ │ └── ...
│ └── runN/ # Run #N isolated results
│ └── ...
└── logs/ # Default (non-parallel) logs
```
### Log Files
```
fin_quant.log # Single run (run_id=0)
fin_quant_run1.log # Parallel run #1
fin_quant_run2.log # Parallel run #2
...
```
### Workspaces
```
RD-Agent_workspace/ # Single run (run_id=0)
RD-Agent_workspace_run1/ # Parallel run #1
RD-Agent_workspace_run2/ # Parallel run #2
...
```
## Usage
### CLI - Single Parallel Run
```bash
# Run with isolated results
predix quant --run-id 1 -m openrouter
```
### CLI - Parallel Runner (Direct)
```bash
# Run 5 experiments with 2 API keys
python predix_parallel.py --runs 5 --api-keys 2
# Run 3 experiments with local model
python predix_parallel.py --runs 3 --model local
# Custom configuration
python predix_parallel.py -n 10 -k 2 -m openrouter
```
### Programmatic Usage
```python
from predix_parallel import main
result = main(runs=5, api_keys=2, model="openrouter")
print(f"Success: {result['success']}/{result['total']}")
```
## API Key Distribution
The system distributes API keys using round-robin assignment:
| Run ID | API Key | Model |
|--------|---------|-------|
| 1 | Key 1 | openrouter |
| 2 | Key 2 | openrouter |
| 3 | Key 1 | openrouter |
| 4 | Key 2 | openrouter |
| 5 | Key 1 | openrouter |
**With 2 API keys:**
- Runs 1, 3, 5 → Key 1
- Runs 2, 4 → Key 2
**LiteLLM Load Balancing:**
When 2 API keys are available, the system configures LiteLLM for parallel request handling:
```
OPENAI_API_KEY=key1,key2
LITELLM_PARALLEL_CALLS=2
```
## Isolation Guarantees
Each parallel run is completely isolated:
### Environment Variables
- `PARALLEL_RUN_ID=N` - Identifies the run
- `RD_AGENT_WORKSPACE` - Points to run-specific workspace
- `OPENAI_API_KEY` - Assigned API key for this run
### No Shared State
- ✅ Separate log files
- ✅ Separate result directories
- ✅ Separate workspace directories
- ✅ Separate database files (optional)
- ✅ No race conditions (no shared mutable state)
### Graceful Degradation
- If a run fails, others continue unaffected
- Each run is independently restartable
- Results are persisted immediately after completion
## Live Dashboard
The parallel runner shows a Rich-based live dashboard:
```
┌─────────────────────────────────────────────────────────┐
│ 🔀 Predix Parallel Run Dashboard │
├──────┬──────────┬──────────┬─────────┬──────────┬───────┤
│ Run │ Status │ Elapsed │ API Key │ Model │ Exit │
├──────┼──────────┼──────────┼─────────┼──────────┼───────┤
│ #1 │ ✅ success│ 02:15:30│ 1 │openrouter│ 0 │
│ #2 │ 🔄 running│ 01:45:12│ 2 │openrouter│ -- │
│ #3 │ 🔄 running│ 01:42:08│ 1 │openrouter│ -- │
│ #4 │ ⏳ pending│ --:--:--│ 2 │openrouter│ -- │
│ #5 │ ❌ failed │ 00:05:23│ 1 │openrouter│ 1 │
├──────┴──────────┴──────────┴─────────┴──────────┴───────┤
│ Summary: 5 total | 1 done | 2 running | 1 pending | 1 failed │
└─────────────────────────────────────────────────────────┘
```
## Signal Handling
- **First Ctrl+C:** Gracefully stops all running subprocesses
- **Second Ctrl+C:** Force kills all remaining processes
- Dashboard updates in real-time during shutdown
## Configuration
### Environment Variables (`.env`)
```bash
# Required for openrouter mode
OPENROUTER_API_KEY=sk-or-your-first-key
OPENROUTER_API_KEY_2=sk-or-your-second-key # Optional
# Required for local mode
OPENAI_API_KEY=local
OPENAI_API_BASE=http://localhost:8081/v1
CHAT_MODEL=qwen3.5-35b
# Optional: Custom model
OPENROUTER_MODEL=openrouter/qwen/qwen3.6-plus:free
```
## Performance
**Expected Speedup:**
- 5 runs with 2 API keys ≈ 2.5× faster than sequential
- 5 runs with local model ≈ 5× faster than sequential (no API rate limits)
**Overhead:**
- ~1 second per run for subprocess startup
- Dashboard refresh: 2 Hz (negligible CPU)
## Error Handling
| Scenario | Behavior |
|----------|----------|
| Run fails | Logged, others continue |
| API key exhausted | Retry with next key |
| Ctrl+C pressed | Graceful shutdown of all runs |
| Disk full | Error logged, run marked failed |
| LLM timeout | Run fails, others unaffected |
## Integration with Existing Code
### factor_runner.py Changes
```python
# Before (shared paths)
log_dir = project_root / "results" / "logs"
factors_dir = project_root / "results" / "factors"
# After (parallel-aware)
parallel_run_id = os.getenv("PARALLEL_RUN_ID", "0")
if parallel_run_id != "0":
log_dir = project_root / "results" / "runs" / f"run{parallel_run_id}" / "logs"
factors_dir = project_root / "results" / "runs" / f"run{parallel_run_id}" / "factors"
```
### CoSTEER/__init__.py Changes
```python
# Intermediate results isolation
parallel_run_id = os.getenv("PARALLEL_RUN_ID", "0")
if parallel_run_id != "0":
results_dir = project_root / "results" / "runs" / f"run{parallel_run_id}" / "costeer"
```
## Testing
```bash
# Run all integration tests
pytest test/integration/test_all_features.py -v
# Test parallel runner imports
python -c "from predix_parallel import ParallelRunner, main; print('✅ OK')"
# Test CLI options
predix quant --help # Should show --run-id option
```
## Future Enhancements
- [ ] Auto-detect optimal number of parallel runs based on API rate limits
- [ ] Result aggregation and comparison across runs
- [ ] Dynamic API key rebalancing (assign more runs to faster key)
- [ ] Support for >2 API keys
- [ ] Run prioritization (run high-priority experiments first)
- [ ] Slack/email notifications on completion