# Reserved Node Mode - Usage Guide

## Overview

The `run_chained.py` script now supports **Node Reservation Mode**, which reserves a fixed set of nodes upfront and runs all jobs sequentially on the same nodes. This eliminates queue wait times between jobs and ensures consistent hardware.

## Benefits

✅ **No queue wait between jobs** - nodes stay allocated for the entire chain  
✅ **Consistent hardware** - same nodes used for all training runs  
✅ **Faster overall completion** - no scheduler overhead between jobs  
✅ **Better resource utilization** - nodes don't sit idle between jobs  
✅ **All existing features work** - checkpoint extraction, script updating, resume capability

## Usage

### Basic Reserved Mode

```bash
./run_chained.py --config chain_config.yaml --reserve-nodes 16 --reserve-time 72:00:00
```

This will:
1. Reserve 16 nodes for 72 hours
2. Run all jobs from `chain_config.yaml` sequentially on those nodes
3. Extract checkpoints and update scripts between jobs (as usual)
4. Log everything to a single master log file

### Standard Mode (Original Behavior)

```bash
./run_chained.py --config chain_config.yaml
```

Each job is submitted separately and queued independently.

### Dry Run with Reservation

```bash
./run_chained.py --config chain_config.yaml --reserve-nodes 16 --dry-run
```

Shows what would happen without actually submitting jobs.

### With Resume Support

```bash
# If a job fails mid-chain, you can resume (not yet implemented for reservation mode)
./run_chained.py --config chain_config.yaml --resume
```

## How It Works

1. **Script validates** all job scripts in the chain
2. **Creates a master script** that runs all jobs sequentially using `srun`
3. **Submits a single SLURM job** that reserves the requested nodes
4. **Monitors progress** and waits for completion
5. **Parses the combined log** to extract checkpoint paths for each job
6. **Updates subsequent scripts** with checkpoint paths (for next time)

## Node Requirements

If your jobs require different numbers of nodes:
- Reserve enough nodes for the **largest** job
- Each job will use only the nodes it needs (specified in config)
- Example: If jobs need 8, 12, and 16 nodes, reserve 16 nodes

```yaml
jobs:
  - name: "Job 1"
    script: "run_job1.sh"
    nodes: 8   # Uses 8 of the 16 reserved nodes
  
  - name: "Job 2"
    script: "run_job2.sh"
    nodes: 16  # Uses all 16 reserved nodes
```

## Log Files

- **Standard mode:** One log per job: `/app/suno/slurm/logs/run_dpo_<job_id>.txt`
- **Reserved mode:** Single combined log: `/app/suno/slurm/logs/reserved_chain_<job_id>.txt`

## Comparison

| Feature | Standard Mode | Reserved Mode |
|---------|--------------|---------------|
| Queue wait between jobs | Yes | No |
| Node consistency | No guarantee | Same nodes |
| Total time | Longer (queue waits) | Faster |
| Individual job logs | Yes | Combined log |
| Resume mid-chain | Yes | Not yet supported |
| Resource efficiency | Lower | Higher |

## Example Workflows

### Long training chain (recommended for reservation)
```bash
# Reserve 16 nodes for 3 days to run 10 training jobs
./run_chained.py \
  --config chain_config.yaml \
  --reserve-nodes 16 \
  --reserve-time 72:00:00
```

### Quick experimentation (standard mode is fine)
```bash
# Run 2-3 jobs, willing to wait in queue
./run_chained.py --config quick_test.yaml
```

### Override parameters with reservation
```bash
# Reserve nodes and override SFT loss scale
./run_chained.py \
  --config chain_config.yaml \
  --reserve-nodes 16 \
  --reserve-time 48:00:00 \
  --sft_loss_scale 0.5
```

## Troubleshooting

### Job fails mid-chain
Check the combined log file: `/app/suno/slurm/logs/reserved_chain_<job_id>.txt`

The log will show which job failed and where in the sequence.

### Not enough nodes reserved
If a job requires more nodes than reserved, it will use only the reserved nodes:
```
WARNING: Job requires 24 nodes but only 16 are reserved. Using 16 nodes.
```

### Time limit too short
If jobs don't complete in the reserved time, the entire chain will be killed:
- Estimate total runtime for all jobs
- Add buffer time (20-30%)
- Set `--reserve-time` accordingly

## Advanced: YAML Configuration

You can specify different parameters per job:

```yaml
chain_name: "DPO Training Chain with Reserved Nodes"
description: "Training progression with node reservation"

settings:
  base_dir: "/home/tony/Work/tony/slurm/crow"
  log_dir: "/app/suno/slurm/logs"

jobs:
  - name: "Round 1"
    script: "run_ipo_crow_r1.sh"
    nodes: 8
    time: "24:00:00"
    model_cache_loss_name: "crow_r1"
    wandb_run_name: "crow_r1_training"
  
  - name: "Round 2"
    script: "run_ipo_crow_r2.sh"
    nodes: 16
    time: "36:00:00"
    model_cache_loss_name: "crow_r2"
    wandb_run_name: "crow_r2_training"
```

Then run with:
```bash
./run_chained.py --config chain_config.yaml --reserve-nodes 16 --reserve-time 72:00:00
```

## Tips

1. **Use reservation mode** for chains with 3+ jobs that you want to complete quickly
2. **Use standard mode** for quick experiments or when nodes are scarce
3. **Always dry-run first** to verify your configuration
4. **Monitor the log file** during execution to track progress
5. **Reserve 20-30% more time** than you think you need