# Prompt Evaluation Harness

Minimal evaluation harness for comparing GPT-5.1 and GPT-4o outputs on song lyric generation prompts.

## Setup

1. Create a `.env` file with your OpenAI API key:
```
OPENAI_API_KEY=your-api-key-here
```

2. Install dependencies:
```bash
uv sync
```

## Usage

### Run Evaluation

Process the first 10 prompts (writes to the next `outputs_XXX.json` file):
```bash
uv run python eval.py -n 10
```

Run the full suite (all prompts, default concurrency 8):
```bash
uv run python eval.py
```

### Custom model/prompt slots

Each eval run consists of two “slots.” You can point each slot at any model/system prompt pair to run A/B, A/A, or B/B tests:

```bash
# Slot A: gpt-5.1 + 5_1.md, Slot B: gpt-4o + 4o.md (default)
uv run python eval.py --model-a gpt-5.1 --prompt-a 5_1.md --model-b gpt-4o --prompt-b 4o.md

# A/A comparison (both slots share the same model + prompt)
uv run python eval.py --model-a gpt-5.1 --prompt-a 5_1.md --model-b gpt-5.1 --prompt-b 5_1.md

# Swap slot roles or try other prompts
uv run python eval.py --model-a gpt-4o --prompt-a 4o.md --model-b gpt-5.1 --prompt-b 5_1.md
```

The output JSON records which system prompt marker was used per slot, so the viewer can render arbitrary combinations.

#### Judge options

The LLM-as-judge pass now runs inline with the main eval loop. Key flags:

| Flag | Default | Description |
| --- | --- | --- |
| `--no-judge` | disabled | Skip the judge entirely |
| `--judge-model` | `gpt-5.1` | Model used to score each prompt |
| `--judge-sample-rate` | `1.0` | Fraction of prompts to judge (e.g., `0.25` for 25%) |
| `--judge-max-concurrent` | `4` | Max simultaneous judge requests |
| `--judge-temperature` | `0.3` | Judge sampling temperature |

Judging roughly doubles token usage. Expect ~$0.05–$0.10 per prompt at full fidelity with GPT-5.1.

#### Concurrency

Use `--max-concurrent` to control how many generation requests are inflight. Lower it if you hit rate limits.

### Checkpointing

Add `--checkpoint-interval N` to save partial results every N prompts:

```bash
uv run python eval.py -n 50 --checkpoint-interval 10
```

Each checkpoint overwrites the target `outputs_XXX.json` with the progress so far (including the judge summary to date). Set the interval to `0` (default) to disable mid-run writes.

### View Results

1. Start a local server (any static server works):
    ```bash
    uv run python -m http.server
    ```
2. Open the viewer at `http://localhost:8000/viewer.html`.
3. Use the dropdown to pick an `outputs_XXX.json` file. Navigation:
    - Left/Right arrow keys
    - Previous/Next buttons
    - "View: Final/Raw" toggle

New judge UI highlights the winning model, per-model failure icons, 1-sentence commentary, and a dismissible stats modal with a confidence-weighted Elo + W-D-L leaderboard plus failure rates.

## Output Format

Results are saved to versioned JSON files (`outputs_001.json`, `outputs_002.json`, etc.) with the following schema:

```json
{
  "system_prompts": {...},
  "prompts_hash": "...",
  "judge_summary": {
    "judge_model": "gpt-5.1",
    "total_judged": 123,
    "model_stats": {
      "gpt-5.1": {"wins": 80, "losses": 35, "ties": 8}
    },
    "elo_ratings": {
      "gpt-5.1": {"rating": 1523.4, "matches": 123, "avg_confidence": 0.71}
    },
    "failure_rates": {
      "gpt-5.1": {
        "instrumental_mismatch": {"count": 2, "rate": 0.016},
        "format_failure": {"count": 1, "rate": 0.008}
      }
    }
  },
  "results": [
    {
      "prompt": "...",
      "completions": {
        "gpt-5.1": {...},
        "gpt-4o": {...}
      },
      "judge": {
        "model_assignment": {"A": "gpt-5.1", "B": "gpt-4o"},
        "winner": "A",
        "confidence": 0.72,
        "commentary": "Model A nailed the requested German lyrics while Model B mixed in English.",
        "failures": {
          "A": {"language_mismatch": false, ...},
          "B": {"language_mismatch": true, "details": "Title stayed in English"}
        }
      }
    }
  ]
}
```

`elo_ratings` captures the confidence-weighted Elo rating (starting at 1000 for new models), match count, and mean judge confidence per model; the viewer leaderboard is derived from the same data.

## Features

- Async streaming API calls with configurable concurrency
- Inline GPT-5.1 judge with random blinding, structured JSON verdicts, and aggregate stats
- Automatic versioning of output files plus hash markers for reproducibility
- Enhanced viewer with raw/final toggles, judge verdict badges, per-category failure icons, and a stats modal
- Keyboard navigation and permanent scrollbars to avoid layout shifts

## Testing

Run a quick smoke test (first prompt, serialized generation + judge) to ensure API access is configured:

```bash
uv run python eval.py -n 1 --max-concurrent 1 --judge-max-concurrent 1
```

The command emits a new `outputs_XXX.json` that should include `judge` and `judge_summary` blocks if the judge step succeeded.