# Data Diff Analysis Tools

Tools for comparing Suno data versions (v0-v9, excluding v7).

## Quick Start

```bash
cd /home/vibert/projects/neon/sunoData/src/sunodata/data_diff
python=/home/vibert/anaconda3/envs/suno_3/bin/python

# Profile-based comparison (ID diffs, stats, plots)
$python run_all_diffs.py --single-pair v0 v1 --sample-size 20

# Line-by-line comparison (field-level diffs)
$python compare_metas.py file1.jsonl file2.jsonl --workers 32 --output diff.json
```

## Tools

| Tool | Purpose | Speed | Output |
|------|---------|-------|--------|
| `run_all_diffs.py` | ID-level diffs, stats, plots | 60-90 min (8 pairs) | JSON, plots, samples |
| `compare_metas.py` | Field-level diffs | 3-6 min (39GB) | JSON with exact differences |

## Usage

### Profile-Based Analysis

```bash
# All sequential pairs
python run_all_diffs.py

# Custom options
python run_all_diffs.py \
  --versions v0 v1 v2 v3 \
  --sample-size 20 \
  --output-dir /path/to/output
```

**Output:** `/home/vibert/data/data_diff/diff_v{X}_to_v{Y}/`
- `summary.json` - Stats (record counts, ID changes)
- `{added,removed}_ids.txt` - ID lists
- `{added,removed}_samples.jsonl` - Sample records
- `tag_changes.json` - Tag distribution shifts
- `stem_changes.json` - Stem/caption changes
- `plots/*.png` - Visualizations

### Line-by-Line Comparison

```bash
# Basic usage
python compare_metas.py file1.jsonl file2.jsonl

# With options
python compare_metas.py file1.jsonl file2.jsonl \
  --chunk-size 20000 \
  --workers 32 \
  --max-diffs 100 \
  --output detailed_diff.json
```

**Performance tuning:**
- Large files (>10GB): `--chunk-size 20000 --workers 32`
- Memory constrained: `--chunk-size 5000 --workers 8`
- Fast SSD: `--chunk-size 50000 --workers 64`

**Output JSON structure:**
```json
{
  "total_differences": 134494,
  "differences": [{
    "line_number": 144,
    "diff": {
      "id1": "abc", "id2": "abc",
      "only_in_file2": ["hook_offset_s"],
      "differing_values": [{"key": "weight", "value1": 1.0, "value2": 2.0}]
    }
  }]
}
```

## Known Version Changes

| Transition | Change | Records Affected |
|------------|--------|------------------|
| v0 → v1 | Added `hook_offset_s` field | 134,494 (0.24%) |
| v0-v3 | Identical IDs | 56,617,603 records |
| v3 → v4 | Added records | +38,957 |
| v4 → v5 | Added `stems_captions` (extreme) | +2 |
| v8 → v9 | Added `stems_captions` (vocals), `vocal_pitch_range` | TBD |

## Module API

```python
from sunodata.data_diff.load_profiles import load_version_data
from sunodata.data_diff.compare_versions import compare_versions
from sunodata.data_diff.extract_samples import extract_samples_from_jsonl

# Load version data
analysis, id_set, run_dir = load_version_data(profile_dir, "v9")

# Compare versions
comparison = compare_versions(old_analysis, new_analysis, old_ids, new_ids, "v0", "v1")

# Extract samples
samples = extract_samples_from_jsonl(jsonl_path, target_ids, sample_size=20)
```

## Dependencies

Python 3.10+, tqdm, matplotlib, numpy

---

**Output directory:** `/home/vibert/data/data_diff/`
**Profile directory:** `/home/vibert/data/suno_data_monitor/outputs/`