## Artist and Cover Feature Eval

Uses ditto_v2 `self_sim` to detect if cover is a regurgitation of the source. Uses ditto_v2 `artist_vox_sim` on stem separated and trimmed vocals to detect if the singer of a persona is maintained

### Get Training Data Stats

1. Get similarity of source/cover pairs in the training data: 
`python filter_cover_ditto.py`

2. Get similarity of artist pairs in the training data: 
`python filter_artist_ditto.py`

Scores are written to npz files in the same folder, to be later used by `eval_cover_artist.ipynb`

### Run Benchmarks of a Given Model on Modal

1. You'll first need to launch a modal worker for the model you want to test, the same as if you were running an A/B test experiment. Add an entry for the model in` MODEL_DEV_FNS`. This step can be skipped if you're testing on a model that is already running in modal.
2. Configure `model_runner_eval.py` parameters:
    * `MODEL_NAME`: Should match the name you added to `MODEL_DEV_FNS`
    * `EVAL_NAME`: which eval suite to run, currently "hard" or "all_genres"
    * `INFER_CONFIG`: any custom GenerationConfig parameters you want to test
    * `EXTRA_TAGS`: additional experiment labels, like config changes
3. Run `modal run model_runner_eval.py` to launch the benchmark. 
* Generation outputs and ditto scores are written to s3: `s3://suno-data-uploads/tasks/feature_eval/cover_persona/TIMESTAMP/`
* The s3 ids of cover and artist mappings are stored locally at `suno_utils/task_eval/modal_runs/TIMESTAMP.json`

### Analyze Benchmark Results

1. Set the `TIMESTAMP` in `score_tasks.py` based on the previous section
2. Run `python score_tasks.py`, which writes results to an npz file `suno_utils/task_eval/scores` 
3. Add the resulting npz paths to `COVER_FILES` and `ARTIST_FILES` in `eval_cover_artist.ipynb`
4. Run `eval_cover_artist.ipynb` to update cover and persona performance plots