# Accent Audio Sample Extraction Tool

## Overview

`extract_accent_audio_samples.py` is a comprehensive tool for extracting audio samples with specific accents and pure English lyrics from captioned vocal stems datasets. It performs validation, audio conversion, and packaging operations.

## Features

- **Strict English Validation**: Uses langdetect library and character filtering to ensure pure English lyrics
- **Multiple Accent Support**: Extract samples for various accent types (German, British, Scottish, Indian, Japanese, Korean, Slavic, etc.)
- **Audio Processing**: Copy and convert audio files from opus to MP3 format
- **Flexible Output**: Export samples as JSON, audio files, and zip archives
- **Progress Tracking**: Real-time progress updates during extraction

## Installation

```bash
# Install required dependencies
pip install langdetect
```

## Usage Examples

### Basic Extraction (JSON only)
```bash
python extract_accent_audio_samples.py \
    --input-jsonl /home/vibert/data/voice_designer/metas_v6_tr_vocal_stems_captioned_w30.jsonl \
    --output-dir /home/vibert/tmp/accent_samples \
    --accents "german,british" \
    --samples-per-accent 10
```

### Full Pipeline (Extract, Convert, Zip)
```bash
python extract_accent_audio_samples.py \
    --input-jsonl /home/vibert/data/voice_designer/metas_v6_tr_vocal_stems_captioned_w30.jsonl \
    --output-dir /home/vibert/tmp/accent_samples \
    --accents "german,british,scottish,indian,japanese,korean,slavic" \
    --samples-per-accent 15 \
    --convert-to-mp3 \
    --cleanup-opus \
    --create-zip
```

### Limited Scan with MP3 Conversion
```bash
python extract_accent_audio_samples.py \
    --input-jsonl /path/to/captioned_dataset.jsonl \
    --output-dir /tmp/accent_test \
    --accents "british,american" \
    --samples-per-accent 5 \
    --max-scan 50000 \
    --convert-to-mp3 \
    --mp3-bitrate 256k
```

### Individual Zip Files Only
```bash
python extract_accent_audio_samples.py \
    --input-jsonl /path/to/dataset.jsonl \
    --output-dir /tmp/accents \
    --accents "german,french,italian" \
    --convert-to-mp3 \
    --cleanup-opus \
    --zip-individual
```

## Command Line Arguments

| Argument | Description | Default |
|----------|-------------|---------|
| `--input-jsonl` | Path to input JSONL file with captioned vocal stems | Required |
| `--output-dir` | Base output directory for all extracted files | Required |
| `--accents` | Comma-separated list of accent types | "german,british,scottish,indian,japanese,korean,slavic" |
| `--samples-per-accent` | Number of samples to extract per accent | 15 |
| `--max-scan` | Maximum records to scan (None for all) | None |
| `--convert-to-mp3` | Convert opus files to MP3 format | False |
| `--mp3-bitrate` | MP3 bitrate for conversion | "192k" |
| `--cleanup-opus` | Remove opus files after MP3 conversion | False |
| `--create-zip` | Create both individual and combined zip archives | False |
| `--zip-individual` | Create individual zip file for each accent | False |
| `--zip-combined` | Create combined zip file with all accents | False |

## Supported Accents

The tool supports the following accent types:
- `german` - German accent
- `british` - British/UK/English accent
- `scottish` - Scottish/Scots accent
- `indian` - Indian/Hindi accent
- `japanese` - Japanese accent
- `korean` - Korean accent
- `slavic` - Slavic/Russian/Eastern European accent
- `irish` - Irish accent
- `australian` - Australian/Aussie accent
- `french` - French accent
- `italian` - Italian accent
- `spanish` - Spanish/Hispanic accent
- `chinese` - Chinese/Mandarin accent
- `american` - American/US accent

## Output Structure

```
output_dir/
├── german_accent_pure_english/          # JSON files
│   ├── german_001_<id>.json
│   ├── german_002_<id>.json
│   └── ...
├── german_accent_pure_english_audio/    # Audio files
│   ├── <id>.mp3
│   └── ...
├── german_accent_audio.zip              # Individual zip
├── british_accent_pure_english/
├── british_accent_pure_english_audio/
├── british_accent_audio.zip
└── all_accent_audio_samples.zip         # Combined zip
```

## Validation Criteria

The tool uses strict validation to ensure pure English lyrics:

1. **Language Detection**: Uses langdetect library to verify English language
2. **Character Filtering**: Rejects text with non-Latin characters (Cyrillic, CJK, Arabic, Hebrew)
3. **English Word Validation**: Requires minimum common English words for longer texts
4. **Annotation Removal**: Strips [Verse], [Chorus] tags before validation

## Performance Notes

- Processing 870,000 records typically takes 5-10 minutes depending on the number of accents
- MP3 conversion time depends on the number and size of audio files
- Expect ~95% rejection rate for accent samples due to strict English validation

## Example Output

```
============================================================
STEP 1: Extracting accent samples with pure English lyrics
============================================================
Extracting samples for accents: german, british
Target: 15 samples per accent
============================================================
  Found german: k0-x7--1caE (1/15)
  Found british: Rv2OyI0nXEE (1/15)
  Scanned 10,000 records, rejected 9,234 non-English...
    german: 8/15
    british: 11/15
...
============================================================
Extraction complete! Scanned 45,678 records
Rejected 43,234 records with non-English lyrics
  german: 15 samples extracted
  british: 15 samples extracted
```

## Troubleshooting

### langdetect not installed
```bash
pip install langdetect
```

### ffmpeg not found (for MP3 conversion)
```bash
# Ubuntu/Debian
sudo apt-get install ffmpeg

# macOS
brew install ffmpeg
```

### Memory issues with large datasets
Use `--max-scan` to limit the number of records processed:
```bash
--max-scan 100000
```

## Integration with Caption Pipeline

This tool is designed to work with the output of `caption_vocal_stems.py`:

```bash
# Step 1: Caption vocal stems
python caption_vocal_stems.py \
    --input-jsonl metas_v6_tr_vocal_stems.jsonl \
    --output-jsonl metas_v6_tr_vocal_stems_captioned.jsonl \
    --prompts-file vocal_description_prompts.json

# Step 2: Extract accent samples
python extract_accent_audio_samples.py \
    --input-jsonl metas_v6_tr_vocal_stems_captioned.jsonl \
    --output-dir /tmp/accent_samples \
    --convert-to-mp3 \
    --create-zip
```