# Data loading pipeline

1. Create database and initialize schema

```
psql -c 'CREATE DATABASE composer_dataset'
export PGDATABASE=composer_dataset
```

2. Load MIDI dataset into database and apply file name annotations

```
python3 -m venv venv
. ./venv/bin/activate
pip install -r requirements.txt
python collect.py data/*
cd file_names
python main.py
```

3. Start postgREST

```
./postgrest.sh
```

4. Run clip extractor

```
cd WavTool/midi-analysis-v2
yarn
yarn run tag 1000000000
```

5. Annotate extracted clip symbolic lengths

```
cd composer/transformer
python3 -m venv venv
. ./venv/bin/activate
./install_requirements.sh
python update_lengths.py
```

6. Import manual lane and clip tags

```
...
```

7. Train classifier base model with drum pitch separation disabled

```
...
```

8. Train classifier head for drums

```
...
```

9. Train classifier head for Good First Generation

```
...
```

10. Run classifier heads on all extracted clips

```
...
```