# Studio Django API

## Prerequisites

The following should be installed on your machine. If they are not, please run the following commands in your terminal.

### 1. Homebrew:

```shell
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

If met with error "zsh: command not found: brew", try:

```shell
echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> /Users/$USER/.zprofile
```

and then:

```shell
eval $(/opt/homebrew/bin/brew shellenv)
```

### 2. Git Credential Manager:

```shell
brew install git
```

and then:

```shell
brew install --cask git-credential-manager
```

### 3. Python 3.10:

```shell
brew install python@3.10
```

and then ensure default python version is 3.10 by running:

```shell
alias python=/opt/homebrew/bin/python3.10
```

### 4. uv:

```
brew install uv
```

### 5. NPM:

```shell
brew install npm
```

### 6. NVM:

```shell
brew install nvm
```

and then:

```shell
source $(brew --prefix nvm)/nvm.sh
```

### 7. yarn:

```
brew install yarn
```

### 8. Docker:

Install [here](https://docs.docker.com/desktop/install/mac-install/), then ensure docker CLI is installed by running `docker --help`. If you get an error, try `brew install docker` to install the CLI.

### 9. Your IDE of choice:

- [Cursor](https://www.cursor.com/) -- mostly everyone uses Cursor these days, we have a company license you can get added to
- [VSCode](https://code.visualstudio.com/download)
- [PyCharm](https://www.jetbrains.com/pycharm/download/?section=mac)

## Bootstrapping Steps for Backend Service

### Step1 Install Postgres

#### Option 1: Use `brew`:

```
brew install postgres
brew install postgresql@14 # on mac
```

Start the service by:

```shell
brew services start postgresql@14
```

#### Option 2: Use macOS [Postgres.app](https://postgresapp.com/)

Start application and start postgres service.

### Step2 Setup Local Database

Once database is setup, run `psql -d template1` and the following commands to create a dev database:

```sql
CREATE DATABASE suno_studio;
CREATE USER suno WITH LOGIN PASSWORD 'suno';
GRANT ALL PRIVILEGES ON DATABASE suno_studio TO suno;
ALTER USER suno CREATEDB;

GRANT USAGE ON SCHEMA public TO suno;
GRANT CREATE ON SCHEMA public TO suno;
```

Once done, Exit with `ctrl-D`

For mac users, a good database GUI is [tableplus](https://tableplus.com/).

This should start database locally at `localhost:5432`. unable to setup the db will cause errors related to port 5432 connection failures.

### Step3 Install redis

```shell
brew install redis
brew services start redis
```

This should start redis at `localhost:6379` (i.e. the default port). You can check your instance with: `redis-cli INCR mycounter` which should output `(integer) 1` if all is well.

Optionally, [RedisInsight](https://github.com/RedisInsight/RedisInsight) is a GUI for viewing local redis state.

### Step4 Setup Env Variables

You will need to get an env file from another team member who has already onboarded.
Place your env file in `.env` in this directory (will not be checked in). May need to `cp .env.test .env` and run with `source .env`.

### Step5 Set up suno_utils

Navigate to the `suno_utils` directory and run the following to setup a virtualenv and sync deps initially:

```shell
uv sync
```

### Step6 Setup Modal

In the `suno_utils` directory and perform the below steps:

1. For the API to work properly with the music creation, you will need to be added to the team's Modal workspace (Ask for an invite from your onboard buddy)
2. Create an auth token. Once you accept the Modal invite, run `uv run modal token new` to create a new token. Make sure you select user `suno-ai` under `Select workspace:` on the `Create token` page. then follow the browser prompts to generate the token. You can verify that you have the corret Modal setup with `less ~/.modal.toml` where you get `token_id` and `token_secret` under `[suno-ai]`

### Step7 Setup Tailscale(nGrok)

[Tailscale Funnel](https://tailscale.com/kb/1223/funnel/) is a useful tool for exposing local APIs to the web. Follow the instructions on its page to set up.

BEFORE running this, ask an admin to configure your machine name. Then update the `CALLBACK_HOST` variable in your .env file to reference your unique machine name.

```shell
CALLBACK_HOST="<your_machine_name>.han-mahi.ts.net"
```

If you installed Tailscale via the Mac OS app store, you might need to add an alias to your `.zshrc` to get the `tailscale` command working.

```shell
alias tailscale="/Applications/Tailscale.app/Contents/MacOS/Tailscale"
```

⚠️ Check your Tailscale app settings and make sure `Allow Incoming Connections` is checked.

Then enable Tailscale Funnel so your machine can be accessed by Modal workers.

Next, restart your machine. If you don't restart your machine, you may see connection errors from Modal the first time you run the application locally. Once you've restarted, run the following command to route traffic from your Tailscale host to your local machine.

```shell
tailscale funnel 8000
```

_Alternative forwarding method_
if your tailscale does not work(check with [admin page](https://login.tailscale.com/admin), if you do not have access permission, then you need to use ngrok). You can use the ngrok:

1. install with `brew install ngrok/ngrok/ngrok`.
2. Run with `ngrok http 8000` (where 8000 is the backend port). Then you will find the Forwarding https address(https://<uuid>.ngrok-free.app -> http://localhost:8000).
3. Copy the https address `<uuid>.ngrok-free.app` to the `CALLBACK_HOST` in your .env file, example `CALLBACK_HOST="b829-2603-3005-b0b-ec0-e0be-18dc-b656-1e48.ngrok-free.app"`

### Step8 Install Elastic Search(Optional)

Install ES by

```shell
docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.15.2
```

Running ES in Docker:

```shell
docker run --name elasticsearch --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" -t docker.elastic.co/elasticsearch/elasticsearch:8.15.2
```

This will spin up a local ES server, which enables search locally (it will not contain any data yet).
You can check your instance with: `curl -X GET "http://localhost:9200/" -H 'Content-Type: application/json'` to check the status of ES.

Elasticsearch primarily uses two ports: 9200 and 9300. Port 9200 is the default HTTP port. Port 9300 is the default transport port, used for node-to-node communication within the cluster.

To ingest the index locally, you can run `uv run manage.py ingest_clips_for_search --rebuild_index` to feed your index to your local ES.

### Step9 Setup DynamoDB Local (Optional)

DynamoDB is used for storing item information with the `item_info_handler`. The handler validates item types and manages fields within each type.

**Prerequisites:**

- Ensure you have an AWS profile configured for staging. If you don't have `--staging` as an AWS profile, you'll need to set it up.
- Set your AWS credentials to point to staging.

Install and run DynamoDB Local:

```shell
docker pull amazon/dynamodb-local
docker run -d -p 8123:8000 amazon/dynamodb-local
```

Verify the local DynamoDB instance is running (should return empty table list initially):

```shell
aws dynamodb list-tables --endpoint-url http://localhost:8123
```

You should see:

```json
{
  "TableNames": []
}
```

Create the `item-info` table locally:

```shell
aws dynamodb create-table \
  --table-name item-info \
  --attribute-definitions \
      AttributeName=itemId,AttributeType=S \
      AttributeName=type,AttributeType=S \
  --key-schema \
      AttributeName=itemId,KeyType=HASH \
      AttributeName=type,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 \
  --endpoint-url http://localhost:8123 \
  --region us-east-2
```

Verify the table was created:

```shell
aws dynamodb list-tables --endpoint-url http://localhost:8123
```

You should now see:

```json
{
  "TableNames": ["item-info"]
}
```

#### DynamoDB Admin Interface (Optional)

For a GUI to manage your local DynamoDB data:

```shell
docker run -d \
  -p 8001:8001 \
  --name dynamodb-admin \
  -e DYNAMO_ENDPOINT=http://host.docker.internal:8123 \
  -e AWS_REGION=us-east-2 \
  -e AWS_ACCESS_KEY_ID=dummy \
  -e AWS_SECRET_ACCESS_KEY=dummy \
  aaronshaf/dynamodb-admin
```

Access the admin interface at `http://localhost:8001`.

**Note about item_info_handler:**
The `item_info_handler` manages item data with validation on the `type` field and associated fields within each type. The composite key structure (itemId + type) allows for flexible item categorization and efficient queries.

### Step10 Start Server with uv

Run the following commands in `glockenspiel/studio_api/`.

`uv` should look for a `pyproject.toml` and `uv.lock`, and from there bootstrap the right version of python, a corresponding virtualenv in `.venv`, and install relevant dependencies.

Upon running a `uv` command you should initially see:

```
$ uv run manage.py runserver

Using Python 3.10.15
Creating virtual environment at: .venv
Installed 268 packages in 1.16s

(normal python program output from here on out)
Environment is not staging or prod. checking database url.
```

#### 1. Run Django migrations

This is **required** when you first start the project or there is any schema update in the project:

```shell
uv run python manage.py migrate
```

If you run into permission issues getting the migrations to run, you may need to:

```
ALTER DATABASE suno_studio OWNER TO suno;
```

Likewise, if you run into an issue with lib-heif file missing, for the Pillow-Heif library (required for HEIC image compatibility for image-to-song), you can install it with:

```
brew install x265 libjpeg libde265 libheif
```

#### 5. Seed gen model data in local db:
```shell
uv run manage.py seed_model_data
```

#### 6. Start the app server:

```shell
uv run python manage.py runserver
```

#### 7. Run a celery task locally (_Optional_):

To run celery job through command line you need to first start the celery worker, open a terminal and navigate to your studio_api directory and run

```shell
uv run celery -A studio_api worker -l info
```

This will display a list of available tasks you can run. Open a separate terminal and run the task you want with

```shell
uv run celery -A studio_api call <task_name>
```

In general, you probably just want the celery beat to schedule tasks for the worker and generally make the app work as expected, which you can start using

```shell
uv run celery --app studio_api beat --loglevel=debug
```

After these steps, you should have the backend service ready. To verify, you can try to create a clip from frontend and see whether you get the updates of the music generation.

## Package Updates:

When a package is too old, you can bump package versions in pyproject.toml and regenerate uv.lock.

## Other Helpful Tools

## Typechecking

We use [Pyright](https://github.com/microsoft/pyright) for static type checking.

To run typechecking across the project, run `uv run pyright`.

We have some project specific type stubs in the `typings` folder to add annotations to some base Django models.

### Stripe CLI

The [Stripe CLI](https://stripe.com/docs/stripe-cli) is helpful for billing development.

```shell
brew install stripe/stripe-cli/stripe
stripe login
```

To run the Stripe event forwarder for local development:

```shell
stripe listen --forward-to localhost:8000/api/billing/webhook/
```

### OpenAPI

Local servers expose a Swagger/OpenAI docs browser at http://localhost:8000/api/docs

## Testing

```shell
uv run pytest
```

If a test runs for longer than (say) 1s, you can mark it as slow with the `@pytest.mark.slow` decorator, and temporarily skip slow tests during local development with:

```shell
uv run pytest -m "not slow"
```

You can check the durations of the ten slowest tests with:

```shell
uv run pytest --durations=10
```

You can generate an html coverage report with the following command. Navigate to `index.html` within the `htmlcov` folder generated by this command.

```shell
uv run pytest --cov --cov-report=html
```

## Updating Django Models

After updating a django model, in order to update the database schema run.

```shell
uv run manage.py makemigrations
uv run manage.py migrate
```

Running `makemigrations` will also update `max_migration.txt` to point to your current latest migration via [django_linear_migrations](https://adamj.eu/tech/2020/12/10/introducing-django-linear-migrations/).
If you run into a conflict in this file in GitHub, it means your migration got assigned the same number as another person's merged migration, and thus you have a merge conflict.
To resolve it, run `uv run rebase_migration bots` (replace `bots` with another namespace if you are working on a different subset of tables) or delete your migration and recreate it via `uv run manage.py makemigrations`.

You should ultimately see that you get a migration with a unique migration number, and that `max_migration.txt` is updated in your PR to point to that latest unique migration number with no merge conflicts.

## Adding dependencies

To add a dependency, run `uv add <dependency_name>==version` which will add the dependency to `pyproject.toml` and regenerate `uv.lock`. Typically we want to pin a specific version of a dependency where possible to avoid accidental breaking changes.

If you need to regenerate the lockfile, run `uv lock`. Be careful to check for significant dependency version upgrades here.

### Authentication

We use [Clerk](https://clerk.com) for social auth from the Web.

### Logging and Monitoring

Logs are piped from Render to BetterStack. Error handling is handled by Sentry.

## Product Details

### Billing

Plans and prices are stored in the database (with pointers to Stripe) so their metadata may be modified in the Django admin.

Monthly and annual plans will be supported, as well as swiching between plans. When switching between plans, a user can upgrade or downgrade at the end of a billing period, or upgrade immediately. (Investigate if proration matters here.)

### External API (WIP)

`POST /v1/gen` - Start a generation. Rate limit: 120 per minute.

`GET /v1/gen/{id}` - Get generation detail (including status). A generation could have multiple results depending on batch size.

`GET /v1/gen` - List generations. Supports cursor (?) pagination

Rate limits. Global rate limit of 20 per second per token.

## Jupyter Lab

To run a local Jupyter Lab, which is useful for debugging and running Django commands, run

```shell
uv run jupyter lab build # if you haven't built before
uv run jupyter lab
```

You will want to put the following codeblock at the top of every Notebook to initialize Django.
Make sure to replace the `PROJECTPATH`.

```python
import os, sys
import django
PROJECTPATH = 'REPLACE WITH PATH TO studio_api'
sys.path.insert(0, PROJECTPATH)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "studio_api.settings")
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"  # https://docs.djangoproject.com/en/4.1/topics/async/#async-safety

os.chdir(PROJECTPATH)
django.setup()
```

#### Schema Generation
The following will generate a full schema dump for the API as well as generating TypeScript types consumed by our web app:

```shell
cd $GLOCKENSPIEL_ROOT/studio_api && uv run manage.py schema_codegen
```

See options for finer-grained control by running:

```shell
cd $GLOCKENSPIEL_ROOT/studio_api && uv run manage.py schema_codegen --help
```


We run a Github Actions workflow `generate-schema` on PRs to ensure the schema checked in to main is up-to-date. Please make sure to run schema generation and commit the result if you've made changes to any Ninja schema.
If you're about to merge a PR with a breaking change, please make sure the affected types are not consumed by mobile clients.

#### Bootstrapping Data

You can copy over production showcase, trending, and metaplaylists over to your local environment by running:

```shell
uv run manage.py copy_production_playlists
```

### Creating a Superuser

To create a superuser, run:

```shell
uv run manage.py createsuperuser
```

You will be prompted for a username, email address, and password. Your email address should be your Suno email.

Navigate and sign in to the Django admin interface at `localhost:8000/margu`.

Verify this user is a superuser by selecting the user and checking the "Superuser status" box under "Permissions" in the Django admin interface.

## Instrumentation

### Custom metrics

To add a Datadog Custom metric:

```python
from studio_api.bots.statsd import get_statsd_client

get_statsd_client().increment(
    "suno.api.generations.count",
    tags=[
        f"model:{clip.model_name}",
    ],
)
```

the `env` and `service` tags will be auto-populated by `ddtrace` in staging and prod.

### Formatting

Before create a PR, run `uv run ruff format . --config ./pyproject.toml` to format your changes under this dir. Run `uv run ruff check --config ./pyproject.toml --fix .` to fix potential python err.

### Staging Env

Staging env app address is https://b.suno.fm/

### Profiling query performance from the shell

On dev, we have `django-extensions` installed, which provides a shell_plus command ([docs](https://django-extensions.readthedocs.io/en/latest/shell_plus.html)). This is an interactive that imports all the models and allows us to run code / queries without using a server. This will run against the db that is set in your `.env` file. To exit, you can run `exit()` or `quit()` or `Ctrl+D`.

To run this, you can use the following command:

```shell
uv run python manage.py shell_plus
```

there are lots of python shells / options for this. one usefull one is if you want to print the sql queries for the django querysets, you can run:

```shell
uv run python manage.py shell_plus --print-sql
```

Then, in the shell you can use `.explain()` on any query to see the query plan. Here are the [docs](https://django-extensions.readthedocs.io/en/latest/shell_plus.html#sql-explain), but one useful flags to pass in is:

```python
.explain(analyze=True) # ACTUALLY RUNS THE QUERY to see the run the execution times
```

**Note: this will run the query, so be careful with it!!**

#### Examples

With `--print-sql` turned on:

```python
In [3]: print(UserReaction.objects.all().values_list("clip__clip_personas__persona__user__discord_info")[:10].explain())
EXPLAIN SELECT "bots_discordinfo"."id"
  FROM "bots_userreaction"
  LEFT OUTER JOIN "bots_generatedclip"
    ON ("bots_userreaction"."clip_id" = "bots_generatedclip"."id")
  LEFT OUTER JOIN "bots_personaclip"
    ON ("bots_generatedclip"."id" = "bots_personaclip"."clip_id")
  LEFT OUTER JOIN "bots_persona"
    ON ("bots_personaclip"."persona_id" = "bots_persona"."id")
  LEFT OUTER JOIN "auth_user"
    ON ("bots_persona"."user_id" = "auth_user"."id")
  LEFT OUTER JOIN "bots_discordinfo"
    ON ("auth_user"."id" = "bots_discordinfo"."user_id")
 LIMIT 10

Execution time: 0.175667s [Database: default]
Limit  (cost=2.73..222.74 rows=10 width=8)
  ->  Nested Loop Left Join  (cost=2.73..31582996409.57 rows=1435523442 width=8)
        ->  Nested Loop Left Join  (cost=2.16..21025783830.00 rows=1435523442 width=4)
              ->  Nested Loop Left Join  (cost=1.58..20989573305.15 rows=1435523442 width=4)
                    ->  Nested Loop Left Join  (cost=1.15..15983518924.45 rows=1435523442 width=16)
                          ->  Nested Loop Left Join  (cost=0.59..5418304471.62 rows=1435523442 width=16)
                                ->  Seq Scan on bots_userreaction  (cost=0.00..31025191.42 rows=1435523442 width=16)
                                ->  Memoize  (cost=0.59..4.12 rows=1 width=16)
                                      Cache Key: bots_userreaction.clip_id
                                      Cache Mode: logical
                                      ->  Index Only Scan using bots_usagelog_pkey on bots_generatedclip  (cost=0.58..4.11 rows=1 width=16)
                                            Index Cond: (id = bots_userreaction.clip_id)
                          ->  Index Scan using bots_personaclip_clip_id_590f8c66 on bots_personaclip  (cost=0.56..7.35 rows=1 width=32)
                                Index Cond: (clip_id = bots_generatedclip.id)
                    ->  Memoize  (cost=0.43..7.22 rows=1 width=20)
                          Cache Key: bots_personaclip.persona_id
                          Cache Mode: logical
                          ->  Index Scan using bots_persona_pkey on bots_persona  (cost=0.42..7.21 rows=1 width=20)
                                Index Cond: (id = bots_personaclip.persona_id)
              ->  Memoize  (cost=0.57..4.03 rows=1 width=4)
                    Cache Key: bots_persona.user_id
                    Cache Mode: logical
                    ->  Index Only Scan using auth_user_pkey on auth_user  (cost=0.56..4.02 rows=1 width=4)
                          Index Cond: (id = bots_persona.user_id)
        ->  Index Scan using bots_discordinfo_user_id_key on bots_discordinfo  (cost=0.56..7.35 rows=1 width=12)
              Index Cond: (user_id = auth_user.id)
```

With `ANALYZE=True`:

```python
In [2]: print(UserReaction.objects.all().values_list("clip__clip_personas__persona__user__discord_info")[:10].explain(analyze=True))
...same output as above...
Planning Time: 2.047 ms
Execution Time: 0.408 ms
```

## Snowflake Relates

https://docs.google.com/document/d/1XaESO8boAppswbSgzvftE8O1gNcZzWHHLb8Jpo-k2nY/edit


## Billing Data Management

### Seeding Billing Data

To set up billing plans, credit packs, and features for local development:

```shell
uv run python manage.py seed_billing_data
```

### Manual Subscription Management

You can manually set or remove user subscriptions using SQL queries:

```sql
-- Set Subscription (replace user_id and plan_id as needed)
UPDATE bots_discordinfo SET
    subscription_plan_id = '76784489-2240-4c35-9a42-f06e14da175b', -- UUID of plan in bots_usageplan
    subscription_period_type = 'month',
    subscription_status = 'active',
    subscription_period_end = '2025-07-22 16:38:36+00' -- a date in the future
WHERE user_id = 1; -- pk of user from auth_user

-- Remove Subscription
UPDATE bots_discordinfo SET
    stripe_customer_id = NULL,
    stripe_subscription_id = NULL,
    subscription_status = NULL,
    subscription_anchor = NULL,
    subscription_period_end = NULL,
    subscription_plan_id = NULL,
    subscription_changing_to = NULL,
    subscription_period_type = NULL,
    active_subscription_platform = NULL
    revcat_subscription_id = NULL
    WHERE user_id = 1; -- pk of user from auth_user
```

## Debugging with Postman

Suno has a Postman workspace which is useful for debugging API calls to studio_api. Please ask a team member to invite you if you haven’t been
invited already. To make requests to either your local backend or production, you’ll first need to generate an auth token. 


### Generating Local Token

1. Make sure you completed the Local Django Admin setup in the previous step. 
2. Login with your local admin creds at [localhost:8000/margu/](http://localhost:8000/margu/) 
3. Navigate to “User Tokens” on the left side bar → “Add User Token”
    1. There should already be a local token with which you can make requests in Postman, but you’re also free to create your own.

### Generating Prod Token 

1. Follow the instructions in the README to launch a [Jupyter notebook](https://github.com/suno-ai/glockenspiel/tree/main/studio_api#jupyter-lab). 
2. Copy and paste the below snippets into two separate cells.  
3. Update the `PROJECTPATH` to the path to the studio_api directory (`$<USER>/suno/glockenspiel/studio_api/` if you followed the exact instructions from [here](https://www.notion.so/Dev-environment-setup-193b01573ccf8019b8a8f9a3ec7bd8ca?pvs=21)). 
4. Ask a teammate for the url to the prod write database, and then update the `DATABASE_URL` variable.
5. Update the email variable with your suno email account. 
6. Run the first cell to initialize the Django environment.  After that’s set up, run the second cell — you should see a new token printed out.

```python

# in cell 1

import os, sys
import django
PROJECTPATH = '<path to studio_api directory>/glockenspiel/studio_api/'
sys.path.insert(0, PROJECTPATH)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "studio_api.settings")
os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"  # https://docs.djangoproject.com/en/4.1/topics/async/#async-safety

os.environ['DATABASE_URL'] = '----' #prod write db url

os.chdir(PROJECTPATH)
django.setup()

```

```python

# in cell 2

from django.contrib.auth.models import User
from studio_api.clips.models import UserToken

my_email = '<put your email here>'
user_object = User.objects.filter(email=my_email).first()
print(user_object)
token, _ = UserToken.objects.update_or_create(user=user_object)
print(token.token)

```

### Making Authenticated Requests 

1. Open up the Suno workspace in Postman.
2. Navigate to the Environments tab on the left sidebar.
3. Update the token value in the proper environment.

[Screenshot](https://photos.app.goo.gl/NsB2ZKVXsk9FJu8s6)

That’s it! Go to the Collections tab and make a request for the endpoint you’re interested in.