# Suno Orpheus: AI Music Chatbot Backend

## Project Overview

Orpheus is an AI-powered chatbot that helps users create music using
Suno's platform. Users can make, find, edit and remix songs through
natural conversation. This repository contains the backend services
for Orpheus, deployed as serverless functions on Modal.

**Related Repositories:**
- **Frontend Chat**: `../ui/app-ui/src/app/(root)/chat` (detailed below)
- **Frontend Studio Components**: `../ui/app-ui/src/components/studio`

## Architecture & Core Technologies

### Deployment & Infrastructure
- **Modal** - Serverless deployment platform for Python functions
- **FastAPI** - High-performance async API framework with streaming responses
- **Redis/Valkey** - Multi-partition streaming, session management, and distributed locking
- **AWS Firehose** - Event logging and analytics pipeline

### AI & Integration Services  
- **OpenAI API** - Chat completions with tool calling (GPT-4 models)
- **Clerk** - Authentication and user management
- **Ably** - Real-time messaging and streaming responses (primary delivery method)
- **Datadog** - Distributed tracing, logging, and monitoring

### Core Architecture: Ably-First Streaming

Orpheus uses an **Ably-first streaming architecture** where:
1. **HTTP endpoints** trigger chat completions but clients ignore the HTTP response
2. **Redis Streams** buffer and partition messages across multiple workers
3. **Ably channels** deliver real-time updates to subscribed clients
4. **Queue workers** consume Redis streams and publish to Ably with rate limiting

```
client ── HTTP (trigger) ──> modal worker ──> Redis streams ──> queue workers ──> Ably ──> client
         <── Ably subscribe (orpheus-chat:{session_id}) ──────────────────────────────────────────┘
```

#### Core Components

##### Chat Processing Pipeline
- **modal_runner_orpheus.py** - Main Modal worker: HTTP endpoints, prompt building, OpenAI streaming, Redis writes
- **modal_runner_orpheus_message_queue.py** - Queue workers: Redis stream consumers, Ably publishers, partition managers
- **streaming.py** - OpenAI stream processing and Redis stream enqueueing
- **tool_handlers.py** - Music-specific tool implementations (search, create, edit)

##### Service Layer
- **auth.py** - Clerk integration and JWT validation
- **history.py** - Session management and chat history persistence with message normalization
- **studio.py** - Integration with Suno's Studio DAW API
- **system_message.py** - System-generated responses using the same streaming pipeline
- **redis_utils.py** - Redis helpers, async pipelines, and Redis Streams operations

## Development Commands

### Local Development
```bash
# Install dependencies
uv sync

# Run tests  
uv run pytest                    # All tests
uv run pytest -m "not slow"     # Skip slow integration tests
uv run pytest tests/test_evals.py  # Evaluation test suite

# Code quality
uv run ruff check               # Linting
uv run ruff format              # Code formatting
uv run pyright                  # Type checking
```

### Modal Deployment
*IMPORTANT*: NEVER deploy modal workers to dev or prod.
ALWAYS ask the user for permission for deploying even the test modal worker.

```bash
# Deploy worker using deployment script (preferred)
./scripts/deploy_modal_worker.sh

# Or use the personal deployment script
./suno_orpheus/deplorpheus.sh          # Deploy to test environment
./suno_orpheus/deplorpheus.sh dev      # Deploy to dev environment  
./suno_orpheus/deplorpheus.sh prod     # Deploy to prod environment (requires confirmation)

# Or manually deploy both workers
modal deploy suno_orpheus/worker/modal_runner_orpheus.py
modal deploy suno_orpheus/worker/modal_runner_orpheus_message_queue.py
```

## Local Development Setup

### Prerequisites
- **Tailscale** - Required for local API routing
- **Modal account** - For deploying personal worker instances
- **Access to staging environment** - Redis, Clerk, etc.

### Setup Steps
1. **Configure deployment type** in `config/settings.py`:
   ```python
   # Add your personal deployment
   TEST_YOUR_NAME = "test-yourname"
   DEPLOYMENT_TYPE = "test-yourname"
   
   # Add to STUDIO_API_BASE_URLS
   "test-yourname": "https://your-machine.han-mahi.ts.net"
   ```

2. **Deploy personal Modal worker**:
   ```bash
   modal deploy suno_orpheus/worker/modal_runner_orpheus.py
   modal deploy suno_orpheus/worker/modal_runner_orpheus_message_queue.py
   ```

3. **Configure frontend** in `ui/app-ui/.env.local`:
   ```bash
   NEXT_PUBLIC_BASE_URL="https://localhost:3000"
   NEXT_PUBLIC_ORPHEUS_ENV="test-yourname"
   ```

### Environment Configuration
- **Production**: `DEPLOYMENT_TYPE = "prod"`
- **Staging**: `DEPLOYMENT_TYPE = "dev"` 
- **Personal dev**: `DEPLOYMENT_TYPE = "test-{name}"`

Each environment has dedicated:
- Studio API endpoints (`STUDIO_API_BASE_URLS`)
- Clerk authentication (`JWKS_URLS`)
- Redis partitions and worker instances

## Code Organization & Architecture

### Directory Structure
```
suno_orpheus/
├── config/                    # Environment and deployment settings
│   └── settings.py           # Deployment types, API URLs, Redis configuration
├── models/                    # Pydantic data models for chat and requests
│   └── chat.py               # ChatRequestBody, ChatHistoryItem, ToolCall models
├── services/                  # Business logic layer
│   ├── auth.py               # Clerk JWT verification and debug token bypass
│   ├── history.py            # Session management and message history normalization
│   ├── redis_utils.py        # Redis helpers and Redis Streams operations
│   ├── streaming.py          # OpenAI streaming and Redis stream enqueueing
│   ├── studio.py             # Studio API integration (search, similarity, playlists)
│   ├── system_message.py     # System-generated responses
│   ├── tool_handlers.py      # Tool execution and Studio API calls
│   └── tracing.py            # Datadog tracing wrappers
├── worker/                    # Modal deployment functions
│   ├── modal_runner_orpheus.py              # Main HTTP API and chat orchestration
│   ├── modal_runner_orpheus_message_queue.py # Redis stream consumers and Ably publishers
│   └── utils.py              # OpenAI client initialization
├── tests/                     # Test suite including evaluation framework
│   ├── eval_test_data/       # Test data for evaluation framework
│   ├── test_evals.py         # Parallel evaluation test runner
│   └── test_*.py             # Unit tests for various components
├── image/                     # Modal base image configuration
│   └── base.py               # Modal image with AWS CLI and Datadog
├── scripts/                   # Deployment and utility scripts
│   └── deploy_modal_worker.sh # Main deployment script
├── deplorpheus.sh             # Personal deployment script with environment checking
├── distrubuted_locking.py     # Redis-based distributed locking (note: typo in filename)
├── orpheus_*.py               # Core chat functionality (prompts, tools, utils, genres)
├── pyproject.toml             # uv package configuration and dependencies
└── workspace.yaml             # Repository workspace configuration
```

### Key Architectural Patterns

#### Tool-Based Chat System
Orpheus uses OpenAI's tool calling to handle music operations:
```python
# Available tools in orpheus_tools.py
tools = [
    "simple_message", "write_lyrics", "generate_song", 
    "search_library", "search_public_clips_descriptive", 
    "search_public_clips_similarity", "create_playlist_with_clips", 
    "generate_image", "listen_to_audio"
]
```

#### Ably-First Streaming Architecture
Real-time responses via Redis Streams → Queue Workers → Ably:
```python
# Producer: streaming.py
async def generate_and_store_orpheus_message():
    # Process OpenAI stream deltas
    # Store in Redis
    # Enqueue to partitioned Redis stream
    add_message_to_partition(session_id, message_frame)

# Consumer: modal_runner_orpheus_message_queue.py
async def consume_and_publish():
    # XREADGROUP from Redis stream partition
    # Publish to Ably channel: orpheus-chat:{session_id}
    # XACK message after publish
```

#### Multi-Partition Architecture
Redis Streams with consistent hashing for scalability:
```python
# Configuration
NUM_PARTITIONS = 50  # Number of queue workers
partition = get_worker_id(session_id) % NUM_PARTITIONS
stream_name = f"orpheus-streaming-partition-{partition}"
```

#### Distributed Locking
Redis-based locking for concurrent request handling:
```python
from suno_orpheus.distrubuted_locking import RedisLock  # Note: typo in filename

async with RedisLock(redis_client, f"session:{session_id}"):
    # Process request safely
```

### Data Models & Types

#### Core Chat Models (`models/chat.py`)
- `ChatRequestBody` - Incoming chat requests
- `ChatHistoryItem` - Individual message storage
- `ContextStateRequestBody` - Session context updates

#### Session Management & Redis Architecture
- **Multi-partition Redis storage** (`NUM_PARTITIONS = 5`)
- **Redis Streams** for message queueing: `orpheus-streaming-partition-{0-4}`
- **Session data**: `chat_{session_id}` hash with metadata
- **Message storage**: `chat_{session_id}:messages` set + individual message hashes
- **User sessions**: `user_{user_id}:sessions` set for session discovery
- **Context state**: `clip_status:{session_id}` and `session_status:{session_id}` hashes
- **Message history normalization** with tool call reordering in `history.py`

#### Ably Channels & Events
- **Channel**: `orpheus-chat:{session_id}`
- **Events**:
  - `orpheus-message`: streaming content and lifecycle messages
  - `orpheus-tool-call`: tool call streaming chunks  
  - `orpheus-session-update`: session metadata updates
- **Rate limiting**: ~40 messages/sec per partition with per-channel spacing
- **Message ordering**: `chunk_index` field for proper client-side rendering

## Development Guidelines

### Code Style & Conventions
- **Single Level of Abstraction** IMPORTANT: keep functions at a single level of abstraction.  Don't mix and match.
- **Helper Functions**: It's fine to use well-named helper functions for organization even if they're only called once.
- **Function length**: Keep functions under 80 lines
- **Error handling**: Tightly scoped try-except blocks, never nested
- **Async patterns**: Use async/await consistently, avoid blocking operations
- **Logging**: Structured logging with Datadog integration

### Testing Strategy
- **Unit tests**: Individual service and utility functions (8+ test files)
- **Integration tests**: OpenAI API interactions and tool calling
- **Evaluation framework**: `tests/test_evals.py` with parallel execution and success rate reporting
- **Test data**: `tests/eval_test_data/` directory with message history fixtures
- **Slow test marker**: Use `pytest -m "not slow"` to skip expensive tests
- **Parallel test runner**: `run_parallel_tests()` function for performance evaluation

### Performance Considerations
- **Modal cold starts**: Minimize imports, use image caching
- **Redis optimization**: Multi-partition architecture for scalability  
- **Streaming responses**: Buffer management (`TOKENS_TO_FLUSH_BUFFER = 20`)
- **OpenAI rate limits**: Proper retry logic and error handling

### Security & Authentication
- **JWT validation**: Clerk integration with proper token verification
- **Authorization**: User-scoped operations and data access
- **Secrets management**: Environment-based configuration
- **Debug access**: `ORPHEUS_DEBUG_TOKEN` for development only

## Monitoring & Observability

### Datadog Integration
- **Distributed tracing**: Request correlation across services
- **Custom metrics**: Chat completion timing, tool usage, error rates
- **Structured logging**: JSON format with contextual fields

### Health Checks & Debugging
- **Debug endpoints**: Chat history access with debug token
- **Redis health**: Connection pooling and retry logic
- **Modal worker status**: Deployment verification endpoints

## Common Development Tasks

### Adding New Tools
1. Define tool schema in `orpheus_tools.py`
2. Implement handler in `tool_handlers.py`
3. Add tool to available tools list
4. Write unit tests for tool functionality

### Modifying Chat Behavior  
1. Update system prompts in `orpheus_prompts.py`
2. Test with evaluation framework in `tests/test_evals.py`
3. Monitor conversation quality metrics

### Deployment & Rollout
1. Test changes in personal dev environment (`test-{name}`)
2. Deploy to staging (`dev` environment)
3. Run evaluation tests to verify quality
4. Deploy to production with monitoring

## Frontend Integration (Next.js)

**Location**: `../ui/app-ui/src/app/(root)/chat` *(separate repository)*

The Orpheus frontend is built with **Next.js 14**, **TypeScript**, **Tailwind CSS**, and integrates with the backend via **Ably real-time messaging**.

### Frontend Architecture

#### Core Components
- **ChatClient.tsx** - Main chat interface with Ably integration
- **OrpheusAblyProvider.tsx** - Ably client setup and authentication
- **useChat.ts** - Chat state management with Zustand + Immer
- **MessageList.tsx** - Renders chat messages with streaming animations
- **ChatInput.tsx** - Input handling with recording, file upload, and reference support

#### Ably Integration Pattern
```typescript
// OrpheusAblyProvider.tsx - Client initialization
const client = new Ably.Realtime({
  authUrl: `${modalBaseUrl}/ably-auth`,
  authHeaders: { Authorization: `Bearer ${token}` },
  clientId: `user:${clerk.session?.user.id}`,
});

// ChatClient.tsx - Event subscriptions
const channel = ably.channels.get(`orpheus-chat:${sessionId}`);
channel.subscribe('orpheus-message', messageHandler);
channel.subscribe('orpheus-tool-call', toolCallHandler);
channel.subscribe('orpheus-session-update', sessionUpdateHandler);
```

#### Frontend Directory Structure
```
../ui/app-ui/src/app/(root)/chat/
├── ChatClient.tsx              # Main chat interface
├── OrpheusAblyProvider.tsx     # Ably authentication & client setup
├── useChat.ts                  # Chat state management (Zustand)
├── ChatInput.tsx               # Input with recording/upload modes
├── MessageList.tsx             # Message rendering with animations
├── components/
│   ├── input/                  # Input components & reference handling
│   │   ├── ChatInputBarContainer.tsx
│   │   ├── ReferenceTypes.ts   # Type definitions for references
│   │   ├── RecordWaveformContainer.tsx
│   │   └── ActionButton.tsx
│   └── messages/               # Message type renderers
│       ├── AssistantMessage.tsx    # Animated text rendering
│       ├── ClipsMessage.tsx        # Music clip displays
│       ├── LyricsMessage.tsx       # Lyrics formatting
│       └── PlaylistMessage.tsx     # Playlist displays
├── [slug]/                     # Dynamic routing
│   ├── ChatPageClient.tsx
│   └── page.tsx
└── utils.ts                    # Helper functions
```

#### Key Frontend Features
- **Real-time streaming**: Ably subscriptions with incremental message rendering
- **Animated text display**: Token-by-token animation in `AssistantMessage.tsx`
- **Multi-modal input**: Text, voice recording, file upload via `ChatInput.tsx`
- **Reference system**: Clip references, lyrics references, and context management
- **Authentication**: Clerk integration with Ably token exchange
- **State management**: Zustand with Immer for complex chat state
- **Recording capabilities**: Waveform visualization and audio capture

#### Message Flow (Frontend)
1. **User input** → `ChatInput.tsx` → HTTP POST `/chat` (trigger only)
2. **Backend processing** → Redis Streams → Queue Workers → Ably publish
3. **Ably events** → `ChatClient.tsx` subscribers → `useChat.ts` state updates
4. **UI updates** → `MessageList.tsx` → `AssistantMessage.tsx` animated rendering

#### Environment Configuration
```bash
# ui/app-ui/.env.local
NEXT_PUBLIC_BASE_URL="https://localhost:3000"
NEXT_PUBLIC_ORPHEUS_ENV="test-yourname"  # Links to backend DEPLOYMENT_TYPE
```

### Frontend Development Setup
1. **Configure backend** with personal `DEPLOYMENT_TYPE` in backend `config/settings.py`
2. **Deploy Modal workers** for your test environment
3. **Set frontend env vars** in `ui/app-ui/.env.local` pointing to your deployment
4. **Start frontend**: `npm run dev` in `ui/app-ui/`
5. **Access chat**: Navigate to `/chat` route in your Next.js app

## Troubleshooting

### Common Issues
- **Modal deployment failures**: Check image dependencies and secrets
- **Redis connection issues**: Verify environment settings and partitions
- **Authentication errors**: Check Clerk configuration and JWT validation
- **Tool calling failures**: Validate OpenAI API responses and error handling
- **Ably connection issues**: Verify DEPLOYMENT_TYPE matches between frontend/backend
- **Frontend build errors**: Check Next.js version compatibility with Ably React SDK