import { Callout } from 'nextra/components'

# DataDog Monitoring Setup
This guide explains how to set up DataDog monitoring (metrics, traces, and logs) for Modal workers.

### 1. Required Secrets

Add these secrets to your Modal worker configuration:

```python
SECRETS = [
    modal.Secret.from_dict(
        {
            "DD_SITE": "datadoghq.com",
            "DD_ENV": DEPLOYMENT_TYPE,  # "dev" or "prod"
            "DD_SERVICE": APP_NAME,  # or a specific service name
            "DD_LOGS_ENABLED": "false",  # Usually false to reduce costs
            "DD_TRACE_ENABLED": "true",   # Enable distributed tracing
            "DD_TRACE_SAMPLING_RULES": PROD_TRACE_SAMPLING_RULES if DEPLOYMENT_TYPE == "prod" else "",
        },
    ),
    modal.Secret.from_name("datadog-metrics"),  # Contains DD_API_KEY
]
```

**What each config does:**

- **DD_SITE**: DataDog site URL (always `datadoghq.com` for US)
- **DD_ENV**: Environment tag for filtering metrics/traces (dev/prod/staging)
- **DD_SERVICE**: Service name that appears in DataDog dashboards
- **DD_LOGS_ENABLED**:
  - `"false"` = Don't send logs to DataDog (recommended for high-volume workers to reduce costs)
  - `"true"` = Send all logs to DataDog (use for critical services only)
- **DD_TRACE_ENABLED**:
  - `"true"` = Enable distributed tracing (APM)
  - `"false"` = Disable tracing
- **DD_TRACE_SAMPLING_RULES**: JSON string controlling trace sampling rates
  - Empty string in dev = trace everything (100%)
  - In prod: typically sample errors at 100%, other requests at 1-10%
  - Example: `'[{"service":"*","name":"error.*","sample_rate":1.0},{"service":"*","name":"*","sample_rate":0.01}]'`


### 2. Send Metrics

Use the `statsd` client to send metrics:

```python
from datadog import statsd

# Counter - count occurrences
statsd.increment("video_gen.client.error", tags=["model:wan", "error:true"])

# Distribution - measure values (duration, size, etc.)
duration_seconds = time.time() - start_time
statsd.distribution("video_gen.client.duration", duration_seconds, tags=["model:wan"])

# Gauge - track current values
statsd.gauge("queue.depth", queue_size, tags=["queue:image_gen"])

# Histogram - statistical distribution
statsd.histogram("request.size", request_bytes, tags=["endpoint:generate"])
```

**Metric types explained:**
<Callout type="warning">
  **Warning**: Using `increment` for counters in modal is currently unreliable. Datadog deduplicates metrics by (timestamp second, tags) and for serverless tags are often the same (since they don’t include host/pod). So we might drop some counts if they’re submitted frequently. You should use `distribution` instead, which will come with a count as well.
</Callout>
- **increment/counter**: Count events (errors, requests, completions)
- **distribution**: Measure durations, sizes, or any numeric values (recommended over histogram)
- **gauge**: Track current state (queue depth, active connections, memory usage)
- **histogram**: Like distribution but with different backend aggregation (use distribution for new code)

### 3. Required Dependencies

Ensure your Modal image includes:

```python
image = (
    modal.Image.debian_slim()
    .pip_install_from_pyproject("path/to/suno_utils/pyproject.toml")  # Includes datadog and ddtrace
    # ... other dependencies
)
```

The suno_utils `pyproject.toml` should already include:
```toml
dependencies = [
    "datadog==0.51.0",  # For metrics
    "ddtrace==2.21.1",  # For tracing (optional, only if using distributed tracing)
]
```

### Additional Dockerfile Commands

```python
base_image = (
    modal.Image.from_dockerfile("path/to/Dockerfile")
    .dockerfile_commands(
        [
            "COPY --from=datadog/serverless-init /datadog-init /app/datadog-init",
            'ENTRYPOINT ["/app/datadog-init"]',
        ]
    )
    .pip_install("datadog", "ddtrace")
)
```

**What this does:**
- Copies the DataDog serverless initialization binary into your container
- Sets it as the entrypoint so DataDog starts before your application
- This is required for containerized workers because they don't have access to Modal's built-in DogStatsD agent


## Trace Sampling Rules

Control tracing costs with sampling rules:

```python
# Development: Trace everything
DD_TRACE_SAMPLING_RULES = ""

# Production: Sample errors at 100%, everything else at 1%
PROD_TRACE_SAMPLING_RULES = (
    '[{"service":"*","name":"error.*","sample_rate":1.0},'
    '{"service":"*","name":"*","sample_rate":0.01}]'
)
```

**Why sampling matters:**
- Tracing every request in production is expensive
- You always want to see errors (100% sampling)
- For successful requests, 1% sampling is usually sufficient for pattern detection
- Adjust `sample_rate` based on your traffic volume and budget

## Viewing Metrics in DataDog

After deployment:

1. **Metrics**: Navigate to Metrics → Explorer in DataDog
2. **Traces**: Navigate to APM → Traces
3. **Logs**: Navigate to Logs → Live Tail (if DD_LOGS_ENABLED=true)

Metrics typically appear within 1-5 minutes of the first worker invocation.