# Feast Feature Store for Suno Recommendations

Feature store for managing ML features in the Suno song/video recommendation system, using **Snowflake** as the offline data source.

## Environments

This feature store supports two environments:

- **DEV**: Development environment with local Redis and dev Snowflake database
- **PROD**: Production environment with production Redis and Snowflake

Each environment has its own configuration file and registry path to prevent conflicts.

## Quick Start

### Development Environment

```bash
# 1. Install dependencies
make install

# 2. Set up environment variables for Snowflake
export SNOWFLAKE_PRIVATE_KEY="$(cat /Users/$USER/Documents/snowflake_secrets/rsa_key.p8)"

# 3. Ensure local Redis is running (for dev)
# Redis should be accessible at localhost:6379

# 4. Apply feature definitions to DEV
# Note: The Makefile automatically sets FEAST_SNOWFLAKE_DATABASE=SUNO_DEV_DELPHINE
make apply-dev
# or simply: make apply (defaults to dev)

# 5. List registered features in DEV
make list-dev
```

### Production Environment

```bash
# 1. Install dependencies
make install

# 2. Set up environment variables
export SNOWFLAKE_PRIVATE_KEY="$(cat /path/to/your/rsa_key.p8)"
export REDIS_RECS_URL="redis://your-prod-redis-url:6379"

# 3. Apply feature definitions to PROD
# Note: The Makefile automatically sets FEAST_SNOWFLAKE_DATABASE=SUNO_PROD
make apply-prod

# 4. List registered features in PROD
make list-prod
```

**Important**: The Makefile automatically sets the `FEAST_SNOWFLAKE_DATABASE` environment variable based on which environment you're using. This ensures that feature definitions read from the correct database (dev vs prod). If you run Feast commands directly (without using `make`), you must set this variable manually:

```bash
# For dev
export FEAST_SNOWFLAKE_DATABASE=SUNO_DEV_DELPHINE
feast apply --feature-store-yaml feature_store_dev.yaml

# For prod
export FEAST_SNOWFLAKE_DATABASE=SUNO_PROD
feast apply --feature-store-yaml feature_store_prod.yaml
```

## Structure

```
feast/
├── feature_store.yaml          # Legacy config (kept for backwards compatibility)
├── feature_store_dev.yaml      # DEV environment configuration
├── feature_store_prod.yaml     # PROD environment configuration
├── pyproject.toml              # Python dependencies
├── feature_repo/               # Feature definitions
│   ├── entities.py             # Entity definitions (user, hook, clip, creator)
│   ├── user_features.py        # User feature views
│   └── ...                     # Other feature definitions
├── Makefile                    # Helper commands for both environments
└── test_connection.py          # Connection testing script
```

## Data Sources

- **Offline Store**: Snowflake (for training data and historical features)
- **Online Store**: Redis (for low-latency real-time serving)

## Expected Snowflake Table Schema

Your Snowflake table should have this structure:

```sql
CREATE TABLE user_watch_behavior (
    user_id INT,
    average_watch_time FLOAT,
    event_timestamp TIMESTAMP,
    created_timestamp TIMESTAMP
);
```

## Usage

### Working with Different Environments

When using Feast in Python code, specify which environment's config to use:

```python
from feast import FeatureStore
import pandas as pd
from datetime import datetime

# For DEV environment
store_dev = FeatureStore(
    repo_path=".",
    config_path="feature_store_dev.yaml"
)

# For PROD environment
store_prod = FeatureStore(
    repo_path=".",
    config_path="feature_store_prod.yaml"
)
```

### Get Features for Training

```python
from feast import FeatureStore
import pandas as pd
from datetime import datetime

# Use the appropriate store for your environment
store = FeatureStore(repo_path=".", config_path="feature_store_dev.yaml")

# Historical features (from Snowflake)
entity_df = pd.DataFrame({
    "user_id": [123, 456, 789],
    "event_timestamp": [datetime(2024, 11, 1)] * 3,
})

training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["user_watch_behavior:average_watch_time"],
).to_df()
```

### Get Features for Serving

```python
# Online features (from Redis)
store = FeatureStore(repo_path=".", config_path="feature_store_dev.yaml")

features = store.get_online_features(
    features=["user_watch_behavior:average_watch_time"],
    entity_rows=pd.DataFrame({"user_id": [123]}),
).to_dict()
```

### Materialize Features to Online Store

Materialization syncs features from Snowflake (offline store) to Redis (online store) for low-latency serving.

**For DEV:**
```bash
# Full materialization (specify date range)
make materialize-dev START_DATE=2025-01-01 END_DATE=2025-01-02

# Incremental materialization (recommended for production workflows)
make materialize-incremental-dev END_DATE=2025-01-02T00:00:00
```

**For PROD:**
```bash
# Full materialization
make materialize-prod START_DATE=2025-01-01 END_DATE=2025-01-02

# Incremental materialization (use this in production)
make materialize-incremental-prod END_DATE=2025-01-02T00:00:00
```

**Using current timestamp:**
```bash
# DEV incremental materialization with current time
make materialize-incremental-dev END_DATE=$(date -u +"%Y-%m-%dT%H:%M:%S")

# PROD incremental materialization with current time
make materialize-incremental-prod END_DATE=$(date -u +"%Y-%m-%dT%H:%M:%S")
```

## Available Commands

Run `make help` to see all available commands. Key commands:

- `make apply-dev` / `make apply-prod` - Apply feature definitions
- `make list-dev` / `make list-prod` - List feature views
- `make validate-dev` / `make validate-prod` - Validate configurations
- `make materialize-dev` / `make materialize-prod` - Materialize features
- `make ui-dev` / `make ui-prod` - Start Feast Web UI

## Environment Differences

| Setting | DEV | PROD |
|---------|-----|------|
| **Project Name** | `suno_recs_dev` | `suno_recs_prod` |
| **Registry Path** | `s3://.../dev/registry.db` | `s3://.../prod/registry.db` |
| **Redis** | `localhost:6379` | `${REDIS_RECS_URL}` |
| **Snowflake Database** | `SUNO_DEV_DELPHINE` | `SUNO_PROD` |
| **Snowflake Warehouse** | `DBT_DEV_MEDIUM` | `DBT_PROD_MEDIUM` |

See [Feast documentation](https://docs.feast.dev/) for more details.