# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Development Commands

### Environment Setup

```bash
# Install dependencies (use -e for editable mode)
uv run pip install -e ".[dev]"

# Start Dagster development server
uv run dagster dev
```

### Code Quality

```bash
# Run linting with ruff
ruff check .
ruff format .

# Run SQL linting with sqlfluff (from dbt directory)
cd src/assets/dbt/analytics
uv run sqlfluff lint
uv run sqlfluff fix
```

Also, make sure that SQL files not-related to DBT always end with a semicolon and a newline after.

### Testing

```bash
# Run Python tests
pytest

# Run dbt tests (from analytics directory)
cd src/assets/dbt/analytics
uv run --env-file <path/to/env/file> dbt test
```

### dbt Development

```bash
# Run specific dbt models (from analytics directory)
cd src/assets/dbt/analytics
uv run --env-file <path/to/env/file> dbt run --select model_name
uv run --env-file <path/to/env/file> dbt run --select model_name+ --full-refresh

# Compile dbt without running
uv run --env-file <path/to/env/file> dbt compile

# Debug dbt configuration
uv run --env-file <path/to/env/file> dbt debug
```

## Project Architecture

### Core Structure

- **`src/definitions.py`**: Central Dagster definitions file that auto-loads jobs, assets, schedules, and sensors from organized directories
- **`src/assets/`**: Contains all data assets organized by source system:
  - `bots/`: Bot-related data processing
  - `dbt/`: dbt models and transformations (primary data transformation layer)
  - `postgres/`: PostgreSQL data assets
  - `snowflake/`: Snowflake-specific assets
  - `trending/`: Trending analysis assets
  - `recommendation/`: Recommendation engine assets
  - `rds_glue/`: RDS and AWS Glue integration
- **`src/jobs/`**: Dagster job definitions
- **`src/schedules/`**: Scheduled job configurations
- **`src/utils/`**: Shared utilities and helper functions

### dbt Integration

- **Location**: `src/assets/dbt/analytics/`
- **Models organized by layer**:
  - `models/raw/`: Direct source references
  - `models/staging/`: Cleaned and standardized data
  - `models/intermediate/`: Business logic (ephemeral)
  - `models/marts/`: Final analytical tables
- **Custom macros**: Partition filtering macros (use `partition_filter_daily()` instead of `is_incremental()`)
- **Automation conditions**: Use `automation_condition=daily_cron_with_eager_historical_backfill_condition` tags instead of legacy `cadence=*` tags

### Key Configuration Files

- **`pyproject.toml`**: Python dependencies and tool configuration (ruff, dagster)
- **`src/assets/dbt/analytics/dbt_project.yml`**: dbt project configuration with model-level automation conditions
- **`src/assets/dbt/analytics/profiles.yml`**: Snowflake connection configuration (dev/stg/prod)
- **`.sqlfluff`**: SQL linting configuration for Snowflake dialect

### Resource Management

- **Snowflake warehouses**: Different sizes available (DBT_DEV_MEDIUM, FACT_HOOK_PLAY_LARGE, etc.)
- **Auto-loading**: The `auto_load_from_directory()` function in `definitions.py` automatically discovers and registers Dagster assets, jobs, schedules, and sensors

### Environment Configuration

- **`.env` file required**: Contains `DBT_USER`, `DBT_SNOWFLAKE_PRIVATE_KEY_PATH`, `DBT_TARGET`
- **Three environments**: dev (local), stg (staging), prod (production)
- **Dynamic database references**: Uses Jinja templating for environment-specific configuration

### Development Workflow

1. Use Dagster UI (`uv run dagster dev`) for asset materialization and monitoring
2. Use automation conditions for scheduling instead of cron-based approaches
3. All dbt models must have automation conditions defined
4. Use partition filtering macros for incremental models
5. Test locally in dev environment before deploying

### Important Notes

- **Never use legacy `cadence=*` tags** - use automation conditions instead
- **Use `partition_filter_daily()` macro** instead of dbt's `is_incremental()`
- **All models require automation conditions** - models without conditions will not run
- **Snowflake key-pair authentication** required for database connections
- **Branch deployments** automatically triggered via GitHub workflows