# Datadog Metrics for Hooks Recs

## Metrics Overview

### Request Metrics
- `recs.request.count` (counter) tags: experiment, surface
- `recs.request.total_ms` (distribution) tags: elc, experiment
- `recs.request.candidate_survival_rate` (distribution) tags: elc, experiment
- `recs.request.recommendation_fulfillment` (distribution) tags: elc, experiment

### Orchestrator Metrics
- `recs.orchestrator.ipc_ms` (distribution) - Modal IPC overhead

### User Data Metrics
- `recs.user.fetch_ms` (distribution) tags: source:watch_history|hook_signals_pipeline|user_profile_pipeline
- `recs.user.signals.size` (distribution) tags: signal:watch_history|liked_hooks|omniplay|long_watch|comment|etc
- `recs.user.elc_mode` (counter) tags: elc:true|false

### Embedding Metrics
- `recs.embed.fetch_ms` (distribution)

### Bucket Metrics
- `recs.bucket.retrieval_ms` (distribution) tags: bucket
- `recs.bucket.candidates` (distribution) tags: bucket

### Cache Metrics
- `recs.cache.hit` (counter) tags: bucket
- `recs.cache.miss` (counter) tags: bucket

### Reranking Metrics
- `recs.rerank.batch_ms` (distribution)

### Fusion Metrics
- `recs.fusion.ms` (distribution)

### Final Metrics
- `recs.final.bucket_share` (distribution) tags: bucket

### Diversity Metrics
- `recs.diversity.video_cover.ratio` (distribution) - Use `1-ratio` formula for user upload ratio

### Warmer Metrics
- `recs.warmer.run` (counter)
- `recs.warmer.duration_ms` (distribution)
- `recs.warmer.bucket_refresh` (counter) tags: bucket

## Tag Definitions

### Common Tags
- `service`: hooks-recs (constant)
- `elc`: true|false (early life cycle mode)
- `experiment`: experiment name or "default"
- `surface`: hooks_feed|unknown
- `bucket`: follow|audio_similarity|video_similarity|genre_similarity|fresh|popular|creator_champion|manual_champion|collaborative_filtering

### Source Tags
- `source`: Pipeline names for user data fetching
  - watch_history: Hook watch history fetch
  - hook_signals_pipeline: Batched fetch of 7 hook engagement signals + CF
  - user_profile_pipeline: Batched fetch of clips + tags

### Signal Tags
- `signal`: Pre-filter signal types for size distributions
  - watch_history: Watch history size
  - liked_hooks: Liked hooks count
  - omniplay, long_watch, comment, etc: Engagement signal counts

## Usage Patterns

### Cache Miss Rate
```
cache_miss_rate = recs.cache.miss / (recs.cache.hit + recs.cache.miss)
```

### User Upload Ratio
```
user_upload_ratio = 1 - recs.diversity.video_cover.ratio
```

### Recommendation Quality
Monitor percentiles (p50, p95, p99) of:
- `recs.request.recommendation_fulfillment` - Are we meeting request counts?
- `recs.request.candidate_survival_rate` - Are filters too aggressive?
- `recs.final.bucket_share` by bucket - Is content diverse?

### Performance Monitoring
- `recs.request.total_ms` - End-to-end latency
- `recs.user.fetch_ms` by source - Redis pipeline performance
- `recs.bucket.retrieval_ms` by bucket - ES query performance

### User Signal Monitoring
- `recs.user.signals.size` percentiles - Watch for p99 explosion in watch_history or other signals