sources: - name: rds config: group: rds database: "{{ target.database }}" schema: prod # Assuming same schema tables: - name: rds_public_clip description: | Public clip table for bots generated clips from RDS, containing clip IDs that are marked as public. This table is populated by the Dagster rds_public_clip asset and is used to determine if a clip is public. tags: ["verified", "partition_type=hourly"] columns: - name: clip_id description: "Clip ID (primary key)" - name: p_date description: "Partition date" - name: p_hour description: "Partition hour" meta: dagster: asset_key: ["rds_public_clip"] - name: rds_survey_responses description: | Survey responses data from RDS containing user feedback and survey responses. This table stores survey answers, ratings, and feedback data collected from users through various survey mechanisms. Used for user satisfaction analysis, product feedback, and user experience research. tags: ["verified", "survey"] meta: dagster: asset_key: ["rds_survey_responses"] columns: - name: id description: "Unique survey response identifier (VARCHAR)" - name: response description: "Survey response content (STRING)" - name: created_at description: "Timestamp when response was submitted (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: question_id description: "Question identifier (VARCHAR)" - name: user_id description: "User who provided the response (NUMBER)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (INT)" - name: rds_survey_questions description: | Survey questions data from RDS containing survey question definitions and metadata. This table stores question text, types, options, and survey structure information. Used for understanding survey design and analyzing response patterns. tags: ["verified", "survey"] meta: dagster: asset_key: ["rds_survey_questions"] columns: - name: id description: "Unique question identifier (VARCHAR)" - name: prompt description: "Text of the survey question (STRING)" - name: type description: "Type of question (multiple choice, text, rating, etc., VARCHAR)" - name: options description: "Available options for multiple choice questions (VARCHAR)" - name: group_id description: "Survey group identifier (VARCHAR)" - name: is_active description: "Whether the question is active (BOOLEAN)" - name: is_required description: "Whether the question is required (BOOLEAN)" - name: order_number description: "Order of question within survey (INT)" - name: created_at description: "Timestamp when question was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (INT)" - name: clip description: | Raw clip data sourced from RDS (Relational Database Service). This table contains the core clip information including basic metadata, user interactions, generation details, and status information. This is the primary source table for clip data before it gets processed into the more comprehensive dim_clip dimension table. The table is clustered by p_date and p_hour for optimal query performance and contains both generated and uploaded clips with their associated metadata, engagement metrics, and content details. tags: ["verified", "partition_type=hourly"] meta: dagster: asset_key: ["clip"] columns: - name: id description: "Unique identifier for the clip (primary key)" - name: created_at description: "Timestamp when the clip was created (stored as string)" - name: updated_at description: "Timestamp when the clip was last updated (stored as string)" - name: time_used description: "Time in seconds taken to generate the clip" - name: metadata description: "JSON metadata containing additional clip information and generation parameters" - name: user_id description: "ID of the user who created the clip" - name: status description: "Current status of the clip (e.g., completed, failed, processing)" - name: discord_message_id description: "Discord message ID if the clip was shared via Discord" - name: prompt_id description: "ID of the prompt used for clip generation" - name: request_id description: "Unique identifier for the generation request that created this clip" - name: is_generated description: "Boolean indicating if the clip was AI-generated (true) or uploaded (false)" - name: s3_id description: "S3 object identifier for the clip's audio file" - name: upvote_count description: "Number of upvotes/likes the clip has received" - name: batch_index description: "Index/order of the clip within a batch generation request" - name: model_name description: "Name of the AI model used for generation" - name: prompt_text description: "Text prompt used for clip generation" - name: daily_theme_id description: "ID of the daily theme if the clip was created for a theme challenge" - name: is_deleted description: "Boolean indicating if the clip has been soft deleted" - name: image_s3_id description: "S3 object identifier for the clip's cover image" - name: is_public description: "Boolean indicating if the clip is publicly visible" - name: dislike_count description: "Number of dislikes the clip has received" - name: flag_count description: "Number of times the clip has been flagged for inappropriate content" - name: play_count description: "Total number of times the clip has been played" - name: skip_count description: "Number of times the clip was skipped during playback" - name: title description: "Title of the clip" - name: slug description: "URL-friendly slug for the clip" - name: p_date description: "Partition date - date when the clip was created (stored as string, used for table clustering)" - name: p_hour description: "Partition hour - hour when the clip was created (used for table clustering)" - name: created_session_id description: "ID of the user session when the clip was created" - name: allow_comments description: "Boolean indicating if comments are allowed on the clip" - name: rds_auth_user description: | User authentication data from RDS containing core user account information and authentication details. This table stores essential user profile data including usernames, email addresses, account status, and administrative flags. Data is synchronized from the PostgreSQL RDS instance to Snowflake for analytics and reporting purposes. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_auth_user"] columns: - name: id description: "Unique user identifier (primary key)" - name: username description: "User's chosen username" - name: date_joined description: "Date when user account was created (stored as string)" - name: is_staff description: "Boolean indicating if user has staff/admin privileges" - name: p_date description: "Partition date for table optimization (stored as string)" - name: p_hour description: "Partition hour for table optimization" - name: email description: "User's email address" - name: rds_discord_info description: "User info from RDS" tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_discord_info"] - name: rds_period_credit_usage description: | Period-based credit usage data from RDS containing user credit consumption and subscription information. This table tracks credit usage per billing period (monthly for subscribers, daily for free users) including credit limits, extra credits, and subscription details. Data is synchronized from PostgreSQL RDS to Snowflake for billing analytics and usage tracking. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_period_credit_usage"] columns: - name: id description: "Unique record identifier (VARCHAR)" - name: created_at description: "When the record was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: period_anchor description: "Period start timestamp (TIMESTAMP_TZ)" - name: credits_used description: "Credits used in this period (NUMBER)" - name: period_credit_limit description: "Credit limit for this period (NUMBER)" - name: extra_credits_balance description: "Extra credits balance (NUMBER)" - name: user_id description: "User identifier (NUMBER)" - name: period_end description: "Period end timestamp (TIMESTAMP_TZ)" - name: stripe_subscription_id description: "Stripe subscription ID (VARCHAR)" - name: plan_level description: "Plan level (NUMBER)" - name: extra_credits_used description: "Extra credits used (NUMBER)" - name: usage_plan_id description: "Usage plan identifier (VARCHAR)" - name: revcat_subscription_id description: "RevenueCat subscription ID (VARCHAR)" - name: is_active description: "Whether the subscription is active (BOOLEAN)" - name: period_extra_credit_limit description: "Extra credit limit for this period (NUMBER)" - name: p_date description: "Partition date for table optimization (VARCHAR)" - name: rds_user_delete_request description: "User deletion requests from RDS" tags: ['raw', 'postgres', 'partition_type=daily'] meta: dagster: asset_key: ["rds_user_delete_request"] - name: rds_session_history description: | Session history data from RDS containing user session tracking information. This table stores session details including creation time, expiration, duration, platform information, and session properties. Data is synchronized from PostgreSQL RDS to Snowflake for user behavior analysis and session management. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_session_history"] columns: - name: session_id description: "Unique session identifier (VARCHAR)" - name: session_type description: "Type of session (NUMBER)" - name: user_id description: "User identifier (NUMBER)" - name: ip description: "IP address (VARCHAR)" - name: created_at description: "When the session was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: expire_time description: "Session expiration time (TIMESTAMP_TZ)" - name: length_in_sec description: "Session length in seconds (NUMBER)" - name: properties description: "Session properties (JSON string, VARCHAR)" - name: platform description: "Platform identifier (web/mobile/app, VARCHAR)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (NUMBER)" - name: rds_usage_plan meta: dagster: asset_key: ["rds_usage_plan"] description: | Usage plan configurations from RDS containing subscription plan details and pricing information. This table stores plan metadata including credit limits, pricing, Stripe integration details, and feature configurations. Data is synchronized from PostgreSQL RDS to Snowflake for subscription analytics and billing analysis. tags: ['raw', 'postgres'] columns: - name: id description: "Unique plan identifier (VARCHAR)" - name: created_at description: "When the plan was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: level description: "Plan level (NUMBER)" - name: name description: "Plan name (VARCHAR)" - name: monthly_price_usd description: "Monthly price in USD (NUMBER)" - name: features description: "Plan features (JSON string, VARCHAR)" - name: stripe_product_id description: "Stripe product ID (VARCHAR)" - name: stripe_annual_price_id description: "Stripe annual price ID (VARCHAR)" - name: stripe_monthly_price_id description: "Stripe monthly price ID (VARCHAR)" - name: daily_credits description: "Daily credit limit (NUMBER)" - name: monthly_credits description: "Monthly credit limit (NUMBER)" - name: annual_price_usd description: "Annual price in USD (NUMBER)" - name: plan_key description: "Plan key identifier (VARCHAR)" - name: rds_bot_jail description: "Bot jail records from RDS" tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_bot_jail"] - name: rds_clip_comment description: | Clip comment data from RDS containing user comments and interactions on audio clips. This table stores comment content, metadata, engagement metrics, and hierarchical comment structures (replies to comments). Data is synchronized from PostgreSQL RDS to Snowflake for analytics and content moderation purposes. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_clip_comment"] columns: - name: id description: "Unique comment identifier (primary key)" - name: created_at description: "Timestamp when comment was created (TIMESTAMP_TZ)" - name: updated_at description: "Timestamp when comment was last updated (TIMESTAMP_TZ)" - name: content description: "Comment text content" - name: track_time_second description: "Time in seconds within the clip where comment was made (FLOAT)" - name: is_deleted description: "Boolean indicating if comment has been deleted" - name: deleted_by description: "User ID who deleted the comment" - name: num_likes description: "Number of likes the comment has received" - name: clip_id description: "ID of the clip this comment belongs to" - name: parent_id description: "ID of parent comment if this is a reply (NULL for top-level comments)" - name: user_id description: "ID of user who wrote the comment" - name: num_replies description: "Number of replies to this comment" - name: p_date description: "Partition date for table optimization" - name: rds_profile_follow description: | Profile follow data from RDS containing user follow relationships and social connections. This table stores follow relationships between users including who follows whom and when the follow relationship was established. Data is synchronized from PostgreSQL RDS to Snowflake for social analytics and community engagement analysis. tags: ['raw', 'postgres', 'partition_type=daily'] meta: dagster: asset_key: ["rds_profile_follow"] columns: - name: id description: "Unique follow relationship identifier (VARCHAR)" - name: followed_profile_id description: "ID of the profile being followed (NUMBER)" - name: following_profile_id description: "ID of the profile doing the following (NUMBER)" - name: created_at description: "When the follow relationship was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: p_date description: "Partition date for table optimization (DATE)" - name: rds_grouped_user_notification description: "Grouped user notification data from RDS" tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_grouped_user_notification"] - name: rds_playlist_clip description: | Playlist clip data from RDS containing associations between playlists and audio clips. This table stores the relationship between playlists and clips including ordering, creation session, and update information. Data is synchronized from PostgreSQL RDS to Snowflake for playlist analytics and content organization analysis. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_playlist_clip"] columns: - name: id description: "Unique identifier (NUMBER)" - name: relative_index description: "Position in playlist (ordering, NUMBER)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: clip_id description: "Clip identifier (VARCHAR)" - name: playlist_id description: "Playlist identifier (VARCHAR)" - name: updated_by_id description: "User who updated the playlist (NUMBER)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (NUMBER)" - name: created_session_id description: "Session that created this entry (VARCHAR)" - name: rds_playlist description: | Playlist data from RDS containing user-created playlists and their metadata. This table stores playlist information including name, description, privacy settings, engagement metrics, and user ownership. Data is synchronized from PostgreSQL RDS to Snowflake for playlist analytics and content discovery analysis. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_playlist"] columns: - name: id description: "Unique playlist identifier (VARCHAR)" - name: created_at description: "When the playlist was created (VARCHAR)" - name: updated_at description: "Last update timestamp (VARCHAR)" - name: name description: "Playlist name (VARCHAR)" - name: description description: "Playlist description (VARCHAR)" - name: user_id description: "User who created the playlist (VARCHAR)" - name: is_public description: "Whether the playlist is public (BOOLEAN)" - name: is_discover_playlist description: "Whether this is a discover playlist (BOOLEAN)" - name: is_trashed description: "Whether the playlist is trashed (BOOLEAN)" - name: dislike_count description: "Number of dislikes (NUMBER)" - name: flag_count description: "Number of flags (NUMBER)" - name: is_public_approved description: "Whether the playlist is approved for public display (BOOLEAN)" - name: play_count description: "Number of plays (NUMBER)" - name: skip_count description: "Number of skips (NUMBER)" - name: upvote_count description: "Number of upvotes (NUMBER)" - name: p_date description: "Partition date for table optimization (VARCHAR)" - name: created_session_id description: "Session that created the playlist (VARCHAR)" - name: rds_video_hook description: | Video hook data from RDS containing video content created from audio clips. This table stores video hook information including rendering status, S3 storage identifiers, user ownership, content settings, and video specifications. Data is synchronized from PostgreSQL RDS to Snowflake for video analytics and content management analysis. tags: ['raw', 'postgres'] columns: - name: id description: "Unique video hook identifier (VARCHAR)" - name: created_at description: "When the video hook was created (TIMESTAMP_TZ)" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ)" - name: original_clip description: "Original clip identifier (VARCHAR)" - name: status description: "Status of video processing (pending, processing, completed, failed, TEXT)" - name: title description: "Video hook title (TEXT)" - name: start_clip_timestamp description: "Start time of clip in seconds (FLOAT)" - name: end_clip_timestamp description: "End time of clip in seconds (FLOAT)" - name: rendered_video_s3_id description: "S3 identifier for rendered video (TEXT)" - name: rendered_video_preview_s3_id description: "S3 identifier for video preview (TEXT)" - name: thumbnail_image_s3_id description: "S3 identifier for thumbnail image (TEXT)" - name: raw_video_upload_id description: "Raw video upload identifier (VARCHAR)" - name: user_id description: "User who created the video hook (INTEGER)" - name: is_test description: "Whether this is a test video hook (BOOLEAN)" - name: allow_comments description: "Whether comments are allowed (BOOLEAN)" - name: caption description: "Video caption/description (TEXT)" - name: show_lyrics description: "Whether to display lyrics (BOOLEAN)" - name: content_rating description: "Content rating (G, PG, R, etc., TEXT)" - name: content_rating_tags description: "Array of content rating tags (ARRAY)" - name: video_duration description: "Duration of video in seconds (FLOAT)" - name: video_streaming_resolutions description: "Available streaming resolutions (ARRAY)" - name: is_deleted description: "Soft delete flag (BOOLEAN)" - name: rendered_watermarked_video_s3_id description: "S3 identifier for watermarked version (TEXT)" - name: staff_review_status description: "Staff review status code (INTEGER)" - name: video_upload_ids description: "Array of video upload identifiers (ARRAY)" - name: creation_source description: "Source of creation (web, mobile, api, etc., TEXT)" - name: metadata_id description: "Metadata identifier (VARCHAR)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (NUMBER)" meta: dagster: asset_key: ['rds_video_hook'] - name: rds_studio_project description: | Studio project data from RDS containing user-created Studio projects and their metadata. This table stores project information including title, creation details, user ownership, and archive status. Data is synchronized from PostgreSQL RDS to Snowflake for Studio analytics and project management analysis. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_studio_project"] columns: - name: id description: "Unique studio project identifier (VARCHAR(255), PRIMARY KEY)" - name: created_at description: "When the project was created (TIMESTAMP_TZ(9))" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ(9))" - name: title description: "Project title/name (VARCHAR(255))" - name: user_id description: "User who owns the project (NUMBER(38,0))" - name: archived description: "Whether the project is archived (BOOLEAN)" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (NUMBER(38,0))" - name: rds_bots_studio_project_clip description: | Studio project clip relationship data from RDS containing associations between Studio projects and audio clips. This table stores the relationship between Studio projects and clips including creation timestamps. Data is synchronized from PostgreSQL RDS to Snowflake for Studio analytics and project content analysis. tags: ['raw', 'postgres', 'partition_type=hourly'] meta: dagster: asset_key: ["rds_studio_project_clip"] columns: - name: id description: "Unique clip-project relationship identifier (NUMBER(38,0), PRIMARY KEY)" - name: clip_id description: "ID of the clip (VARCHAR(255))" - name: studio_project_id description: "ID of the studio project (VARCHAR(255))" - name: created_at description: "When the relationship was created (TIMESTAMP_TZ(9))" - name: updated_at description: "Last update timestamp (TIMESTAMP_TZ(9))" - name: p_date description: "Partition date for table optimization (DATE)" - name: p_hour description: "Partition hour for table optimization (NUMBER(38,0))"