version: 2 models: - name: int_orpheus__cleaned description: | Staged Orpheus objects data with parsed tool calls and message processing. This model processes raw Orpheus DynamoDB data to extract structured object information, including tool call details, message content, and conversation flow analysis. Objects represent interactions in the Orpheus AI system including user messages, AI responses, and tool function calls. config: group: create tags: [ "partition_type=hourly", "automation_condition=hourly_cron_with_eager_historical_backfill_condition", ] meta: dagster: partition_start_date: "2025-10-06" # Matches ORPHEUS_PROJECT_START_DATE partition_start_hour: 16 end_offset: 0 backfill_policy: max_partitions_per_run: -1 columns: - name: full_id description: Composite unique identifier combining session ID and event type - name: session_id description: Unique identifier for the conversation session - name: user_id description: User ID from dim_user table (internal user ID) - name: user_uid description: User UID from dim_user table (alternative user ID in UUID format) - name: clerk_id description: Clerk ID from the original data (external authentication ID) - name: workspace_id description: Workspace identifier where the conversation took place - name: type description: Event type extracted from the original type field (before the timestamp suffix) - name: name description: Name or label associated with the chat - name: content description: Raw content of the message or event (null if empty) - name: finish_reason description: Reason why the AI response was finished (null for user messages) - name: visibility description: Visibility setting for the event or message - name: tool_call_id_used description: Tool call ID that was used in this event - name: tool_call_id description: Parsed tool call ID from the tool_calls JSON array - name: tool_call_name description: Name of the tool function that was called - name: tool_call_arguments description: JSON object containing the arguments passed to the tool function - name: tool_calls description: Raw tool calls JSON array from the original data - name: tc_sm_message description: Message content from simple_message tool calls - name: tc_genre_example description: Genre example from write_lyrics or generate_image tool calls - name: tc_prompt description: Prompt text from write_lyrics or generate_image tool calls - name: created_at description: Processed creation timestamp with microsecond precision - name: updated_at description: Last update timestamp with microsecond precision - name: p_date description: Date partition for data organization - name: p_hour description: Hour partition for data organization - name: message description: Processed message content, combining content and tool call message fields - name: author description: Author of the message - 'user' for user messages, 'orpheus' for AI responses - name: message_type description: Type of message - 'chat' for regular messages, 'internal' for tool function calls - name: tc_sm_choices description: Choices data from simple_message tool calls - name: tc_sm_current_prompt description: Current prompt data from simple_message tool calls as parsed JSON - name: int_orpheus__messages description: | Processed Orpheus messages with conversation flow analysis and user behavior metrics. This model builds upon int_orpheus__cleaned to add temporal analysis, session detection, and user interaction patterns. It filters for message-type events and calculates timing metrics, session boundaries, and user satisfaction indicators. config: group: create tags: [ "partition_type=hourly", "automation_condition=hourly_cron_with_eager_historical_backfill_condition", ] meta: dagster: partition_start_date: "2025-10-06" # Matches ORPHEUS_PROJECT_START_DATE partition_start_hour: 16 end_offset: 0 backfill_policy: max_partitions_per_run: 24 columns: - name: full_id description: Composite unique identifier combining session ID and event type - name: session_id description: Unique identifier for the conversation session - name: user_id description: User ID from dim_user table (internal user ID) - name: user_uid description: User UID from dim_user table (alternative user ID in UUID format) - name: clerk_id description: Clerk ID from the original data (external authentication ID) - name: type description: Event type (filtered to 'message' only) - name: finish_reason description: Reason why the AI response was finished (null for user messages) - name: message description: Processed message content, combining content and tool call message fields - name: author description: Author of the message - 'user' for user messages, 'orpheus' for AI responses - name: message_type description: Type of message - 'chat' for regular messages, 'internal' for tool function calls - name: tool_call_name description: Name of the tool function that was called - name: tool_call_id_used description: Tool call ID that was used in this event - name: tool_call_id description: Parsed tool call ID from the tool_calls JSON array - name: tc_sm_choices description: Choices data from simple_message tool calls - name: created_at description: Processed creation timestamp with microsecond precision tests: - not_null - name: updated_at description: Last update timestamp with microsecond precision - name: p_date description: Date partition for data organization - name: p_hour description: Hour partition for data organization - name: previous_global_message_time description: Timestamp of the previous user message across all chats for this user - name: previous_chat_message_time description: Timestamp of the previous user message within the current chat session - name: time_between_messages_in_seconds_all_chats description: Seconds elapsed since the user's last message across all their chats - name: time_between_messages_in_seconds_chat description: Seconds elapsed since the user's last message in the current chat - name: time_orpheus_answered_message_in_seconds description: Seconds elapsed between user message and Orpheus chat response - name: time_between_tool_calls_in_seconds description: Seconds elapsed between consecutive tool calls - name: time_diff_any_message_in_seconds description: Seconds elapsed between the last message in the session and the current message - name: is_new_session description: Boolean flag indicating if this message starts a new user session (30+ min gap) - name: session_number description: Sequential session number for the user across all their interactions - name: message_number description: Sequential message number within the current chat session - name: max_message_number_in_chat description: Total number of messages in the current chat session - name: images_url_created description: Array of image URLs created in this message (when message type is 'image') - name: clipids_created description: Array of clip IDs created in this message (when clips are generated) - name: is_from_choices description: Boolean flag indicating if the user message was selected from provided choices - name: is_user_annoyed_heuristic description: Boolean flag indicating potential user frustration based on keyword analysis - name: is_user_last_message_in_chat description: Boolean flag indicating if this is the last user message in the chat session - name: user_session_id description: Unique identifier for the user session (MD5 hash of user_id and session_number) - name: int_orpheus__clips description: | Comprehensive clip metrics and metadata with Orpheus context integration. This model combines clip initial metrics with Orpheus session data and clip metadata to provide a complete view of clips, including engagement metrics, creation context, and whether clips were created through Orpheus AI interactions. It enriches the base clip metrics with Orpheus-specific session information and clip creation details. config: group: create tags: [ "partition_type=hourly", "automation_condition=hourly_cron_with_eager_historical_backfill_condition", ] meta: dagster: partition_start_date: "2025-10-06" # Matches ORPHEUS_PROJECT_START_DATE partition_start_hour: 16 end_offset: -3 # Waits for clip_initial_metrics to be available backfill_policy: max_partitions_per_run: -1 columns: - name: clip_id description: Unique identifier for the clip tests: - not_null - name: user_id description: User ID who created the clip tests: - not_null - name: user_uid description: User UID from dim_user table (alternative user ID in UUID format) - name: clip_created_at description: Timestamp when the clip was created - name: window_minutes description: Time window in minutes for the metrics calculation - name: window_start_time description: Start time of the metrics calculation window - name: window_end_time description: End time of the metrics calculation window - name: play_cnt description: Total number of plays for the clip - name: play_duration_seconds description: Total play duration in seconds - name: n_like_events description: Number of like events for the clip - name: n_undolike_events description: Number of undo like events for the clip - name: net_like_events description: Net like events (likes minus undo likes) - name: n_dislike_events description: Number of dislike events for the clip - name: n_undodislike_events description: Number of undo dislike events for the clip - name: net_dislike_events description: Net dislike events (dislikes minus undo dislikes) - name: n_share_events description: Number of share events for the clip - name: n_download_events description: Number of download events for the clip - name: n_makepublic_events description: Number of make public events for the clip - name: n_makeprivate_events description: Number of make private events for the clip - name: net_public_toggle_events description: Net public toggle events (public minus private) - name: net_positive_interactions description: Net positive interactions (likes minus dislikes) - name: is_satisfactory_indirect description: Boolean flag indicating if clip is satisfactory based on indirect metrics - name: is_satisfactory_direct description: Boolean flag indicating if clip is satisfactory based on direct metrics - name: is_satisfactory description: Boolean flag indicating if clip is satisfactory overall - name: metadata description: Raw metadata JSON from clip_initial_metrics - name: p_date description: Date partition for data organization tests: - not_null - name: p_hour description: Hour partition for data organization - name: credit_cost description: Credit cost for creating the clip (from clip metadata) - name: lang description: Language of the clip (from clip metadata) - name: source description: Source of the clip creation (from clip metadata) - name: clip_type description: Type of the clip (from clip metadata) - name: task_name description: Task name associated with the clip (from clip metadata) - name: user_session_id description: Orpheus user session ID if clip was created through Orpheus - name: create_mode description: Mode of clip creation - 'orpheus' if created through Orpheus, otherwise from clip metadata - name: int_orpheus__tool_calls description: | Dedicated view of Orpheus tool calls with parsed arguments and metadata. This model filters the cleaned Orpheus data to focus specifically on tool call events, providing easy access to tool call details, parsed arguments, and associated metadata. It serves as a specialized view for analyzing tool usage patterns and function call details. config: group: create tags: [ "partition_type=hourly", "automation_condition=hourly_cron_with_eager_historical_backfill_condition", ] meta: dagster: partition_start_date: "2025-10-06" # Matches ORPHEUS_PROJECT_START_DATE partition_start_hour: 16 end_offset: 0 backfill_policy: max_partitions_per_run: 24 columns: - name: tool_call_id description: Unique identifier for the tool call tests: - not_null - name: tool_call_name description: Name of the tool function that was called - name: tool_call_arguments description: JSON object containing the arguments passed to the tool function - name: tc_sm_message description: Message content from simple_message tool calls - name: tc_sm_choices description: Choices data from simple_message tool calls - name: tc_sm_current_prompt description: Current prompt data from simple_message tool calls as parsed JSON - name: tc_genre_example description: Genre example from write_lyrics or generate_image tool calls - name: tc_prompt description: Prompt text from write_lyrics or generate_image tool calls - name: created_at description: Timestamp when the tool call was created tests: - not_null - name: updated_at description: Timestamp when the tool call was last updated - name: p_date description: Date partition for data organization tests: - not_null - name: p_hour description: Hour partition for data organization