Here’s a clean roadmap you can drop into **`PLAN.md`**. --- # mvmaker Roadmap ## ✅ What’s Done (Milestone M0) - **Backend skeleton (FastAPI) running locally** on `:8000`. - **TikTok download (Step 1)** - `POST /api/tiktok_download` → saves MP4 in `backend/storage//`. - `GET /api/raw_video/{job_id}` → fetches the MP4. - **Audio extraction (Step 2)** - `POST /api/tiktok_extract_audio?job_id=...` → creates `audio.wav`. - `GET /api/raw_audio/{job_id}` → fetches the WAV. - Frontend basic page served from FastAPI; ping + render form working. --- ## 🎯 Next Big Goal **Generate a synced video from:** 1. TikTok video transcription (source timing) 2. Song transcription (target timing) 3. Final song audio …using the ramping/vidmaker pipeline. We’ll also integrate **Suno API** to generate that song audio later. --- ## 📌 Milestones ### M1 – Transcribe TikTok Audio (Step 3) **Goal:** Convert the extracted WAV into a word-timestamped JSON via Replicate (`villesau/whisper-timestamped`). **Backend** - Add endpoint: `POST /api/tiktok_transcribe_audio?job_id=...` - Loads `audio_path.txt` - Calls `transcribe_with_replicate(audio.wav)` - Saves `tiktok_transcription.json` - Returns link: `/api/tiktok_json/{job_id}` - Harden error messages (invalid token, rate limits, file too big). **Frontend** - Button “Transcribe TikTok Audio” (step-by-step section) - Show link to download/view transcription JSON. **Acceptance Criteria** - Given a TikTok URL → can click through steps & end up with a valid transcription JSON. - Server returns clean errors if Replicate token missing/invalid. --- ### M2 – Render Using Existing Pipeline **Goal:** Reuse your `vidmaker` flow to create the synced video. **Inputs:** - Video: from Step 1 (`raw_video`) - Video transcription: from M1 (`tiktok_transcription.json`) - Song audio: (placeholder MP3 for now) - Song transcription: (placeholder JSON for now) **Backend** - Keep `/api/render` as is, but allow passing `job_id_video` & direct JSON bodies/paths to skip re-upload. - Option A: Upload 4 files from frontend (simple). - Option B: New endpoint `/api/render_from_jobs` that looks up stored paths. **Frontend** - Section “Render Final Video” with: - Select existing job (dropdown) or paste IDs - Upload / pick song + song transcription - Submit and poll status → show video. **Acceptance Criteria** - Final MP4 downloads & plays with audio in sync. --- ### M3 – Suno API Integration (Generate Song) **Goal:** Generate a song from your audio (or text prompt) using Suno’s API, then feed it into the pipeline. **Backend** - Endpoint `POST /api/suno_generate` (needs Suno API token): - Takes prompt or audio - Returns song file URL/path - Save `song.mp3` to storage and record in `audio_song_path.txt` **Then:** - Transcribe song (same Replicate endpoint as TikTok) - Save `song_transcription.json` **Acceptance Criteria** - End-to-end path: no manual MP3 upload needed; Suno → JSON → render. --- ### M4 – Combine Transcriptions & Ramping **Goal:** Programmatic alignment (you already have `vidmaker.py`). **Backend** - Move `build_combined_sync()` logic into a utility (done). - Endpoint `POST /api/combine_sync`: - Inputs: `video_json`, `song_json` (paths or files) - Outputs: `combined_sync.json` - Endpoint `POST /api/vidmaker`: - Inputs: `clip`, `song`, `combined_sync.json` - Outputs: final video (Alternatively just keep `/api/render` to wrap these steps.) **Acceptance Criteria** - Combined JSON is generated and stored. - Video is produced with continuous ramping, no drift. --- ### M5 – Frontend Polish - Split UI into clear sections: 1. Download TikTok 2. Extract Audio 3. Transcribe (TikTok / Song) 4. Combine & Render - Show job IDs, statuses, links to JSON/MP4/WAV. - Add progress indicators (polling or WebSockets). - Handle errors gracefully (toast messages, red text). --- ### M6 – Productionize - Env management: `.env` (REPLICATE_API_TOKEN, SUNO_API_KEY, etc.) - Persistent object storage (S3/R2) instead of local disk. - Queue/background worker (Redis RQ/Celery) if jobs get heavy. - Auth/rate-limits if public. - Dockerize all (you already have a Dockerfile) and deploy (Coolify, Fly.io, etc.). --- ## 🧩 Task Checklist - [ ] **M1**: `/api/tiktok_transcribe_audio` endpoint - [ ] Implement `transcribe_with_replicate()` (done) & call it - [ ] Save `tiktok_transcription.json`, return link - [ ] Frontend button + poll - [ ] **M2**: Render flow using stored paths or uploads - [ ] Option to choose stored TikTok job_id for video & transcription - [ ] Keep existing `/api/render` for manual uploads - [ ] **M3**: Suno integration (optional, future) - [ ] Add backend endpoint to call Suno - [ ] Save song MP3 + transcribe with Replicate - [ ] Feed into render - [ ] **M4**: Combine sync + ramping internally - [ ] Expose `/api/combine_sync` or inline in render - [ ] Handle bad anchors (ratio filter) automatically - [ ] **M5**: Frontend UX - [ ] Separate sections & step-by-step UI - [ ] Job list / history view - [ ] Error surface (show `error` field) - [ ] **M6**: Ops / Deployment - [ ] Use S3 instead of local disk - [ ] Add auth (API keys / login) - [ ] Logging & monitoring (Sentry etc.) --- ## 🗂 File/Endpoint Reference | Step | What | Endpoint / Script | Output | | ---- | ---------------- | -------------------------------------------- | ---------------------------------------------------- | | 1 | Download TikTok | `POST /api/tiktok_download` | `video_path.txt`, `/api/raw_video/{id}` | | 2 | Extract Audio | `POST /api/tiktok_extract_audio` | `audio_path.txt`, `/api/raw_audio/{id}` | | 3 | Transcribe Audio | `POST /api/tiktok_transcribe_audio` (to add) | `tiktok_transcription.json`, `/api/tiktok_json/{id}` | | 4 | Render Final | `POST /api/render` or new combined endpoint | `output.mp4`, `/api/download/{id}` | | 5 | Combine Sync | internal util + `vidmaker.py` logic | `combined_sync.json` | --- ## 🔐 Env Vars ``` REPLICATE_API_TOKEN=... WHISPER_MODEL=villesau/whisper-timestamped:c5b122b7e513b1b5a6ef849891c538869b77cc932cbd0f8203e11d3b357553b8 SUNO_API_KEY=... # later ``` --- When you’re ready, I’ll implement **Step 3 endpoint** and wire it into the UI.