# CI Benchmark Tool

The **CI Benchmark tool** allows you to measure impact of pull requests on runtime performance of GitHub Actions workflows - it's useful for detecting build speed regressions in repos, where CI performance is an important consideration (ex: mobile).

The benchmark tool is intended to be used together with [Slash Command Dispatch](https://github.com/peter-evans/slash-command-dispatch), which allows you to trigger on demand by commenting `/benchmark` on any PR in your repo.

Once dispatched, the tool will runs the specified workflow multiple times on both the base branch and PR branch, waits for them to complete, and posts statistical analysis of performance differences back on the PR via a comment:

## Setup

- Update all workflows in your repository that you would like to benchmark and include a `workflow_dispatch` trigger with `concurrency_key` input, which is applied to the concurrency group:

  ```yaml
  on:
    workflow_dispatch:
      inputs:
        concurrency_key:
          description: "Additional key to use for concurrency"
          required: false
          type: string

  concurrency:
    group: ${{ github.repository }}-${{ github.workflow }}-${{ github.ref }}-${{ inputs.concurrency_key || 'default' }}
    cancel-in-progress: true
  ```

- Create a new Github Actions workflow file in `.github/workflows/commands-processor.yml`, and replace `ci.yml` with the name of the worflow that you'd like to benchmark by default. For more info on slash commands, see [peter-evans/slash-command-dispatch](https://github.com/peter-evans/slash-command-dispatch).

  > **Note**: Secrets `SUNO_CI_APP_ID` and `SUNO_CI_APP_PRIVATE_KEY` are available by default for all repos in the `suno-ai` org.

  ```yaml
  on:
    issue_comment:
      types: [created]

  env:
    DEFAULT_WORKFLOW: "ci.yml"
    DEFAULT_RUNS: "5"
    DEFAULT_TIMEOUT: "30"

  jobs:
    command-dispatch:
      if: ${{ github.event_name == 'issue_comment' }}
      runs-on: ubuntu-latest
      steps:
        - name: Generate a token
          id: suno-ci
          uses: actions/create-github-app-token@v2
          with:
            app-id: ${{ secrets.SUNO_CI_APP_ID }}
            private-key: ${{ secrets.SUNO_CI_APP_PRIVATE_KEY }}

        - name: Dispatch slash commands
          uses: peter-evans/slash-command-dispatch@v4
          id: slash-command-dispatch
          with:
            token: ${{ steps.suno-ci.outputs.token }}
            reaction-token: ${{ steps.suno-ci.outputs.token }}
            commands: ""
            config: >
              [
                {
                  "command": "benchmark",
                  "permission": "write",
                  "issue_type": "pull-request",
                  "dispatch_type": "workflow",
                  "repository": "suno-ai/ci",
                  "static_args": [
                    "repository=${{ github.repository }}",
                    "comment-id=${{ github.event.comment.id }}",
                    "pr-number=${{ github.event.issue.number }}",
                    "workflow=${{ github.event.client_payload.slash_command.args.named.workflow || env.DEFAULT_WORKFLOW }}",
                    "runs=${{ github.event.client_payload.slash_command.args.named.runs || env.DEFAULT_RUNS }}",
                    "timeout=${{ github.event.client_payload.slash_command.args.named.timeout || env.DEFAULT_TIMEOUT }}",
                    "workflow_base=${{ github.event.client_payload.slash_command.args.named.workflow_base }}"
                  ]
                }
              ]

        - name: Edit comment with error message
          if: steps.slash-command-dispatch.outputs.error-message
          uses: peter-evans/create-or-update-comment@v4
          with:
            comment-id: ${{ github.event.comment.id }}
            body: |
              > ${{ steps.slash-command-dispatch.outputs.error-message }}
            token: ${{ steps.suno-ci.outputs.token }}
  ```

- Merge the changes.

- Post a new comment on any PR in your repository to kick off the benchmark on that PR:

```
/benchmark [workflow=...] [runs=<>] [timeout=<minutes>] [workflow_base=...]
```

### Command Parameters

All parameters are optional and the defaults are specified in the `env` section of the `commands-processor.yml` file:

- **workflow**: The workflow to run.
- **runs**: Number of runs to perform.
- **timeout**: Timeout for each run in minutes.
- **workflow_base**: The workflow to run for the base branch in case the name of the workflow changed in the PR being benchmarked.