Run tests in CI

Run Revyl tests automatically on every pull request. This guide covers GitHub Actions, the Revyl CLI in CI, and integrating test results into your workflow.

Set up with an agent

Paste one prompt and your coding agent does the rest.

Setup prompt

Run this project's Revyl tests in CI on every pull request.

Read https://docs.revyl.com/tests/ci.md and follow it. It's the current Revyl guide for this, so prefer it over anything you remember about Revyl.

On GitHub, use GitHub Actions: put the critical tests in one `smoke-tests` workflow and run `revyl workflow run smoke-tests` on each pull request. Store `REVYL_API_KEY` as a CI secret; I'll add it myself.

You're done when a pull request runs the Revyl tests and shows their result.

Ask me before anything that needs my login or approval.

Prerequisites

Revyl account with an API key
App and tests already created (see Your First Test)

Step 1: Store your API key

Add REVYL_API_KEY as a repository secret in GitHub:

Settings → Secrets and variables → Actions → New repository secret

Name: REVYL_API_KEY Value: your API key from Account → API Keys

Run your workflow on every pull request with its saved choices. Tests saved on Default branch run on the pull request's source branch when it has a usable build, otherwise on the app's default branch. Tests created in the dashboard start on All branches · Latest build instead; see New saved choices, or use Option C.

name: Revyl Tests

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Revyl CLI
        run: |
          curl -fsSL https://revyl.com/install.sh | sh
          echo "$HOME/.revyl/bin" >> "$GITHUB_PATH"

      - name: Run smoke tests
        env:
          REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
        run: revyl workflow run smoke-tests

The installer downloads the native CLI binary. The CLI exits with code 0 on pass, 1 on failure.

Option B: Build, upload, and test

Build your app fresh, upload the artifact, and run the full suite against it:

name: Build and Test

on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Revyl CLI
        run: |
          curl -fsSL https://revyl.com/install.sh | sh
          echo "$HOME/.revyl/bin" >> "$GITHUB_PATH"

      - name: Build, upload, and run workflow
        env:
          REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
        run: revyl workflow run smoke-tests --build --platform android

revyl workflow run <workflow> --build builds your app on the runner with the build settings in .revyl/config.yaml, uploads the artifact to the pull request's source branch, and runs the workflow on that exact build. --platform chooses the platform to build; tests for other platforms keep their saved choices. Without --build, the workflow runs its saved choices, as in Option A.

Option C: Test an artifact from your own build job

After your CI build succeeds, upload its artifact and pass the returned build ID to the workflow, so the run tests that exact build. The upload goes to the pull request's source branch; see Choose where CI uploads go.

- name: Install Revyl CLI
  run: |
    curl -fsSL https://revyl.com/install.sh | sh
    echo "$HOME/.revyl/bin" >> "$GITHUB_PATH"

- name: Upload Android build
  id: upload
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
    REVYL_ANDROID_APP_ID: ${{ vars.REVYL_ANDROID_APP_ID }}
  run: |
    revyl build upload --file ./build/app.apk \
      --app "$REVYL_ANDROID_APP_ID" --platform android \
      --yes --json > build.json
    BUILD_ID=$(jq -er 'select(.success == true and .count == 1) | .build.build_id | select(type == "string" and length > 0)' build.json)
    echo "build_id=$BUILD_ID" >> "$GITHUB_OUTPUT"

- name: Run workflow on the uploaded build
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
    WORKFLOW_ID: ${{ vars.WORKFLOW_ID }}
    REVYL_ANDROID_APP_ID: ${{ vars.REVYL_ANDROID_APP_ID }}
    REVYL_BUILD_ID: ${{ steps.upload.outputs.build_id }}
  run: revyl workflow run "$WORKFLOW_ID" --android-app "$REVYL_ANDROID_APP_ID" --android-build-id "$REVYL_BUILD_ID"

For one test, use revyl test run "$TEST_ID" --build-id "$REVYL_BUILD_ID". Use an existing test for this app/platform. Runtime app overrides replace the workflow’s saved app configuration; unspecified platforms use each test’s saved choice. An exact Android override does not select an iOS build.

Option D: Use the Revyl GitHub Actions

Upload with upload-build, then run that build with run-test:

- uses: RevylAI/revyl-gh-action/upload-build@main
  id: upload
  with:
    app-id: ${{ vars.REVYL_APP_ID }}
    version: ${{ github.event.pull_request.head.sha || github.sha }}
    file-path: ./app.apk
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
- uses: RevylAI/revyl-gh-action/run-test@main
  with:
    test-id: ${{ vars.REVYL_TEST_ID }}
    build-id: ${{ steps.upload.outputs.build-id }}
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}

upload-build uploads to the pull request's source branch unless you set branch. run-test and run-workflow follow the same branch rules as the CLI. See the Revyl GitHub Actions README for every input.

Step 3: Sync tests from CI

Keep remote tests in sync with your repo after merge to main:

- name: Sync tests
  if: github.event_name == 'push' && github.ref == 'refs/heads/main'
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
  run: revyl test push --force

CLI in CI (without GitHub Action)

Install and run the CLI directly in any CI:

curl -fsSL https://revyl.com/install.sh | sh
export PATH="$HOME/.revyl/bin:$PATH"
export REVYL_API_KEY=${{ secrets.REVYL_API_KEY }}

revyl test run login-flow --no-wait --json    # Queue and exit immediately
revyl workflow run smoke-tests --no-wait      # Queue a workflow

Build upload from CI

revyl build upload --file build/App.app.zip --platform ios --yes    # Upload pre-built artifact
revyl build --profile release --platform android --json            # Build one recipe in the cloud

See CI Builds for setup details.

Useful CI Patterns

Retries

revyl workflow run smoke-tests --retries 2

Async execution

revyl workflow run regression --no-wait

JSON output for downstream processing

- name: Run with JSON output
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
  run: |
    result=$(revyl test run login-flow --json)
    echo "Report: $(echo $result | jq -r '.report_link')"

Version-tagged builds

- name: Build tagged with the commit
  env:
    REVYL_API_KEY: ${{ secrets.REVYL_API_KEY }}
    COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
  run: revyl build --profile release --platform android --version "$COMMIT_SHA"

Pass both selectors in CI. No profile is stored as active/default, and a non-interactive build fails with the available choices if selection remains ambiguous.

Keep main or staging current

Pull request builds don't refresh main or staging after merge. To keep a branch as your source of truth, upload each new build to it and run your tests on that build, explore it, or both, after every merge or on a schedule. See Keep a source-of-truth branch current.

GitLab CI

test:
  script:
    - curl -fsSL https://revyl.com/install.sh | sh
    - export PATH="$HOME/.revyl/bin:$PATH"
    - revyl workflow run smoke-tests
  variables:
    REVYL_API_KEY: $REVYL_API_KEY

To test a merge request's own build on GitLab, see Choose where CI uploads go.

Buildkite

steps:
  - label: "Revyl tests"
    command: |
      curl -fsSL https://revyl.com/install.sh | sh
      export PATH="$$HOME/.revyl/bin:$$PATH"
      revyl build upload --file ./app.apk --app "$$REVYL_ANDROID_APP_ID" \
        --platform android --yes --json > build.json
      BUILD_ID=$$(jq -er '.build.build_id' build.json)
      revyl workflow run smoke-tests --android-app "$$REVYL_ANDROID_APP_ID" \
        --android-build-id "$$BUILD_ID"

Provide REVYL_API_KEY and REVYL_ANDROID_APP_ID as Buildkite secrets or environment variables. $$ defers each variable to run time.

CI-Friendly Flags

See CI-Friendly Flags.

Exit Codes

Code	Meaning
`0`	Test/workflow passed
`1`	Test/workflow failed or error

Environment Variables

Variable	Description
`REVYL_API_KEY`	API key for authentication (required in CI)

Prerequisites

Step 1: Store your API key

Step 2: Add a workflow file

Option A: Run tests with saved choices

Option B: Build, upload, and test

Option C: Test an artifact from your own build job

Option D: Use the Revyl GitHub Actions

Step 3: Sync tests from CI

CLI in CI (without GitHub Action)

Build upload from CI

Useful CI Patterns

Retries

Async execution

JSON output for downstream processing

Version-tagged builds

Keep main or staging current

GitLab CI

Buildkite

CI-Friendly Flags

Exit Codes

Environment Variables

On this page

PrerequisitesCopy Anchor Link

Step 1: Store your API keyCopy Anchor Link

Step 2: Add a workflow fileCopy Anchor Link

Option A: Run tests with saved choicesCopy Anchor Link

Option B: Build, upload, and testCopy Anchor Link

Option C: Test an artifact from your own build jobCopy Anchor Link

Option D: Use the Revyl GitHub ActionsCopy Anchor Link

Step 3: Sync tests from CICopy Anchor Link

CLI in CI (without GitHub Action)Copy Anchor Link

Build upload from CICopy Anchor Link

Useful CI PatternsCopy Anchor Link

RetriesCopy Anchor Link

Async executionCopy Anchor Link

JSON output for downstream processingCopy Anchor Link

Version-tagged buildsCopy Anchor Link

Keep main or staging currentCopy Anchor Link

GitLab CICopy Anchor Link

BuildkiteCopy Anchor Link

CI-Friendly FlagsCopy Anchor Link

Exit CodesCopy Anchor Link

Environment VariablesCopy Anchor Link

On this page

Prerequisites

Step 1: Store your API key

Step 2: Add a workflow file

Option A: Run tests with saved choices

Option B: Build, upload, and test

Option C: Test an artifact from your own build job

Option D: Use the Revyl GitHub Actions

Step 3: Sync tests from CI

CLI in CI (without GitHub Action)

Build upload from CI

Useful CI Patterns

Retries

Async execution

JSON output for downstream processing

Version-tagged builds

Keep main or staging current

GitLab CI

Buildkite

CI-Friendly Flags

Exit Codes

Environment Variables