Skip to content

test-companion: trim comments to what the code can't say #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
francisf merged 1 commit into from
Nov 11, 2026
Merged

test-companion: trim comments to what the code can't say #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 4 additions & 18 deletions .github/workflows/test-companion-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,5 @@
# Runs the Test Companion Action's own unit tests.
#
# The Action runs on customers' machines with their credentials, so its tests
# have to run on every change to it.
#
# The `paths` filter is safe only while this check is NOT required. A required
# check with a paths filter never reports on a pull request that misses the
# filter, and then blocks that pull request from merging for ever. If this is
# ever made required, remove the filter.
# If this check is ever made required, remove the `paths` filter: a required
# check that never reports blocks the pull request.
name: Test Companion tests

on:
Expand All @@ -23,8 +16,6 @@ on:
permissions:
contents: read

# One run per pull request or branch. A newer push makes the older run's
# result irrelevant, so it is cancelled rather than left to finish.
concurrency:
group: test-companion-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Expand All @@ -37,19 +28,14 @@ jobs:
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
# The tests need no git credentials, so none are left on disk.
persist-credentials: false

# The Action has no dependencies: Node's standard library plus `gh`.
# There is nothing to install.
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: "22"

# The exit code alone is not enough. When a test file throws while it
# loads (a missing import, say), `node --test` prints "fail 0" and
# exits 0, even though none of that file's tests ran. The only sign
# is a `not ok` line, so fail on that as well.
# `node --test` exits 0 when a test file fails to load; only a
# `not ok` line shows it.
- name: Run the unit tests
working-directory: test-companion
run: |
Expand Down
2 changes: 0 additions & 2 deletions CODEOWNERS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
* @browserstack/automate-public-repos

# The Test Companion action. Changes to it run an AI agent in customers' jobs,
# so they also need a review from its team.
/test-companion/ @browserstack/automate-public-repos @browserstack/test-companion
/.github/workflows/test-companion-tests.yml @browserstack/automate-public-repos @browserstack/test-companion
100 changes: 41 additions & 59 deletions test-companion/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,42 +12,33 @@ comes back as one comment and one check.
2. Copy [`starter-workflow.yml`](starter-workflow.yml) to
`.github/workflows/test-companion.yml` in your repository, as-is.

What runs is decided by `testcompanion-pipeline.yml` in your repository, not by
the workflow file.

**Runner requirements:** a Linux runner (GitHub's `ubuntu-latest` works as is)
with Node 20 or later, the `gh` command installed system-wide (in `/usr/bin` or
`/usr/local/bin`), and password-free `sudo` for the job.

`sudo` is what keeps your tokens away from the AI agent. Test Companion runs
the agent as a separate user with no admin rights, which cannot read this
job's tokens, secrets or temp files. Its `gh` can only read from GitHub,
through a relay; every write (saving tests, opening a pull request) is made by
the Action itself after the run. On a runner without password-free `sudo` the
run stops and says why, unless you set `allow-unisolated: true`.

Use the starter file itself rather than writing your own. Besides the triggers,
it carries safeguards that are easy to leave out:

- a filter that skips comments not meant for Test Companion, so they don't start
a job;
- `persist-credentials: false` on checkout, so the workflow token is not left in
`.git/config` where commands the agent runs could read it;
- the concurrency rules that keep one run per pull request, and let closing a
pull request cancel its run.
Which jobs run, on which events and with which limits is set in
`testcompanion-pipeline.yml` in your repository, not in the workflow file.

## Pin the CLI version
**Runner requirements:** Linux (GitHub's `ubuntu-latest` works as is), Node 20
or later, `gh` installed system-wide (`/usr/bin` or `/usr/local/bin`), and
password-free `sudo` for the job.

`sudo` keeps your tokens away from the AI agent. The agent runs as a separate
user with no admin rights that cannot read this job's tokens, secrets or temp
files. Its `gh` can only read from GitHub, through a relay; every write is made
by the Action after the run. Without password-free `sudo` the run stops and
says why, unless you set `allow-unisolated: true`.

Use the starter file rather than writing your own. It includes safeguards that
are easy to miss:

`cli-version` is optional and you should set it once you know which version you
want.
- a filter so comments not meant for Test Companion don't start a job;
- `persist-credentials: false` on checkout, so the workflow token is not left
in `.git/config` where the agent could read it;
- concurrency rules that keep one run per pull request and let closing a pull
request cancel its run.

Without a pin the CLI upgrades itself the first time it runs on a release day.
That spends your pipeline's time on a download, and it means two runs a day
apart are not the same software.
## Pin the CLI version

**Both lines or neither.** A version with no checksum is refused — "refusing an
unverified pinned install" — and the job stops before it reports anything. Take
both from the release you tested with.
Without a pin the CLI upgrades itself on a release day, so two runs a day apart
may not run the same software. Set `cli-version` and `cli-sha256` together,
from the release you tested; a version without a checksum is refused.

```yaml
- uses: browserstack/github-actions/test-companion@v1.1.0
Expand All @@ -64,46 +55,37 @@ both from the release you tested with.
|---|---|---|
| `username` | yes | BrowserStack username, from your secret store |
| `access-key` | yes | BrowserStack access key, from your secret store |
| `cli-version` | no | Pin the CLI. Strongly recommended — see above |
| `cli-version` | no | Pin the CLI. Recommended, see above |
| `cli-sha256` | no | Checksum for the pinned version |
| `build-effect` | no | Whether this step goes red. `fail-on-break` (the default) only when the run itself broke, could not start or hit a limit, not when tests failed. `report` never. `fail` on anything that is not a success, test failures included. The Test Companion check always reports truthfully either way |
| `build-effect` | no | When this step fails. `fail-on-break` (default): the run broke, could not start or hit a limit, not test failures. `report`: never. `fail`: anything but success. The check reports the real outcome either way |
| `workflow` | no | Manual runs only; event-driven runs read the pipeline file |
| `preview` | no | Report what would happen and change nothing |
| `allow-unisolated` | no | `"true"` runs the agent as the job's own user when it cannot be locked down. It can then reach this job's tokens and secrets. Off by default |
| `app-id` / `app-private-key` | no | Use the GitHub App's identity. Keep the workflow's `permissions:` block anyway: the workflow token is still the fallback |
| `allow-unisolated` | no | `"true"` runs the agent as the job's own user when it cannot be locked down, where it can reach this job's tokens and secrets. Off by default |
| `app-id` / `app-private-key` | no | Post as the GitHub App. Keep the workflow's `permissions:` block: the workflow token is the fallback |

## Approving a plan

Before a job changes anything, Test Companion posts its plan and waits for
someone with write access to reply `go`. Read the plan before you reply.

Without the GitHub App, Test Companion posts as `github-actions[bot]`, the same
account every other workflow in your repository uses. It can only recognise its
own plans by that account and a hidden marker. So a different workflow that
copies outside text into a comment could imitate a plan. A reply of `go` only
ever approves a plan for the pull request's current commit. With the App, Test
Companion posts under its own account, which no other workflow can use.
Without the GitHub App, Test Companion posts as `github-actions[bot]`, the
account every workflow in your repository shares, and recognises its own plans
by that account and a hidden marker. Another workflow that copies outside text
into a comment could imitate a plan. A `go` only ever approves a plan for the
pull request's current commit. With the App, Test Companion posts under its own
account, which no other workflow can use.

## How the CLI is installed

Each run downloads the CLI installer from
`test-companion.browserstack.com/cli/install.sh`. The installer script itself
is not checked against a fixed checksum, so a run trusts that address. Pinning
`cli-version` with `cli-sha256` (above) makes the installer refuse any CLI
build other than the one you tested.

## What runs is not decided here

This file is awkward to change once it is in many repositories, so it stays
near-empty on purpose. Which jobs run, on which events, with which spending
limits, lives in `testcompanion-pipeline.yml` in your own repository, which you
can edit freely without touching the workflow above.
Each run downloads the installer from
`test-companion.browserstack.com/cli/install.sh`. The installer itself is not
checked against a fixed checksum, so a run trusts that address. Pinning
`cli-version` with `cli-sha256` makes it refuse any CLI build but the one you
tested.

## Notes for maintainers

This is a **composite** action with **no dependencies** and no `dist/` bundle to
rebuild — unlike the JavaScript actions elsewhere in this repository. The
sources in `src/` are what run.
A composite action with no dependencies and no `dist/` bundle: the sources in
`src/` are what run.

Tests: `cd test-companion && node --test`. They need no network and no
credentials.
Tests: `cd test-companion && node --test`. No network or credentials needed.
Loading
Loading