PYTHON-6167 Split the test-non-standard task family into per-suite task families by blink1073 · Pull Request #3131 · mongodb/mongo-python-driver

blink1073 · 2026-10-16T10:43:41Z

Summary

Splits the shared test-non-standard Evergreen task family into one family per suite, so a task's name and tags identify the suite it runs. The encryption and load balancer matrices now run as separate sync and async tasks.

Before: 1325 tasks across 76 build variants
After: 833 tasks across 65 build variants

Also ensures a good cross-section of tests in PRs for coverage.

Before: 97 tasks, Time Taken: 21h 50m
After: 60 tasks, Time Taken: 12h 36m

Motivation

The test-non-standard family was shared by the encryption, load balancer, doctest, and PyOpenSSL suites, and only the build variant decided which suite a task actually ran. That made failures hard to attribute.

Additionally, we were not testing sync and async separately for encryption or load balancer, or any of the other non-standard tests.

Changes

Replaces test-non-standard with one family per suite; a task's name identifies its suite, and distro tags decide which variant runs it.
Ensure that sync and async tasks run the appropriate tests for all task families
Builds OpenSSL 3 on RHEL 8 so free-threaded PyOpenSSL cells can run there, using the updated drivers-evergreen-tools submodule.

Testing

python3 -m pytest .evergreen/scripts/test_generate_config.py passes, covering the new suite-identity, sync/async-split, and matrix-size invariants.
New tests pass in Evergreen: XXXX Patch URL

Checklist for Author

Did you update the changelog (if necessary)? (CI-only change.)
Is there test coverage?
Is any followup work tracked in a JIRA ticket? If so, add link(s). N/A

Checklist for Reviewer

Does the title of the PR reference a JIRA Ticket?
Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
Is all relevant documentation (README or docstring) updated?

…sk families

Copilot

🟡 Changes recommended

Unresolved critical and moderate issues affect shell safety and test correctness.

2 open findings

What changed in this PR

Splits the shared Evergreen non-standard test family into suite-specific families with sync/async selection and updated platform matrices.

Changes:

Generates per-suite task families and selectors.
Adds sync/async filtering and OpenSSL 3 setup.
Regenerates Evergreen variants, tasks, and functions.

File	Summary
`.evergreen/scripts/utils.py`	Adds `--sync`; moderate issue (1 vote) with positional argument handling.
`.evergreen/scripts/test_generate_config.py`	Adds matrix and suite-shape assertions.
`.evergreen/scripts/setup_tests.py`	Adds sync filtering and OpenSSL setup; critical shell-injection issue (2 votes).
`.evergreen/scripts/generate_config.py`	Generates suite-specific tasks; moderate issues (2, 1, and 1 votes) affecting load-balancer splitting, disabled test commands, and compressor propagation.
`.evergreen/generated_configs/variants.yml`	Updates build-variant selectors.
`.evergreen/generated_configs/tasks.yml`	Regenerates suite-specific task matrices.
`.evergreen/generated_configs/functions.yml`	Forwards new task environment variables.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Unresolved critical and moderate findings affect test coverage and task configuration.

2 open findings

2 resolved since last review

🧠 Review effort: Lite

…o-less names across all platforms

Copilot

🔵 Needs a closer look

Three moderate review findings remain unresolved in task generation and configuration.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Allowlist TEST_CRYPT_SHARED for crypt_shared test tasks

.evergreen/scripts/generate_config.py:1615

The crypt_shared tasks now put TEST_CRYPT_SHARED in the run tests command vars, but the environment allowlist still omits it. setup_tests.py reads this value to enable the crypt_shared setup, so the new crypt_shared tasks will run without that configuration and test the regular libmongocrypt path instead. Add TEST_CRYPT_SHARED to this allowlist.

2 resolved since last review

_{Review effort: Lite}

Copilot

🟡 Changes recommended

Address the critical environment-value shell-injection issue and preserve TEST_CRYPT_SHARED for encryption tests.

2 open issues

Shell-escape values written to env.sh · New
Persist TEST_CRYPT_SHARED for encryption tests · New

_{Review effort: Lite}

Copilot

🟡 Changes recommended

Unresolved critical and moderate findings affect shell safety, option parsing, test coverage, and compression execution.

4 open issues

_{Review effort: Lite}

…THON-6167 # Conflicts: # .evergreen/generated_configs/functions.yml # .evergreen/scripts/generate_config.py # .evergreen/scripts/setup_tests.py

…v values

Copilot

🔵 Needs a closer look

The broad Evergreen matrix and platform-specific OpenSSL/environment changes depend on an unavailable submodule and placeholder patch, while static generator tests cannot verify cross-platform task selection or end-to-end execution, requiring human Evergreen validation.

2 open issues

2 resolved since last review

_{Review effort: Lite}

Copilot

🔵 Needs a closer look

Human assessment is needed because the configuration change may remove the baseline Linux standard test matrix, creating a potentially critical gap in the generated build variants.

No open issues 🎉

2 resolved since last review

_{Review effort: Lite}

…erator tests to sanity checks, tighten prose

…THON-6167

…THON-6167 # Conflicts: # .evergreen/scripts/test_generate_config.py

# Conflicts: # .evergreen/scripts/run_tests.py # .evergreen/scripts/test_generate_config.py

Copilot

🟡 Changes recommended

Active moderate and critical findings remain: the rhel8 OCSP variant is not included in PR builds, and default-family tasks no longer filter to the default test suite.

2 open issues

Preserve default-suite marker filtering · New
Restore PR-tagged rhel8 OCSP variant · New

_{Review effort: Lite}

blink1073 added 2 commits October 16, 2026 05:42

PYTHON-6167 Split the test-non-standard task family into per-suite ta…

53cfac2

…sk families

Address review: consolidate per-suite task families and share task names

e2edd5a

blink1073 requested a lite review from Copilot October 16, 2026 16:51

Copilot started reviewing on behalf of blink1073 October 16, 2026 16:52

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread .evergreen/scripts/setup_tests.py Resolved

Comment thread .evergreen/scripts/generate_config.py Resolved

blink1073 added 2 commits October 16, 2026 13:03

Address review: consolidate per-suite task families and share task names

e975c96

trim tests

ebd77de

blink1073 requested a lite review from Copilot October 16, 2026 18:41

Copilot started reviewing on behalf of blink1073 October 16, 2026 18:42

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread .evergreen/scripts/generate_config.py Resolved

Comment thread .evergreen/scripts/utils.py Resolved

Address review: per-version per-suite task families with shared distr…

65326c9

…o-less names across all platforms

blink1073 requested a lite review from Copilot October 16, 2026 22:21

Copilot started reviewing on behalf of blink1073 October 16, 2026 22:22

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Address review: per-version per-suite task families with shared names

18ec45e

blink1073 requested a lite review from Copilot October 16, 2026 22:41

Copilot started reviewing on behalf of blink1073 October 16, 2026 22:41

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread .evergreen/scripts/setup_tests.py Resolved

Comment thread .evergreen/scripts/generate_config.py Resolved

Address review: shell-escape env values and persist TEST_CRYPT_SHARED

6008d53

blink1073 requested a lite review from Copilot October 17, 2026 01:18

Copilot started reviewing on behalf of blink1073 October 17, 2026 01:19

View session

Copilot AI reviewed Oct 17, 2026

View reviewed changes

Comment thread .evergreen/scripts/setup_tests.py Resolved

Comment thread .evergreen/scripts/utils.py Resolved

blink1073 added 3 commits November 3, 2026 06:05

Merge branch 'main' of github.com:mongodb/mongo-python-driver into PY…

23dff52

…THON-6167 # Conflicts: # .evergreen/generated_configs/functions.yml # .evergreen/scripts/generate_config.py # .evergreen/scripts/setup_tests.py

PYTHON-6167 Escape backslashes first when shell-escaping persisted en…

c730df1

…v values

trim prose

4cc35d7

blink1073 requested a lite review from Copilot November 3, 2026 12:22

Copilot started reviewing on behalf of blink1073 November 3, 2026 12:23

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

blink1073 added 2 commits November 3, 2026 06:42

Pin ambient LD_LIBRARY_PATH in the crypt_shared escape test

0b9e787

Unescape bash escapes in read_env to fix SASL PLAIN auth source

2e5994a

blink1073 requested a lite review from Copilot November 3, 2026 12:51

Copilot started reviewing on behalf of blink1073 November 3, 2026 12:52

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

blink1073 added 4 commits November 6, 2026 06:50

PYTHON-6167 Split per-suite families into suite_families.py, trim gen…

6caf44a

…erator tests to sanity checks, tighten prose

add suite families file

0f7af72

rename the file

71313d3

Merge branch 'main' of github.com:mongodb/mongo-python-driver into PY…

71993e7

…THON-6167

blink1073 force-pushed the PYTHON-6167 branch from 3e48983 to 71993e7 Compare November 10, 2026 12:14

blink1073 added 3 commits November 10, 2026 06:35

Run the auth-aws and numpy pins on Linux only, combine the OCSP variants

1e9c7d5

Run encryption on macOS and Windows in PRs

57fb036

Merge branch 'main' of github.com:mongodb/mongo-python-driver into PY…

fee73ed

…THON-6167 # Conflicts: # .evergreen/scripts/test_generate_config.py

aclark4life mentioned this pull request Nov 11, 2026

PYTHON-6263 Remove the blacken-docs pre-commit hook #3214

Merged

7 tasks

Merge remote-tracking branch 'upstream/main' into PYTHON-6167

30d25c2

# Conflicts: # .evergreen/scripts/run_tests.py # .evergreen/scripts/test_generate_config.py

blink1073 requested a lite review from Copilot November 13, 2026 14:20

Copilot started reviewing on behalf of blink1073 November 13, 2026 14:20

View session

Copilot AI reviewed Nov 13, 2026

View reviewed changes

Comment thread .evergreen/scripts/task_suite_families.py Resolved

Comment thread .evergreen/scripts/generate_config.py Resolved

blink1073 marked this pull request as ready for review November 13, 2026 15:28

blink1073 requested a review from a team as a code owner November 13, 2026 15:28

blink1073 requested a review from aclark4life November 13, 2026 15:28

Conversation

blink1073 commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Testing

Checklist for Author

Checklist for Reviewer

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

blink1073 commented Oct 16, 2026 •

edited

Loading