Skip to content

feat(pypi): use the uv.lock dependency graph in the hub - #4059

Draft
aignas wants to merge 33 commits into
bazel-contrib:mainfrom
aignas:aignas.chore.alias-src-targets
Draft

aignas wants to merge 33 commits into
bazel-contrib:mainfrom
aignas:aignas.chore.alias-src-targets

Conversation

@aignas

@aignas aignas commented Aug 16, 2026 •

Copy link
Copy Markdown
Collaborator

When a hub is created from a uv.lock file, wire the package dependencies
in the hub from the uv.lock dependency graph instead of the wheel
METADATA.

parse_requirements renders the dependency edges of each uv.lock entry
as PEP 508 requirements, keeping the extras and the markers of the edges.
The optional dependencies of a package are included only for the extras
that its dependents request. The hub_builder collects them per hub and
drops the edges to packages that are not in the hub, e.g. ones only used
on other platforms. The hub_repository passes them to pkg_aliases as
requires_dist, which then creates the pkg and whl targets in the
hub, and the members of a dependency group as group_deps.

The hub keeps using the wheel METADATA for a package when:

  • the hub comes from requirements files, so their output is unchanged;
  • the uv.lock entries of the package that are used on the target
    platforms have different dependencies, e.g. forks per Python version;
  • the dependencies differ between the pip.parse calls for different
    Python versions of the same hub;
  • the package has no dependencies in the hub.

This threads the dependencies through the existing pkg_aliases
support (#4105) instead of passing them to the whl_deps_repo as inline
metadata, because the hub then owns the dependency graph and the
per-package repos are only needed for the sources. pkg_aliases is
fixed to create the _pkg and _whl group implementation targets and
to pass the rules it needs, as this code path was only used with mocks.

Follow-ups:

  • Point the hub whl_map directly at the shared extracted wheel repos
    for the packages with known dependencies and skip creating their
    whl_deps_repo. This needs pkg_aliases to also create the targets
    for packages without dependencies.
  • Combine the dependencies of forked uv.lock entries using their
    resolution-markers instead of falling back to METADATA.
  • WORKSPACE pip_parse with uv_lock still uses the wheel METADATA.

Stacked on #3856.

Summary:
- Thread deps from uv.lock file into the hub_repository
- Use them to define the dep graph
- Alter the pkg_aliases and render_pkg_aliases to be able
  to use the dep graph information if it is provided.

TODO:
- [ ] Split the whl_library_targets into separate files, I think they
  have stabilized enough.
- [ ] Have a separate PR just for threading the dependencies to
  render_pkg_aliases. We probably need to handle various edge-cases
  where we have different package versions in the uv.lock file.
- [ ] Have a separate PR to enable usage of the dependency graph. This
  is now probably doing a few shortcuts.
- [ ] If we have uv.lock, define fewer wheel repos - only define the
  sources without passing the dep_template in there.
- [ ] Test coverage for the above.
aignas added 7 commits August 23, 2026 22:54
This is the common denominator and for now no warnings are printed, but
there are opportunities to do this.

This approach is way more surgical than the previous one.

Fixes bazel-contrib#2948
rickeylev pushed a commit to rickeylev/rules_python that referenced this pull request Aug 25, 2026
…ontrib#4105)

Summary:
- If `requires_dist` are passed we create a `py_library` using the
  aliased `srcs` target.
- Currently this is only exercised by tests.

Split out of bazel-contrib#4059
aignas added 20 commits August 26, 2026 21:11
The hub repository references the srcs and whl_file targets of each
spoke repository, so whl_deps_repo needs to alias them to the repository
holding the extracted wheel.
The experimental_index_url is resolved using envsubst to decide if the
bazel downloader should be used. Pass the unresolved template to the
simpleapi_download and whl_library instead of the resolved value, so that
credentials that may be stored in the environment variables do not end up
in the whl_library arguments or the lock file.
Extract each wheel downloaded by the bazel downloader once for every
unique URL, filename and integrity combination and replace the
whl_library repositories with whl_deps_repo repositories that wire the
dependencies by reading the METADATA from the extracted repository.

The download arguments, such as the index_url and envsubst, are taken
from the first registration of the wheel. The URLs are compared without
resolving envsubst, so that credentials do not end up in the arguments.

Wheels with patches, annotations or extra hub aliases are still extracted
in the whl_library repository.
The wheels downloaded by the bazel downloader are now extracted in a
repository named after the wheel filename, so check for the python tag
of the wheel in addition to the python version in the spoke repo name.
Labels are rendered with a single @ in WORKSPACE mode.
Move the lookup of the repos that have extra hub aliases into
shared_whl_srcs so that it is covered by the unit tests.
Wheels that are extracted differently, e.g. with a different
pip_data_exclude, now get their own extracted repo instead of falling back
to a regular whl_library. Also name the envsubst news entry after the PR.
The previous approach on this branch is dropped: parts of it landed
separately (bazel-contrib#4087, bazel-contrib#4101, bazel-contrib#4105, bazel-contrib#4106) and the rest is obsolete. The
tree is reset to the state of bazel-contrib#3856 (shared extracted wheel repos) so
that the uv.lock dependency graph plumbing can be reimplemented on top
of it.
When `pkg_aliases` creates the `pkg` and `whl` targets in the hub for a
package that is part of a dependency group, the public labels are
aliases to the group. Create the `_pkg` and `_whl` targets instead, which
the group implementation in `//_groups` depends on, so that the names do
not clash.
Render the dependency edges of each uv.lock entry as PEP 508
requirements, keeping the extras and the markers of the edges. The
optional dependencies are included only for the extras that the
dependents request. If the entries of a package that are used on the
target platforms have different dependencies, the dependencies are left
unknown, so that the wheel METADATA is used instead.
The downloaded wheels are now extracted in a repository named after the
wheel filename, which also owns the lazily fetched archive repository.
The hub has a single dependency graph for all of its python versions, so
the uv.lock dependencies of a package are used only if they are the same
for all python versions. The dependencies on packages that are not in
the hub, e.g. ones only used on other platforms, are dropped, like the
`whl_library` does with the `packages` from `config.bzl`.
When `pkg_aliases` is given `requires_dist`, `whl_library_deps_targets`
creates the `py_library` and `env_marker_setting` targets, which were
missing from the default `rules` struct, because only the unit tests
with mocks used this code path.
Pass the uv.lock dependencies of each package to the `hub_repository`
and render them as the `requires_dist` of `pkg_aliases`, so that the hub
creates the `pkg` and `whl` targets wired with the dependencies from the
uv.lock file instead of the wheel METADATA. The members of a dependency
group are passed as `group_deps`.
@aignas aignas changed the title feat: use uv.lock deps for the dep graph instead feat(pypi): use the uv.lock dependency graph in the hub Nov 8, 2026
uv.lock always writes the extras of a dependency as a list, so drop the
string handling and fix the test fixture that used a string.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant