Skip to content

build(python): migrate pydevlake and plugins to Pydantic 2 - #9352

Merged
klesh merged 1 commit into
apache:mainfrom
DoDiODev:pr/pydantic-v2
Nov 10, 2026
Merged

klesh merged 1 commit into
apache:mainfrom
DoDiODev:pr/pydantic-v2

Conversation

@DoDiODev

@DoDiODev DoDiODev commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Pre Checklist

Please complete ALL items in this checklist, and remove before submitting

  • I have read through the Contributing Documentation.
  • I have added relevant tests. (existing suites adapted; no new behavior)
  • I have added relevant documentation. (N/A — no user-facing change)
  • I will add labels to the PR, such as pr-type/bug-fix, pr-type/feature-development, etc.
    (I can't set labels myself; suggested: pr-type/refactor, component/plugins)

Summary

Moves the Python plugin framework from Pydantic 1 to Pydantic 2, as agreed in
#9092. Pydantic 1 does not support Python 3.14, so this is the first step
before the Python runtime (3.11, EOL 2027-10) can be upgraded.

Package Before After
pydantic 1.10.27 2.14.3
sqlmodel 0.0.8 0.0.38
SQLAlchemy 1.4.41 2.0.57

Ported: pydevlake, azuredevops, glpi, redmine, zephyr and the e2e
fakeplugin. The three Poetry locks only change the Pydantic stack (plus its
new dependencies pydantic-core, annotated-types, typing-inspection;
sqlalchemy2-stubs is dropped).

pydevlake

  • Pydantic 2 APIs: model_config/ConfigDict, field_validator,
    model_fields, model_dump_json, RootModel, Literal instead of
    Field(const=True).
  • Optional[...] fields get an explicit = None default (in Pydantic 2 they
    are required otherwise). This accounts for most of the diff in the domain
    layer and plugin models/migrations.
  • sa_column=Column(...) → sa_type=... (+ sa_column_kwargs for the
    created_at/updated_at defaults), so columns are not shared between
    inherited tables.
  • ToolTable sets coerce_numbers_to_str=True to keep the Pydantic 1
    behavior for APIs that return ids as numbers.
  • autoextract uses model_validate: SQLModel table models skip validation
    in __init__, so raw strings would no longer be coerced to
    datetime/enums.
  • RawModel keeps its column types (data/input binary, url TEXT), so
    existing raw tables stay compatible. Two tests wrote str into data;
    they now encode to bytes like the collector does (SQLAlchemy 2 rejects
    str for binary columns).
  • message.py: SerializeAsAny for RemoteScope.data, otherwise Pydantic 2
    serializes the base type and drops plugin-specific scope fields.
  • model_info.py: reads json_schema_extra / annotations instead of
    field.field_info.extra / field.required.
  • Test harness: expected and actual models are re-validated before comparing,
    so both sides have the same field types.

Plugins

  • glpi: GLPI expands dropdowns to 'Name (id)'; these values are reduced to
    the id before validation of the integer columns. The names are still read
    from the raw data for enrichment, so the output does not change.
  • redmine, zephyr: class Config → model_config.
  • fakeplugin: ids as strings, Optional defaults.

Dependabot

  • The pydantic major-version ignore is removed.
  • The sqlmodel ignore stays: 0.0.48 rejects naive datetimes (pydevlake uses
    datetime.now()/utcnow()) and pulls in SQLAlchemy 2.1. That is a separate
    follow-up.

Does this close any open issues?

Part of #9092 (Pydantic step; Python runtime upgrade follows separately).

Screenshots

N/A

Other Information

Tests

  • cd backend/python && sh ./run_tests.sh — passed:
    azuredevops 8 passed / 1 skipped, glpi 72 passed / 1 skipped,
    redmine 31 passed / 1 skipped, zephyr 26 passed, pydevlake 13 passed.
  • make unit-test and make e2e-test (MySQL and PostgreSQL, includes the
    Python fakeplugin) — passed in the fork CI: run 42621392320 (16/16 jobs)

Not in this PR

  • Python runtime 3.11 → 3.13/3.14 (next PR).
  • sqlmodel ≥ 0.0.48 / SQLAlchemy 2.1 (timezone-aware datetimes).
  • Pinning the uv installer in backend/Dockerfile and the lake-builder image.

Move pydevlake from Pydantic 1.10 / SQLModel 0.0.8 / SQLAlchemy 1.4 to
Pydantic 2.14 / SQLModel 0.0.38 / SQLAlchemy 2.0 and port azuredevops,
glpi, redmine, zephyr and the e2e fakeplugin.

- Optional fields get explicit None defaults (required in Pydantic 2).
- Config classes, validators and __fields__ use the Pydantic 2 APIs.
- ToolTable keeps v1 behavior for numbers in str fields
  (coerce_numbers_to_str).
- autoextract validates the extracted values so they are coerced.
- glpi reduces expanded dropdown values ('Name (id)') to the id before
  extraction; the names are still read from the raw data.
- Regenerate the Poetry locks; Dependabot no longer ignores Pydantic
  majors, SQLModel stays ignored (0.0.48 rejects naive datetimes).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@klesh klesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thanks for your contribution.

@klesh
klesh merged commit 0fcdbde into apache:main Nov 10, 2026
11 checks passed
@DoDiODev
DoDiODev deleted the pr/pydantic-v2 branch November 10, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants