Skip to content

[mypyc] Report errors for native classes with unsupported bases - #22374

Merged
JukkaL merged 4 commits into
masterfrom
mypyc-bulkfix-1186-defaultdict-base
Nov 5, 2026
Merged

JukkaL merged 4 commits into
masterfrom
mypyc-bulkfix-1186-defaultdict-base

Conversation

@JukkaL

@JukkaL JukkaL commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Fixes mypyc/mypyc#1186. Fixes mypyc/mypyc#1103.

A native class that inherits from collections.defaultdict was treated like a regular dict subclass, but defaultdict instances have an extra field, so the native instance layout was too small and compiled code could segfault or behave incorrectly:

from collections import defaultdict

class DD(defaultdict[str, int]):
    def __init__(self) -> None:
        super().__init__(int)

DD()["x"]  # Segfault

Inheriting from collections.OrderedDict has the same problem (the compiled module failed to import).

The problem is more general: native classes assume that non-mypyc base classes have the layout of object (or dict/BaseException), which isn't true for many classes implemented in C (e.g. random.Random or io.BytesIO). Stubs don't describe instance layouts, so now a native class can only inherit from a non-mypyc class defined in a stub if the class is in an allowlist. Otherwise mypyc reports an error with the usual note about using @mypyc_attr(native_class=False) as a workaround. This also covers defaultdict and OrderedDict. Classes defined in Python sources are still accepted, unless they define __slots__ or are compiled separately into a C extension.

The allowlist only includes standard library classes with evidence of being subclassed in real code (at least two of the 1000 most downloaded PyPI projects, a large private codebase, or mypy itself) that are documented in the CPython docs and have a compatible runtime layout on Python 3.10-3.15, including the free-threaded build. Native subclasses of each entry were compiled and exercised on Python 3.10, 3.14 (including free-threaded) and 3.15. This keeps the list size manageable and avoids internal classes that are likely to change.

The entries are the full names mypy uses (those of the modules where typeshed defines the classes), so a test checks that they still match how mypy resolves the classes on the Python version and platform the tests run on.

Classes that keep weak references to the instance (threading.Thread, logging handlers, ...) are left out for now and listed in a comment, since weakrefs to native subclasses of interpreted classes aren't supported yet.

…ases

Fixes mypyc/mypyc#1186. Fixes mypyc/mypyc#1103.

A native class that inherits from `collections.defaultdict` was treated
like a regular `dict` subclass, but `defaultdict` instances have an
extra field, so the native instance layout was too small and compiled
code could segfault or behave incorrectly:

```python
from collections import defaultdict

class DD(defaultdict[str, int]):
    def __init__(self) -> None:
        super().__init__(int)

DD()["x"]  # Segfault
```

Inheriting from `collections.OrderedDict` has the same problem (the
compiled module failed to import).

The problem is more general: native classes assume that non-mypyc base
classes have the layout of `object` (or `dict`/`BaseException`), which
isn't true for many classes implemented in C (e.g. `random.Random`,
`io.BytesIO`, `datetime.date`). Stubs don't describe instance layouts,
so now a native class can only inherit from a non-mypyc class defined
in a stub if the class is in an allowlist (`SAFE_STUB_BASE_CLASSES` in
`mypyc/irbuild/safe_bases.py`). Otherwise mypyc reports an error with
the usual note about using `@mypyc_attr(native_class=False)` as a
workaround. This also covers `defaultdict` and `OrderedDict`. Classes
defined in Python sources are still accepted, unless they define
`__slots__` or are compiled separately into a C extension.

The allowlist only includes standard library classes with evidence of
being subclassed in real code (at least two of the 1000 most
downloaded PyPI projects, a large private codebase, or mypy itself)
that are documented in the CPython docs and have a compatible runtime
layout on Python 3.10-3.15, including the free-threaded build. Native
subclasses of each entry were compiled and exercised on Python 3.10,
3.14 (including free-threaded) and 3.15. This keeps the list size
manageable and avoids internal classes that are likely to change.

The entries are the full names mypy uses (those of the modules where
typeshed defines the classes), so a test checks that they still match
how mypy resolves the classes on the Python version and platform the
tests run on.

Classes that keep weak references to the instance (`threading.Thread`,
logging handlers, ...) are left out for now and listed in a comment,
since weakrefs to native subclasses of interpreted classes aren't
supported yet.
Comment thread mypyc/irbuild/prepare.py Outdated
Comment on lines +628 to +629
if is_compiled_module_path(path):
return base, "defined in a separately compiled extension module"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be something like "defined in an extension module not produced by mypyc"? "separately compiled" could be confusing considering mypyc has a separate compilation mode.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@JukkaL
JukkaL merged commit 67c7290 into master Nov 5, 2026
25 checks passed
@JukkaL
JukkaL deleted the mypyc-bulkfix-1186-defaultdict-base branch November 5, 2026 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants