Skip to content

Fix quadratic processing time for consecutive abbreviation definitions - #1702

Open
deadstrobe5 wants to merge 4 commits into
Python-Markdown:masterfrom
deadstrobe5:perf/abbr-consecutive
Open

deadstrobe5 wants to merge 4 commits into
Python-Markdown:masterfrom
deadstrobe5:perf/abbr-consecutive

Conversation

@deadstrobe5

@deadstrobe5 deadstrobe5 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

AbbrBlockprocessor.run records one definition per call and puts the rest of the block back on the stack, so a block of N definitions is re-tested by every block processor N times and the remainder is copied N times. A glossary of abbreviations appended to every page (the setup Material for MkDocs documents with pymdownx.snippets auto_append) hits this on every page.

This records the following definitions in the same pass, the same way consecutive reference definitions were handled in #1678. A later definition is only taken when the block starts with a definition, the later one fits on one line, and it is followed by another definition or the end of the block. In any other case a processor with higher priority could claim the remainder (a Setext underline, a table separator or a definition list line after it), so that definition is left to the next pass and the output does not change. A test covers the Setext case.

Median of 5 alternating runs in one process, CPython 3.14, markdown.Markdown(extensions=[...]).convert():

Document Before After
2 KB page + 50 definitions (9 extensions) 5.41 ms 2.18 ms
2 KB page + 200 definitions (9 extensions) 79.5 ms 3.92 ms
2 KB page + 200 definitions (abbr only) 30.3 ms 2.18 ms
10,000 definitions (abbr only, one run) 178 s 0.32 s

Pages without definitions are unchanged. I compared the HTML and the recorded abbreviations of the old and new code on 50,000 generated documents (definitions inside lists, block quotes and indented blocks, empty and '' titles, multi-line definitions, mixed with tables, headers and definition lists) and found no differences.

AI Assistance Disclosure

  • No AI tools were used in preparing this PR.

  • AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

    Claude Code (Claude Opus 5.5) was used for everything ticked below. The review attestation in the box above is left for the human submitter.

    AI tools were use to assist with the following:

    • Code
    • Code Comments
    • Documentation
    • Changelog Entry
    • Commit Message
    • The Description Above
    • Other: ______________

Checklist

@waylan waylan added the requires-changes Awaiting updates after a review. label Nov 10, 2026
@deadstrobe5

Copy link
Copy Markdown
Contributor Author

Agreed, fixed in 796d415 (plus c66f873 for line length). The processor now takes every consecutive definition, whatever follows, so that input gives <p>===</p>. I dropped the rule that left the last definition for another processor, which was there only to keep the old output.

Current master is inconsistent here: two definitions before === already give <p>===</p>, but with three or more, master turns the last definition into an <h1>. I added a test for the three-definition case (it fails on master and passes here). Against master, the output differs only in this pattern: 19 of 4,168 documents (the test fixtures, the docs, and random mixes of definitions, headers, lists and text), all of them definitions followed by ===. 200 definitions still go from 15.5 ms to 0.63 ms. Also merged master to fix the changelog conflict.

Comment thread markdown/extensions/abbr.py Outdated
if m is None or '\n' in m.group():
break
matches.append(m)
for m in matches:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that there is no code between these 2 loops, they should be able to be combined into a single loop.

Although, why not just call finditer up on line 160 and then loop through the results?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b39a7c0: it is one loop over finditer now.

The loop stops at the first line that is not a definition, so the rest of the block still goes back as one block. Taking every match in the block would split the text between definitions into separate blocks, so a definition line in the middle of a list or paragraph would act like a blank line (in a fuzz run a tight list became a loose one).

Against master, 30,000 random documents differ only where a run of definitions is directly followed by a ===/--- line or a : definition list line; those definitions are now consumed, as in your setext example. 200 definitions: 19.3 ms before, 0.8 ms now.

@waylan waylan Nov 12, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks better. Thank you.

Taking every match in the block would split the text between definitions into separate blocks, so a definition line in the middle of a list or paragraph would act like a blank line (in a fuzz run a tight list became a loose one).

Then let's add some tests for that, unless they already exist. I'm assuming they don't or you wouldn't have mentioned it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-changes Awaiting updates after a review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants