Repository navigation
bcm2712: Use all 20 MIP1 outputs - #7729
Merged
Merged
Conversation
nbuchwitz
force-pushed
the
devel/pcie-mip1-bump
branch
from
October 31, 2026 21:28
55e66be to
b789b8b
Compare
nbuchwitz
force-pushed
the
devel/pcie-mip1-bump
branch
from
November 2, 2026 15:34
b789b8b to
ba5b936
Compare
Contributor
|
From the incomplete Broadcom documentation that I have, the MIP to SPI mappings look correct. There was a register bit on 2712C1 that the firmware always sets to enable this (expanded but discontiguous) mapping, I believe it's hardcoded to a 1 in 2712D0. |
Contributor
Author
|
Thanks! Has the firmware always set that bit on C1, or is there a minimum version? |
Contributor
|
All rpi-eeprom releases supporting Pi 5 set this bit on C1. |
nbuchwitz
force-pushed
the
devel/pcie-mip1-bump
branch
from
November 7, 2026 10:41
ba5b936 to
9c77be0
Compare
MIP1, the MSI controller behind pcie1 on BCM2712, only describes 8
outputs. I measured the wiring on a CM5 and found 20, routed to GIC
SPIs that are not contiguous.
Patch 1 documents multiple msi-ranges entries. Patch 2 fixes multi-MSI
blocks for non-power-of-two counts and is meant for stable. Patch 3
lets the driver take several ranges. Patch 4 adds the outputs and
depends on patch 3.
Tested on rpi-7.3.y on a Revolution Pi Connect 5 (CM5) with three
LAN7430 NICs behind a PCIe switch, 6 MSI-X vectors each. DPC on the
switch ports uses another 6 outputs. Vectors per NIC:
8 outputs 20 outputs
--------------------------------------------
without DPC 6, 2, INTx 6, 6, 6
with DPC 2, INTx, INTx 6, 6, 2
Industrial Ethernet setups often pin each NIC's interrupts to its own
CPU. That only works on MIP1. The RC MSI target keeps everything on
CPU0. Two NVMe drives behind a switch or an FPGA design [1] hit the
same limit.
A test module requesting 3 MSI vectors per NIC gets aligned blocks with
the series. Without patch 2 the block is misaligned, its last vector
lands outside the described outputs and the device's own MSI never
arrives. The old single-range node still works.
[1] https://forums.raspberrypi.com/viewtopic.php?t=398134
Signed-off-by: Nicolai Buchwitz <[email protected]>
--- b4-submit-tracking ---
# This section is used internally by b4 prep for tracking purposes.
{
"series": {
"revision": 1,
"change-id": "20261030-investigate-bcm2712-mip1-v2-4ff97c32b04a",
"prefixes": [],
"presubject": ""
}
}
…msi-ranges The MIP outputs are not wired to one contiguous block of GIC SPIs, so a single msi-ranges entry cannot describe all of them. Define how several entries map to the outputs, and limit the legacy brcm,msi-offset to a single entry, the only case the driver has ever handled. Signed-off-by: Nicolai Buchwitz <[email protected]>
…unts Multi-message MSI needs a naturally aligned block of message data, but ilog2() rounds the request size down. A request for three vectors reserves two slots and maps the third onto the next slot, which may already belong to another device. Reserve exactly the requested number of slots at a start aligned to the next power of two. Fixes: 32c6c05 ("irqchip: Add Broadcom BCM2712 MSI-X interrupt controller") Cc: [email protected] Signed-off-by: Nicolai Buchwitz <[email protected]>
The MIP outputs are not wired to one contiguous block of GIC SPIs, so a single msi-ranges entry can only describe part of a MIP. Accept one entry per contiguous run, in output order, and look up the SPI for each message. brcm,msi-offset keeps its meaning for the existing single range and is rejected when more than one range is given. Signed-off-by: Nicolai Buchwitz <[email protected]>
MIP1 serves pcie1, the external PCIe x1 port on Pi 5 and CM5, but only 8 of its outputs are described. A single multi-queue device can use most of them, and further devices get fewer vectors or none. The remaining wiring was measured on a CM5 by raising each MIP output and checking which SPI became pending. 20 outputs are wired, the others showed no route. The wiring is assumed to be the same on all BCM2712 steppings but has only been verified on CM5. MIP output GIC SPI 0..1 196..197 2..3 200..201 4..5 206..207 6 240 7 246 8..15 255..262 16 275 17 282 18 263 19 283 Signed-off-by: Nicolai Buchwitz <[email protected]>
nbuchwitz
force-pushed
the
devel/pcie-mip1-bump
branch
from
November 12, 2026 10:53
9c77be0 to
bd378e6
Compare
Contributor
|
I have a big chungus dual SFP+ 10GbE network card, served by the bnx2x driver. The driver has a useful feature where you can specify the number of queues per interface - and an MSIx interrupt gets assigned per-queue. With /proc/interrupts
So it's dynamically sizing - is it a bug in this driver or the MIP allocation? |
Contributor
|
Ah, there's a hidden reservation for a "CNIC" vector, whatever that is. Note the MIP-MSI domain index going from 0 - 2...8. So it's allocating 20 vectors when 8 queues are enabled. Works on my machine. |
Contributor
Author
|
Thanks for testing! Earlier today I've sent the msi allocation bugfix upstream and once it is merged I will send the rest too. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've been experimenting with more MSI vectors on MIP1, as our RevPi Connect 5 has 3x LAN7430s on pcie1. They don't fit into the 8 MIP1 outputs (each tries to claim 6 outputs), so our DT uses the rc MSI controller instead. Unfortunately it can't pin IRQs, which is not great in most industrial fieldbus setups.
The patches seem to work for my limited PCIe knowledge, but I'd really value @P33M's (and others) feedback on it. If this can work, I'd try to upstream the patches as usual.