Skip to content

Add a multithreaded TPC digitizer device - #16090

Merged
sawenzel merged 5 commits into
AliceO2Group:devfrom
sawenzel:tpc-digi-mt-v2
Nov 12, 2026
Merged

sawenzel merged 5 commits into
AliceO2Group:devfrom
sawenzel:tpc-digi-mt-v2

Conversation

@sawenzel

@sawenzel sawenzel commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

This PR adds an optional TPC digitizer device that digitizes all sectors in one process with several threads, instead of one process per lane.

With lanes, every process loads and recalculates its own copy of the space-charge distortion maps, which in PbPb costs about 1 GB of memory and several minutes of CPU per lane. In the new device the maps are prepared once and shared by all threads, while each thread owns a digitizer with its own random rings, hit chains and output file. It is enabled with --tpc-digi-threads N together with --tpc-chunked-writer; the chunked merger and everything downstream are unchanged.

The effect on one anchored PbPb timeframe (run 572106, distortion type 2) is substantial:

8 lanes 8 threads
peak PSS 15.05 GiB 7.95 GiB
wall time 216 s 69 s

The lanes run with --n-threads-distortions 1, as in O2DPG, and the threaded device with 8.

Note that nothing changes by default. The digits of the threaded device are identical to those of the lane spec with --tpc-sector-seeding for 1, 4 and 8 lanes and threads, with and without distortions, also when run through the O2DPG workflow up to the TPC clusterization.

This builds on #16082, #16083 and #16085, which are merged. It moves the per-sector loop into a shared function, prepares the digitizer for concurrent use and adds the device itself. The last two commits address review comments.

Overall, this halves the memory of the TPC digitization and makes it about three times faster.

https://its.cern.ch/jira/browse/O2-7309

Assisted by Claude Code.

@sawenzel

sawenzel commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Added commits after internal review.

sawenzel and others added 5 commits November 10, 2026 22:56
This moves the per-sector loop of the TPC digitizer spec into a free function, as a preparation for MT digitization.

- digitizeSector runs the collision loop and the flushes of one sector and returns the digit groups.
- The output of each flush is handed to a callback, so the function makes no DPL calls.
- The lane spec now calls it; the chunked writer and the DPL outputs stay in the spec.
- The output is unchanged.

https://its.cern.ch/jira/browse/O2-7309

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This makes several TPC digitizers usable side by side in one process, as a preparation for MT digitization.

- Each DigitContainer keeps its own copy of the dead channel map, because CDBInterface::getDeadChannelMap() rebuilds the shared map on every call.
- The one-time settings report moves from the flush path into updateCalibrations().
- Digitizer::setSCDistortionsFrom() shares the distortion maps and their scaling with another digitizer.
- digitizeSector() takes an optional mutex for the calls that read calibrations through the CDBInterface.
- The output is unchanged.

https://its.cern.ch/jira/browse/O2-7309

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This adds an optional single TPC digitizer device that digitizes the sectors with several threads instead of lanes.

- --tpc-digi-threads N replaces the TPC lanes by one device; it requires --tpc-chunked-writer.
- Each thread owns a digitizer with sector seeding, its own hit chains and its own tpc_driftime_digits_lane file, so the existing merger is unchanged.
- The CCDB objects are handled once on the device thread, and the distortion maps are recalculated once and shared.
- The digits are identical to the lane spec with --tpc-sector-seeding.

https://its.cern.ch/jira/browse/O2-7309

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This addresses the review of the multithreaded TPC digitizer device.

- The workers copy the random rings of the first worker instead of filling their own.
- The workers no longer flush the process-wide debug streamer; the device flushes it after the join.
- The Digitizer serialises its calibration reads itself, so digitizeSector needs no mutex.
- The CCDB handling, the calibration files and the common options of both devices are in TPCDigitizerConditions.
- The device stops on a second timeframe, which the chunked digit files cannot hold.
- Shorter comments in TPCSectorDigitization.h.

https://its.cern.ch/jira/browse/O2-7309

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This addresses the second review of the multithreaded TPC digitizer device.

- Old lane files are found with o2::utils::listFiles, as in the chunked merger.
- Unused includes and the SpaceCharge alias are removed from the device.
- setSectorSeedingFrom fills the rings of the source digitizer, so initSectorSeeding is private.
- The seeding documentation states that the sectors share the ring contents.

https://its.cern.ch/jira/browse/O2-7309

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@alibuild

Copy link
Copy Markdown
Collaborator

Error while checking build/O2/fullCI_slc9 for 57422d7 at 2026-11-11 05:18:

## sw/BUILD/O2-sim-challenge-test-latest/log
./sim-challenge.log[ERROR] SHUTTING DOWN DUE TO SIGNALED EXIT IN COMPONENT 208
./sim.log[ERROR] SHUTTING DOWN DUE TO SIGNALED EXIT IN COMPONENT 208


## sw/BUILD/o2checkcode-latest/log
--
========== List of errors found ==========
++ echo 'Found 0 errors and 0 warnings.'
Found 0 errors and 0 warnings.
++ [[ 0 -gt 0 ]]
++ [[ 0 -gt 0 ]]
++ [[ 0 -gt 0 ]]
++ mkdir -p /sw/INSTALLROOT/0f36dfe2fa6d50f46c47eb8336eeef00082e4a49/slc9_x86-64/o2checkcode/1.0-local64/etc/modulefiles
++ alibuild-generate-module
+ HAS_BIN=
+ HAS_LIB=
+ HAS_CMAKE=
+ HAS_ROOT=
+ '[' 0 '!=' 0 ']'
+ cat
+ printf 'if ![ is-loaded '\''BASE/1.0'\'' ] {\n module load BASE/1.0\n}'
+ echo 'BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release'
BUILD_REQUIRES=CMake alibuild-recipe-tools defaults-release
+ FULL_BUILD_REQUIRES='Python-modules-list MPFR googlebenchmark UUID date defaults-release ninja-fortran bz2 pytorch_cpuinfo json-c Alice-GRID-Utils ninja alibuild-recipe-tools CMake FairCMakeModules O2-customization double-conversion safe_int'
+ echo 'FULL_BUILD_REQUIRES=Python-modules-list MPFR googlebenchmark UUID date defaults-release ninja-fortran bz2 pytorch_cpuinfo json-c Alice-GRID-Utils ninja alibuild-recipe-tools CMake FairCMakeModules O2-customization double-conversion safe_int'
FULL_BUILD_REQUIRES=Python-modules-list MPFR googlebenchmark UUID date defaults-release ninja-fortran bz2 pytorch_cpuinfo json-c Alice-GRID-Utils ninja alibuild-recipe-tools CMake FairCMakeModules O2-customization double-conversion safe_int
++ env
++ cut -f1 -d=
++ grep -v '^DEFAULT_'
++ grep -v PKGREVISION
++ grep -v ALIBUILD_RECIPE_TOOLS
++ grep REVISION
++ sed -e s/_REVISION//
+ for x in $(env | cut -f1 -d= | grep -v "^DEFAULT_" | grep -v PKGREVISION | grep -v ALIBUILD_RECIPE_TOOLS | grep REVISION | sed -e 's/_REVISION//')
++ eval 'echo $LIBFFI_REVISION'
+++ echo 14
+ REVISION_VALUE=14
++ eval 'echo $LIBFFI_VERSION'
+++ echo v3.2.1-alice1
+ VERSION_VALUE=v3.2.1-alice1
++ eval 'echo $LIBFFI_ROOT'
+++ echo /sw/slc9_x86-64/libffi/v3.2.1-alice1-14
+ ROOT_PATH_VALUE=/sw/slc9_x86-64/libffi/v3.2.1-alice1-14
+ '[' -z 14 ']'
+ echo 'Python-modules-list MPFR googlebenchmark UUID date defaults-release ninja-fortran bz2 pytorch_cpuinfo json-c Alice-GRID-Utils ninja alibuild-recipe-tools CMake FairCMakeModules O2-customization double-conversion safe_int'
+ tr '[:lower:]' '[:upper:]'
+ tr - _
+ tr ' ' '\n'
[0 more errors; see full log]

Full log here.

@sawenzel
sawenzel merged commit 4c92364 into AliceO2Group:dev Nov 12, 2026
12 checks passed
@sawenzel
sawenzel deleted the tpc-digi-mt-v2 branch November 12, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants