diff --git a/DATA/production/configurations/mapCreationGrid/README.md b/DATA/production/configurations/mapCreationGrid/README.md index 31dd71544..44acd07e6 100644 --- a/DATA/production/configurations/mapCreationGrid/README.md +++ b/DATA/production/configurations/mapCreationGrid/README.md @@ -20,10 +20,24 @@ left as-is. 1. **Build the slot lists** (local, once per batch of runs): ```bash - ./discoverResiduals.sh + ./discoverResiduals.sh --calib-pass ``` - `` lists one run per line: ` `, e.g. `LHC26ak 572557`. + `` lists one run per line: ` `, e.g. `LHC26ak 572557`. (pp only, and only if + you'll set `MAKE_HYBRID_MAPS=true` in step 2: add ` ` — see "Hybrid time + sampling (pp)" under step 2.) + + **MC:** use `--mc` instead of `--calib-pass`, with the MC production tag in place of the period: + + ```bash + printf 'LHC26j19_residuals %s\n' 568336 568337 568338 > batch_mc.txt # + ./discoverResiduals.sh batch_mc.txt --mc + ``` + + This searches `/alice/sim///0//ResidualTPCmerging/Stage_0/` for the production's + merged residuals, falling back to the per-job files under `/alice/sim///0//` with + a warning when there are none (`--mc-cycle ` for a production cycle other than `0`). In step 2, also set `MC_MODE=true` and `BAD_RANGES_NAME=""`. Details: "MC productions" in + "The chain, in order" below. 2. **Submit stage 1 (map creation).** In `submitMapCreation.sh`, set: - `BAD_RANGES_NAME` — the bad-time-ranges file for this data period (keep the default if it still @@ -43,6 +57,39 @@ left as-is. This takes hours per map — watch progress on alimonitor and wait for it to finish (or mostly finish) before continuing. + **pp hybrid production** (fine-length maps early in each fill, coarse-length for the rest — see + "Hybrid time sampling (pp)" below): also set `MAKE_HYBRID_MAPS=true`, `SLOT_LENGTH_MIN_COARSE`, + `FINE_WINDOW_MIN`, and it's worth pinning `RUNS_FILTER` explicitly since `./lists/` accumulates across + batches. For example: + + ```bash + JOBNAME="mapcreation-pp-lowfield-v1" # unique per submission + BAD_RANGES_NAME="" # or "" to disable -- the default is PbPb-specific + RUNS_FILTER="566139,566159,566160,566161,566188,566189,566190,566191" + MAKE_HYBRID_MAPS=true + SLOT_LENGTH_MIN=5 # fine length (first FINE_WINDOW_MIN min of each fill) + SLOT_LENGTH_MIN_COARSE=60 # coarse length (rest of the fill) + FINE_WINDOW_MIN=60 + ``` + + Needs step 1's batch file to have carried ` ` for these runs (see step 1 + above) — `submitMapCreation.sh` fails fast, before submitting anything, if any run named in + `RUNS_FILTER` is missing that. + + **MC production** (lists from `discoverResiduals.sh --mc` in step 1): set `MC_MODE=true`. Each run + then becomes one map over all of its files, with no time cut. `SLOT_LENGTH_MIN` is ignored, and + `MAKE_HYBRID_MAPS` must stay `false`. For example: + + ```bash + JOBNAME="mapcreation-MC-LHC26j19-v1" # unique per submission + BAD_RANGES_NAME="" # the data's bad-ranges list does not apply to MC + MC_MODE=true + MAKE_HYBRID_MAPS=false + ``` + + It should report `MC_MODE=true -- one map per run, no time cut` and one slot per run. Step 3 is the + same as for data. + 3. **Submit stage 2 (smoothing + FastTransform).** In `submitPostProcess.sh`, set `STAGE1_JOBNAME_TAG` to the exact `JOBNAME` used in step 2, then: @@ -117,13 +164,21 @@ without redoing the expensive stage-1 work for that slot. ./discoverResiduals.sh [--calib-pass cpass2_residuals] ``` -`` is one run per line: ` `, e.g.: +`` is one run per line: ` [ ]`, e.g.: ``` LHC26ak 572557 LHC26ak 572558 ``` +The last two columns are **optional** and only needed for **hybrid map creation** (see below) — omit them +for a normal, non-hybrid production. When given, both must be present together: `` is the LHC fill +number (shared by every run in it) and `` is that run's start-of-run (SOR) timestamp, +UTC milliseconds (13 digits) — the same metadata columns the old SLURM-era hybrid tooling +(`makeStaggerd.py`/`combinedSampling.py`) used. When present, `discoverResiduals.sh` writes them to a +`.fillinfo.txt` sidecar next to that run's slot-list index, which `stageSlotsForGrid.py --hybrid` +reads later, at submission time. + This queries AliEn (`/alice/data/////`) for `o2tpc_residuals_*.root` files and writes per-run slot lists under `./lists//`. Runs in parallel (`N_PARALLEL=8` in the script). Safe to re-run; each run is independent. The period is resolved **per line**, so a batch file @@ -221,6 +276,44 @@ it adds to it, and Step 2 by default puts *everything* under `./lists/` into the silently recalibrates the earlier batch from scratch at hours per slot. Either use a fresh `listsDir` per batch, or set `RUNS_FILTER` in the Step-2 wrapper (see below). +#### MC productions + +MC residuals live under `/alice/sim//////o2tpc_residuals.root`, +not under a data period's calib pass. Use `--mc`, with the production tag in place of the period: + +```bash +cat > batch_mc.txt < for another one +``` + +The year is taken from the tag (`LHC26…` → 2026). `--calib-pass` is rejected with `--mc`, since the MC +path has no such level. + +The production also merges the per-job files, about 150 jobs into one +`/ResidualTPCmerging/Stage_0//o2tpc_residuals.root` (same trees, ~15 GB). When any +merged file exists, the run's list holds **only** the merged files. A search over the whole run +directory would list every residual twice, once per-job and once merged, and each `alien://` open costs +~7 s in stage 1, so a few dozen merged files beat thousands of per-job ones. A `Stage_0//` without +the file is a merge job that failed or is still running: its share of the statistics is missing, +and discovery prints a `WARNING: only / merge jobs produced o2tpc_residuals.root` for that run. +Wait for the merging, or accept the smaller sample. A run with no merged file at all falls back to the +per-job files, with a warning. + +MC file names carry no `_` label, so there is nothing to cut windows from: +`makeResidualLists.sh` writes one list per run (`residuals.all..txt`), and Step 2 needs +`MC_MODE=true`. Each run then becomes **one** map over all of its files, with window bounds `0 0`, which +switches off the macro's per-TF time cut (`firstTFTime > 0` is its condition). `MAX_TRACKS_PER_SLICE` +still caps the statistics. MC paths carry no `_orbit` field, so `mapCreationJob.sh` leaves the file order +as listed. Staging refuses to mix the two kinds: data lists with `--mc`, or `MAKE_HYBRID_MAPS=true` with +`MC_MODE=true`, are errors, and MC lists without `--mc` are skipped with a warning naming the flag. + +Two things work differently in stage 2: `mapMeta.txt` gets `tsStart=0`, `tsEnd=0`, and `fill=-1`. The +bad-time-ranges list is applied by the anchor run's orbits, so set `BAD_RANGES_NAME=""` unless you want +the data's bad ranges cut from the MC too. + ### Step 2 — configure and submit stage 1 (map creation) Edit the config block at the top of **`submitMapCreation.sh`**: @@ -229,7 +322,11 @@ Edit the config block at the top of **`submitMapCreation.sh`**: |---|---| | `JOBNAME` | Identifies this production. Tags the AliEn manifest (`mapSlots..tsv`) and the small `stage1Workdir..txt` pointer file (see Step 3) — **must be unique per submission** so concurrent/repeated productions don't clobber each other's manifest or pointer. Stage 2 needs to know this value (see Step 4). | | `TOPWORKDIR` | Static top-level bucket under your GRID home for this job's working directory (replaces `grid_submit.sh`'s generic `selfjobs` default with something identifiable). | -| `SLOT_LENGTH_MIN` | Target length of **one map**, in minutes. Each run is cut into whole steps of this length; a trailing remainder shorter than half a step is absorbed into the last window rather than left as a runt that would fail `MIN_TRACKS_PER_SLICE`. `0` = one map per run. A 20 min run at `5` gives 4 maps, a 60 min run gives 12 — the count follows the run length instead of being fixed. | +| `SLOT_LENGTH_MIN` | Target length of **one map**, in minutes. Each run is cut into whole steps of this length; a trailing remainder shorter than half a step is absorbed into the last window rather than left as a runt that would fail `MIN_TRACKS_PER_SLICE`. `0` = one map per run. A 20 min run at `5` gives 4 maps, a 60 min run gives 12 — the count follows the run length instead of being fixed. With `MAKE_HYBRID_MAPS=true`, this is reused as the **fine** map length (see below). | +| `MAKE_HYBRID_MAPS` | `true`/`false` (default `false`). Hybrid time sampling for pp: `SLOT_LENGTH_MIN`-length ("fine") maps for the first `FINE_WINDOW_MIN` minutes of each **fill**, `SLOT_LENGTH_MIN_COARSE`-length ("coarse") maps for the rest of that fill — see "Hybrid time sampling (pp)" below. | +| `SLOT_LENGTH_MIN_COARSE` | Only used when `MAKE_HYBRID_MAPS=true`: target length of one **coarse** map, in minutes. Same `0`/trailing-remainder semantics as `SLOT_LENGTH_MIN`, applied independently within the coarse region. | +| `FINE_WINDOW_MIN` | Only used when `MAKE_HYBRID_MAPS=true`: duration, in minutes, of the fine-sampling window measured from the fill's start (default `60`). | +| `MC_MODE` | `true`/`false` (default `false`). `true` for lists from `discoverResiduals.sh --mc`: one map per run, no time cut, `SLOT_LENGTH_MIN` ignored — see "MC productions" under Step 1. | | `PACKAGESPEC` | O2/O2DPG software tag to run under — set this to the async-reco tag matching your data. Also determines whether `MACRO_SOURCE=official` can find the macros (see "Where the macros come from" above). | | `ASUSER` | AliEn account this job runs/is charged as, e.g. `pwg_pp` (your account must already be a member of that group) instead of your own personal account/quota. Determines both which quota pays for the job and where its `${ALIEN_JDL_OUTPUTDIR}` lands (see Step 3). Verify on alimonitor that the first job is really attributed correctly before relying on it for a big batch. | | `MACRO_SOURCE` / `MACRO_LOCAL_PATH` | See "Where the macros come from" above. | @@ -259,6 +356,32 @@ Note: the macro no longer reads IDC scalers from CCDB directly (`useCTPLumi`/`nS are gone from its signature) — luminosity values are joined back offline by `SmoothingExtrapolate.C` in stage 2 instead (its own `USE_CTP_LUMI`, see Step 4). +#### Hybrid time sampling (pp) + +pp fills need denser time sampling right after the fill starts, while space charge is still building up, +than for the rest of the fill — the same physics motivation as the old SLURM-era +`makeStaggerd.py`/`combinedSampling.py` hybrid tooling this mirrors. Set `MAKE_HYBRID_MAPS=true` and +`stageSlotsForGrid.py` cuts `SLOT_LENGTH_MIN`-length ("fine") windows for the first `FINE_WINDOW_MIN` +minutes of each **fill**, `SLOT_LENGTH_MIN_COARSE`-length ("coarse") windows for the rest of it — instead +of `SLOT_LENGTH_MIN` uniformly everywhere. + +The cutoff is anchored to the **fill's own start**, not each run's — deliberately, since a pp fill +routinely spans several runs, and the build-up being sampled densely is a property of the fill, not of +where a run happens to start within it. Concretely: the fill's start is the earliest `startTimestampMs` +among that fill's runs present under `listsDir` for *this* submission (same "what's actually in this +production" scoping `RUNS_FILTER` already uses elsewhere in this doc) — so a run that starts 70 minutes +into its fill gets windowed entirely coarse, even though it is only 10 minutes old itself, while a run +straddling the cutoff gets fine windows before it and coarse windows after, with no runt window on either +side of the boundary (each region absorbs its own short trailing remainder independently, same as the +non-hybrid case). + +This needs every selected run's `discoverResiduals.sh` batch-file line to have carried the optional +` ` columns (see Step 1) — that's what produces each run's `.fillinfo.txt` +sidecar. If any selected run is missing it, `stageSlotsForGrid.py` fails loudly and locally, before +anything is submitted, listing exactly which run(s) — not a silent fall-back to per-run windowing, which +would silently change the physics. A non-hybrid submission (`MAKE_HYBRID_MAPS=false`, the default) needs +none of this — plain 2-column batch-file rows are enough, same as before this feature existed. + Then: ```bash @@ -378,7 +501,32 @@ logical slot: FT_voxRes.__.TDof.it0.GausSmooth.Extrapolation.root voxResQA.voxRes.__.TDof.it0.GausSmooth.Extrapolation_1D.root voxResQA.voxRes.__.TDof.it0.GausSmooth.Extrapolation_2D.root + stage1Source.txt + mapMeta.txt +``` + +`stage1Source.txt` is a one-line pointer to the stage-1 map this slot was built from (written right +after smoothing, so it exists even if the FastTransform step fails). `mapMeta.txt` is the per-slot +summary, one `key=value` per line, written only once `FT_*.root` is staged: + ``` +stage1= +stage2= +meanIDC=... +medianIDC=... +meanCTP=... +medianCTP=... +tsStart= +tsEnd= +fill= +``` + +The IDC values come from the smoothed map's UserInfo (offline IDC join in `SmoothingExtrapolate.C`) and +the CTP values from stage 1. `tsStart`/`tsEnd` are the manifest window in ms, the same values as in the +file names. `fill` is looked up in `GLO/Config/GRPLHCIF` at `tsStart`; it is `-1` if the CCDB lookup +failed. + +Both files are non-fatal: if writing or staging one fails, the slot still succeeds. Stage 1's and stage 2's output for the same slot are *not* grouped under one tidy parent directory — that's the direct cost of letting the JDL own output placement instead of managing it ourselves (see diff --git a/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh b/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh index b2d714c4b..52e765e6a 100755 --- a/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh +++ b/DATA/production/configurations/mapCreationGrid/discoverResiduals.sh @@ -9,10 +9,30 @@ # # Usage: # ./discoverResiduals.sh [--calib-pass ] +# ./discoverResiduals.sh --mc [--mc-cycle ] +# +# --mc: the batch file lists MC productions instead of data periods. Field 1 is then the MC production +# tag and field 2 the anchor run, e.g. "LHC26j19_residuals 568336", and the search path becomes +# /alice/sim///// (cycle = --mc-cycle, default 0) instead of +# /alice/data///// -- or rather its ResidualTPCmerging/Stage_0/ subdir, +# the production's merged residuals, whenever that holds any (see per_run). MC residual files carry no time label in their +# names, so makeResidualLists.sh writes them as one timeless list per run; submit those with +# MC_MODE=true in submitMapCreation.sh (stageSlotsForGrid.py --mc). # # Batch file format (one run per line, whitespace-separated): -# +# [ ] # e.g.: LHC26ak 572557 +# LHC26ac.b5p 569945 11500 1773347865674 +# +# The last two columns are OPTIONAL and only needed for hybrid map creation (pp: fine-length maps for +# the first part of a fill, coarse-length for the rest -- see submitMapCreation.sh's MAKE_HYBRID_MAPS). +# When given, both must be present together: is the LHC fill number (shared by every run in it, +# used to anchor the fine/coarse cutoff to the FILL's start rather than each run's own start -- a pp +# fill routinely spans several runs), is that run's start-of-run (SOR) timestamp, +# UTC milliseconds (13 digits), same convention as the old SLURM-era makeStaggerd.py's metadata file. +# When present, per_run() writes them to /.fillinfo.txt, which +# stageSlotsForGrid.py --hybrid reads later, at submission time. Omit both columns for a +# non-hybrid production -- nothing downstream needs them then. # # PERIOD is derived from the first field by stripping from the first '.' onward # (e.g. LHC26ab.b5p → LHC26ab — a trailing '.xyz' suffix is tolerated but not @@ -29,12 +49,20 @@ set -euo pipefail # --------------------------------------------------------------------------- BATCH_FILE="" CALIB_PASS="cpass2_residuals" +CALIB_PASS_SET=0 +MC_MODE=0 +MC_CYCLE="" while [[ $# -gt 0 ]]; do case "$1" in --calib-pass) [[ $# -lt 2 ]] && { echo "ERROR: --calib-pass requires an argument"; exit 1; } - CALIB_PASS="$2"; shift 2 ;; + CALIB_PASS="$2"; CALIB_PASS_SET=1; shift 2 ;; + --mc) + MC_MODE=1; shift ;; + --mc-cycle) + [[ $# -lt 2 ]] && { echo "ERROR: --mc-cycle requires an argument"; exit 1; } + MC_CYCLE="$2"; shift 2 ;; -*) echo "ERROR: Unknown option: $1"; exit 1 ;; *) @@ -43,8 +71,19 @@ while [[ $# -gt 0 ]]; do esac done -[[ -z "${BATCH_FILE}" ]] && { echo "Usage: $0 [--calib-pass ]"; exit 1; } +[[ -z "${BATCH_FILE}" ]] && { echo "Usage: $0 [--calib-pass ] | [--mc [--mc-cycle ]]"; exit 1; } [[ -f "${BATCH_FILE}" ]] || { echo "ERROR: batch file not found: ${BATCH_FILE}"; exit 1; } +# The data and MC paths share no component after the year, so a flag of the other mode is a mistake, +# not something to ignore. +if (( MC_MODE )); then + (( CALIB_PASS_SET )) && { echo "ERROR: --calib-pass is a data-only option, the MC path has no calib-pass level"; exit 1; } + MC_CYCLE="${MC_CYCLE:-0}" + [[ "${MC_CYCLE}" =~ ^[0-9]+$ ]] || { echo "ERROR: --mc-cycle must be numeric, got '${MC_CYCLE}'"; exit 1; } +else + [[ -n "${MC_CYCLE}" ]] && { echo "ERROR: --mc-cycle needs --mc"; exit 1; } + # Never used in data mode, but per_run takes it positionally and GNU parallel would drop an empty word. + MC_CYCLE="0" +fi # --------------------------------------------------------------------------- # Parse the batch file into one " " entry per line @@ -53,7 +92,7 @@ done # token on every row, so a batch legitimately spanning two periods must not silently inherit the # first row's period for all of its runs (that produced a valid-looking /alice/data//// query, which just finds nothing and degrades into a per-run warning). -mapfile -t ENTRIES < <(awk 'NF > 0 && !/^[[:space:]]*#/ {print $1, $2}' "${BATCH_FILE}") +mapfile -t ENTRIES < <(awk 'NF > 0 && !/^[[:space:]]*#/ {print $1, $2, $3, $4}' "${BATCH_FILE}") [[ ${#ENTRIES[@]} -gt 0 ]] || { echo "ERROR: no runs found in ${BATCH_FILE}"; exit 1; } # Validate every entry up front -- a malformed row should stop the batch here, not surface as one @@ -61,14 +100,24 @@ mapfile -t ENTRIES < <(awk 'NF > 0 && !/^[[:space:]]*#/ {print $1, $2}' "${BATCH RUNS=() PERIODS_SEEN=() for entry in "${ENTRIES[@]}"; do - read -r _tok _run <<< "${entry}" + read -r _tok _run _fill _sor <<< "${entry}" [[ -n "${_tok}" && -n "${_run}" ]] \ - || { echo "ERROR: malformed line in ${BATCH_FILE} (need ' '): '${entry}'"; exit 1; } + || { echo "ERROR: malformed line in ${BATCH_FILE} (need ' [ ]'): '${entry}'"; exit 1; } [[ "${_run}" =~ ^[0-9]+$ ]] \ || { echo "ERROR: run number is not numeric in ${BATCH_FILE}: '${entry}'"; exit 1; } # Field 1 example: LHC26ab.b5p → PERIOD=LHC26ab, YEAR=2026 [[ "${_tok%%.*}" =~ LHC([0-9]{2}) ]] \ || { echo "ERROR: cannot extract year from period token '${_tok}' (line: '${entry}')"; exit 1; } + # fill/startTimestampMs are optional, but only as a pair -- stageSlotsForGrid.py --hybrid needs + # both or neither, so one without the other is a malformed row, not a partially-specified one. + if [[ -n "${_fill}" || -n "${_sor}" ]]; then + [[ -n "${_fill}" && -n "${_sor}" ]] \ + || { echo "ERROR: fill and startTimestampMs must both be given or both omitted in ${BATCH_FILE}: '${entry}'"; exit 1; } + [[ "${_fill}" =~ ^[0-9]+$ ]] \ + || { echo "ERROR: fill number is not numeric in ${BATCH_FILE}: '${entry}'"; exit 1; } + [[ "${_sor}" =~ ^[0-9]{13}$ ]] \ + || { echo "ERROR: startTimestampMs must be a 13-digit UTC-ms timestamp in ${BATCH_FILE}: '${entry}'"; exit 1; } + fi RUNS+=("${_run}") PERIODS_SEEN+=("${_tok%%.*}") done @@ -93,8 +142,13 @@ MAKE_LISTS_SCRIPT="${SCRIPT_DIR}/makeResidualLists.sh" # --------------------------------------------------------------------------- echo "Batch file : ${BATCH_FILE}" -echo "Period(s) : ${PERIODS_UNIQ[*]}" -echo "Calib pass : ${CALIB_PASS}" +if (( MC_MODE )); then + echo "MC prod(s) : ${PERIODS_UNIQ[*]}" + echo "MC cycle : ${MC_CYCLE}" +else + echo "Period(s) : ${PERIODS_UNIQ[*]}" + echo "Calib pass : ${CALIB_PASS}" +fi echo "Runs (${#RUNS[@]}): ${RUNS[*]}" echo @@ -113,6 +167,10 @@ per_run() { local CALIB_PASS="$3" local WORK_DIR="$4" local MAKE_LISTS_SCRIPT="$5" + local MC_MODE="$6" # 1: field 1 is an MC production tag, search /alice/sim + local MC_CYCLE="$7" # MC production cycle (path level after the tag), unused for data + local FILL="$8" # field 3 of this batch-file row, empty if omitted + local SOR="$9" # field 4 of this batch-file row, empty if omitted # Period/year come from THIS row, so a multi-period batch resolves each run against its own period. # Already validated in the caller's pre-flight loop, so a parse failure here would be a bug. @@ -124,12 +182,46 @@ per_run() { local RUN_DIR="${WORK_DIR}/${RUN}" mkdir -p "${RUN_DIR}" + # Only for hybrid map creation (stageSlotsForGrid.py --hybrid reads this later); a non-hybrid batch + # file omits fill/SOR and no sidecar is written -- nothing downstream needs one then. + if [[ -n "${FILL}" && -n "${SOR}" ]]; then + printf '%s\t%s\n' "${FILL}" "${SOR}" > "${RUN_DIR}/${RUN}.fillinfo.txt" + fi + local ALIEN_PATH="/alice/data/${YEAR}/${PERIOD}/${RUN}/${CALIB_PASS}/" + if (( MC_MODE )); then + ALIEN_PATH="/alice/sim/${YEAR}/${PERIOD}/${MC_CYCLE}/${RUN}/" + fi echo "========================================" echo "Processing run ${RUN} (period ${PERIOD}, year ${YEAR})" echo "========================================" + # MC productions merge their per-job residuals into ResidualTPCmerging/Stage_0//o2tpc_residuals.root + # (~150 jobs per file, same trees). Read those when present: the recursive search over the run dir + # would otherwise list every residual twice (per-job + merged), and each alien:// open costs ~7 s, + # so ~40 merged files instead of thousands of per-job ones. A merge job that failed or is still + # running leaves its Stage_0// without the file, and its share of the statistics is then + # missing -- warned about, not fatal. Without any merged file, fall back to the per-job files (the + # recursive search then cannot pick up merged ones, there are none). + if (( MC_MODE )); then + local MERGED_PATH="${ALIEN_PATH}ResidualTPCmerging/Stage_0/" + local nMerged nMergeJobs + # grep on the file name, not just non-empty lines: alien.py prints its "Not a directory" error + # for a missing path as a line too. + nMerged=$(alien.py find "${MERGED_PATH}" "o2tpc_residuals*.root" 2>/dev/null | grep -c 'o2tpc_residuals[^/]*\.root$' || true) + if (( nMerged > 0 )); then + nMergeJobs=$(alien.py ls "${MERGED_PATH}" 2>/dev/null | grep -c '/$' || true) + echo " [${RUN}] Using merged residuals: ${nMerged} file(s) in ${nMergeJobs} merge job dir(s) under ${MERGED_PATH}" + if (( nMerged < nMergeJobs )); then + echo " [${RUN}] WARNING: only ${nMerged}/${nMergeJobs} merge jobs produced o2tpc_residuals.root (failed or still running) -- this run's map misses that share of the statistics." + fi + ALIEN_PATH="${MERGED_PATH}" + else + echo " [${RUN}] WARNING: no merged residuals under ${MERGED_PATH} -- falling back to the per-job files under ${ALIEN_PATH}" + fi + fi + echo " [${RUN}] Calling makeResidualLists.sh (alien path: ${ALIEN_PATH})..." ( cd "${RUN_DIR}" @@ -155,17 +247,18 @@ echo "========================================" echo "Launching ${#RUNS[@]} runs with N_PARALLEL=${N_PARALLEL}..." echo "========================================" -# Fan out over the full " " entries, not bare run numbers, so each job carries its -# own period. --colsep splits each entry into {1}=period token and {2}=run; note that under --colsep -# a bare {} means column 1 ONLY, so both columns must be passed explicitly. --tagstring uses {2} to -# keep the log prefix to just the run number. -# shellcheck disable=SC1083 # {1}/{2} are GNU parallel replacement strings, not brace expansions +# Fan out over the full " [ ]" entries, not bare run +# numbers, so each job carries its own period (and fill/SOR, when given). --colsep splits each entry +# into {1}=period token, {2}=run, {3}=fill, {4}=SOR -- {3}/{4} come through as empty strings for a +# 2-column row (awk already padded ENTRIES to 4 fields, see above), which per_run treats as "no +# fillinfo sidecar for this run". --tagstring uses {2} to keep the log prefix to just the run number. +# shellcheck disable=SC1083 # {1}/{2}/{3}/{4} are GNU parallel replacement strings, not brace expansions parallel \ --jobs "${N_PARALLEL}" \ --line-buffer \ --colsep ' ' \ --tagstring "[{2}]" \ - per_run {1} {2} "${CALIB_PASS}" "${WORK_DIR}" "${MAKE_LISTS_SCRIPT}" \ + per_run {1} {2} "${CALIB_PASS}" "${WORK_DIR}" "${MAKE_LISTS_SCRIPT}" "${MC_MODE}" "${MC_CYCLE}" {3} {4} \ ::: "${ENTRIES[@]}" # --------------------------------------------------------------------------- diff --git a/DATA/production/configurations/mapCreationGrid/postProcessJob.sh b/DATA/production/configurations/mapCreationGrid/postProcessJob.sh index e555d09cc..1c038d91b 100755 --- a/DATA/production/configurations/mapCreationGrid/postProcessJob.sh +++ b/DATA/production/configurations/mapCreationGrid/postProcessJob.sh @@ -230,13 +230,27 @@ cmd="(time root.exe -b -q -l -x '${QA_MACRO_PATH}+Og(\"ls ${OUTPUT_MAP_SMOOTH}\" echo "$(date) ${cmd}" eval "${cmd}" || echo "WARNING: voxResQA.C failed for this slot -- QA plots will be missing, but this doesn't affect ${OUTPUT_MAP_SMOOTH} itself" >&2 +# voxResQA.C names its c1D/c2D canvases (and so their PNGs) "c1D_", where +# fileIDName is derived from the input file name and just repeats what QA_SUFFIX already says. That +# pushes the full LFN (OutputDir + file name) past AliEn's 255-character limit for two-digit TD +# indices: upload succeeds, catalogue registration fails, and the subjob ends in ERROR_SV. Drop the +# redundant part, same naming as cdXZvsRow.png. +for prefix in c1D c2D; do + for png in "${prefix}"_*"${QA_SUFFIX}.png"; do + [[ -f "${png}" ]] && mv -f "${png}" "${prefix}${QA_SUFFIX}.png" + done +done + QA_FILE_1D="voxResQA${QA_SUFFIX}_1D.root" QA_FILE_2D="voxResQA${QA_SUFFIX}_2D.root" # --- [2/2] Final O2 FastTransform --- OUTPUT_FT="FT_${OUTPUT_MAP_SMOOTH}" -cmd="(time root.exe -b -q -l -x '${FASTTRANSFORM_MACRO_PATH}+Og(\"${OUTPUT_MAP_SMOOTH}\", \"${OUTPUT_FT}\", ${USE_SMOOTHED}, ${USE_CTP_LUMI})')" +# One thread: the job requests a single core (submitPostProcess.sh --cores 1), and the macro's default is +# nThreads=8. invertSigns/meanIDC/meanCTP/debug are passed only to reach nThreads, at the macro's defaults. +FT_NTHREADS=1 +cmd="(time root.exe -b -q -l -x '${FASTTRANSFORM_MACRO_PATH}+Og(\"${OUTPUT_MAP_SMOOTH}\", \"${OUTPUT_FT}\", ${USE_SMOOTHED}, ${USE_CTP_LUMI}, true, 0, 0, 0, ${FT_NTHREADS})')" echo "$(date) ${cmd}" eval "${cmd}" @@ -255,6 +269,71 @@ if ! alien.py cp -f "file:${OUTPUT_FT}" "${ALIEN_OUTPUT_DIR}/${OUTPUT_FT}"; then fi echo "Staged ${OUTPUT_FT} to ${ALIEN_OUTPUT_DIR}" +# --- Per-slot metadata (mapMeta.txt): one key=value per line, for downstream bookkeeping without opening +# any ROOT file. Rates are read from the SMOOTHED map's UserInfo (the stage-1 map still carries the +# meanIDC/medianIDC=0 placeholders; SmoothingExtrapolate.C joins in the real values), timestamps are the +# manifest window (same values as in the file names; 0 0 for MC), fill number from GLO/Config/GRPLHCIF at +# the window start (for MC at the map's meanTimeMS, i.e. the anchor run). Written after the FT is staged, so a mapMeta.txt never points at an FT that doesn't exist. +# Generated inline rather than shipped as a separate .C: grid_submit.sh only ships this one script. +# Non-fatal, like stage1Source.txt -- bookkeeping, not a deliverable. +MAP_META_FILE="mapMeta.txt" +cat > writeMapMeta.C <<'EOF' +#include +#include +#include "TFile.h" +#include "TTree.h" +#include "TList.h" +#include "CCDB/BasicCCDBManager.h" +#include "DataFormatsParameters/GRPLHCIFData.h" + +void writeMapMeta(const char* mapFile, const char* outFile, const char* stage1, const char* stage2, long tsStart, long tsEnd) +{ + TFile f(mapFile); + TTree* t = nullptr; + f.GetObject("voxResTree", t); + TList* userInfo = t ? t->GetUserInfo() : nullptr; + auto get = [userInfo](const char* key) { + auto* o = userInfo ? userInfo->FindObject(key) : nullptr; + return o ? std::string(o->GetTitle()) : std::string("NA"); + }; + + auto& ccdbmgr = o2::ccdb::BasicCCDBManager::instance(); + ccdbmgr.setURL("http://alice-ccdb.cern.ch"); + ccdbmgr.setFatalWhenNull(false); + // MC windows are 0 0 (no time label); the CCDB at timestamp 0 returns the first GRPLHCIF ever stored, + // i.e. a wrong fill. Use the map's mean time (the anchor run's) instead, and -1 if that is missing too. + long tsFill = tsStart; + if (tsFill <= 0) { + const std::string meanTime = get("meanTimeMS"); + tsFill = (meanTime == "NA") ? 0 : std::stol(meanTime); + } + auto* grpLHC = (tsFill > 0) ? ccdbmgr.getForTimeStamp("GLO/Config/GRPLHCIF", tsFill) : nullptr; + const int fill = grpLHC ? grpLHC->getFillNumber() : -1; + + std::ofstream out(outFile); + out << "stage1=" << stage1 << "\n" + << "stage2=" << stage2 << "\n" + << "meanIDC=" << get("meanIDC") << "\n" + << "medianIDC=" << get("medianIDC") << "\n" + << "meanCTP=" << get("meanCTP") << "\n" + << "medianCTP=" << get("medianCTP") << "\n" + << "tsStart=" << tsStart << "\n" + << "tsEnd=" << tsEnd << "\n" + << "fill=" << fill << "\n"; +} +EOF +cmd="root.exe -b -q -l -x 'writeMapMeta.C+(\"${OUTPUT_MAP_SMOOTH}\", \"${MAP_META_FILE}\", \"${STAGE1_OUTPUT_LOCATION}/${STAGE1_OUTPUT_FILE}\", \"${ALIEN_OUTPUT_DIR}/${OUTPUT_FT}\", ${firstTSIter}, ${lastTSIter})'" +echo "$(date) ${cmd}" +if eval "${cmd}" && [[ -s "${MAP_META_FILE}" ]]; then + cat "${MAP_META_FILE}" + alien.py cp -f "file:${MAP_META_FILE}" "${ALIEN_OUTPUT_DIR}/${MAP_META_FILE}" || \ + echo "WARNING: failed to stage ${MAP_META_FILE} -- metadata only, not fatal" >&2 +else + echo "WARNING: writeMapMeta.C failed -- ${MAP_META_FILE} will be missing for this slot, not fatal" >&2 +fi +# Same *.txt JDL_OUTPUT collision as stage1Source.txt above -- remove the local copy once staged. +rm -f "${MAP_META_FILE}" + # QA plots -- non-fatal staging, same reasoning as the non-fatal QA run itself above: if voxResQA.C # failed or produced nothing, there's nothing to stage, and that shouldn't fail this otherwise-good slot. for qaFile in "${QA_FILE_1D}" "${QA_FILE_2D}"; do diff --git a/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py b/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py index eae34838d..a1978fae6 100755 --- a/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py +++ b/DATA/production/configurations/mapCreationGrid/stageSlotsForGrid.py @@ -76,9 +76,34 @@ def parse_args(argv): "including ones left from an earlier batch, which would be recalibrated from " "scratch at hours per slot. Naming a run that has no lists is an error.") p.add_argument("--slot-length-min", type=float, default=0, - help="target length of ONE map, in minutes. 0 = one window per run.") + help="target length of ONE map, in minutes. 0 = one window per run. With --hybrid, " + "this is the FINE length (see --hybrid).") p.add_argument("--overlap-margin-ms", type=int, default=1000, help="a file whose label overlaps a window by <= this is not assigned to it") + p.add_argument("--hybrid", action="store_true", + help="hybrid time sampling (pp): --slot-length-min windows for the first " + "--fine-window-min minutes of each FILL, --slot-length-min-coarse windows for " + "the rest of that fill. The cutoff is anchored to the FILL's start (earliest " + "SOR among that fill's runs present under --lists-dir), not each run's own " + "start -- a pp fill routinely spans several runs, and the space-charge " + "build-up this targets is a fill-level effect. Needs a .fillinfo.txt " + "sidecar (one line, ' ') next to every selected run's " + "index, written by discoverResiduals.sh when its batch file carries the " + "optional fill/SOR columns -- missing for a selected run is a hard error.") + p.add_argument("--slot-length-min-coarse", type=float, default=None, + help="COARSE window length in minutes, for the part of a fill at or after " + "--fine-window-min. Required with --hybrid (no default -- forces a deliberate " + "choice); same '0 = one window' semantics as --slot-length-min, applied " + "independently within the coarse region. Ignored without --hybrid.") + p.add_argument("--fine-window-min", type=float, default=60, + help="duration of the fine-sampling window, in minutes, measured from the FILL's " + "start. Only used with --hybrid.") + p.add_argument("--mc", action="store_true", + help="MC input (discoverResiduals.sh --mc). MC residual files carry no time label, " + "so there is nothing to window on: each run becomes ONE window holding all of " + "its files, with bounds 0 0, which switches off staticMapCreatorCPM.C's per-TF " + "time cut (it only applies for firstTFTime > 0). --slot-length-min is ignored; " + "--hybrid is an error. A list with time-labelled (data) files is an error too.") p.add_argument("--window-lists", action="store_true", help="also write one plain file list per window, for inspecting which files a " "window got. OFF by default: on Lustre each is a metadata round trip and 120 " @@ -105,13 +130,13 @@ def read_slot_files(index_path): return out -def plan_windows(run_first, run_last, slot_ms, margin_ms): - """Whole steps of slot_ms across [run_first, run_last]. +def plan_windows_region(seg_first, seg_last, slot_ms, margin_ms): + """Whole steps of slot_ms across [seg_first, seg_last]. A trailing remainder shorter than half a step is absorbed into the last window rather than left as - a runt that would fail minTracksPerSlice. Returns [(winFirst, winLast, effectiveMargin)]. + a runt that would fail minTracksPerSlice. Returns [(winFirst, winLast, effectiveMargin, slot_ms)]. """ - span = run_last - run_first + span = seg_last - seg_first if slot_ms <= 0 or span <= 0: n_win = 1 else: @@ -122,21 +147,47 @@ def plan_windows(run_first, run_last, slot_ms, margin_ms): wins = [] for w in range(n_win): if n_win == 1: - ws, we = run_first, run_last + ws, we = seg_first, seg_last else: - ws = run_first + w * slot_ms - we = run_last if w + 1 == n_win else run_first + (w + 1) * slot_ms - 1 + ws = seg_first + w * slot_ms + we = seg_last if w + 1 == n_win else seg_first + (w + 1) * slot_ms - 1 # Clamp the margin to a tenth of the window so a short window can never be emptied by it. - wins.append((ws, we, min(margin_ms, (we - ws) // 10))) + wins.append((ws, we, min(margin_ms, (we - ws) // 10), slot_ms)) return wins -def assign(paths, wins, run_first, slot_ms): +def plan_windows(run_first, run_last, slot_ms, margin_ms, fine_cutoff_ts=None, slot_ms_coarse=None): + """Whole steps across [run_first, run_last]. Returns [(winFirst, winLast, effectiveMargin, slot_ms)]. + + Without fine_cutoff_ts this is exactly the old single-region behavior (one call to + plan_windows_region at slot_ms). With it, the run is split into up to two independent regions -- + [run_first, cutoff) at slot_ms (FINE) and [cutoff, run_last] at slot_ms_coarse (COARSE) -- each + planned on its own, so each region's own short trailing remainder is absorbed into that region's + own last window and never crosses the fine/coarse boundary. cutoff is clamped into + [run_first, run_last + 1] so a run entirely before/after its fill's fine cutoff correctly collapses + to a single all-fine or all-coarse region instead of producing an empty one. + """ + if fine_cutoff_ts is None: + return plan_windows_region(run_first, run_last, slot_ms, margin_ms) + + cutoff = max(run_first, min(run_last + 1, fine_cutoff_ts)) + wins = [] + if cutoff > run_first: + wins += plan_windows_region(run_first, min(run_last, cutoff - 1), slot_ms, margin_ms) + if cutoff <= run_last: + wins += plan_windows_region(cutoff, run_last, slot_ms_coarse, margin_ms) + return wins + + +def assign(paths, wins): """Fan every file out to the windows its label overlaps by more than that window's margin. - Only the windows a label can possibly touch are examined, derived directly from the label bounds, - rather than scanning all of them per file: with 10-minute labels and 5-minute windows that is 2-3 - candidates instead of the whole run's worth. Returns (buckets, grazed, orphans). + Scans every window per file rather than deriving a candidate index range from a uniform stride: + with --hybrid, a run's windows are no longer an even step from run_first (the fine and coarse + regions each have their own slot_ms), so that arithmetic shortcut no longer applies. Windows per + run are few enough (order 10s-100s) for the brute-force scan to cost nothing measurable -- it also + collapses the hybrid and non-hybrid paths into one, simpler code to trust. Returns + (buckets, grazed, orphans). """ n_win = len(wins) buckets = [[] for _ in range(n_win)] @@ -144,17 +195,8 @@ def assign(paths, wins, run_first, slot_ms): orphans = [] for lo, hi, line in paths: - if slot_ms > 0 and n_win > 1: - # Windows are contiguous steps of slot_ms from run_first, so a label maps to an index - # range. +/-1 guards the boundaries and the last window, which extends to run_last. - first = max(0, (lo - run_first) // slot_ms - 1) - last = min(n_win - 1, (hi - run_first) // slot_ms + 1) - else: - first, last = 0, n_win - 1 - hit = False - for i in range(first, last + 1): - ws, we, margin = wins[i] + for i, (ws, we, margin, _slot_ms) in enumerate(wins): if hi < ws or lo > we: continue if min(hi, we) - max(lo, ws) > margin: @@ -168,6 +210,20 @@ def assign(paths, wins, run_first, slot_ms): return buckets, grazed, orphans +def read_fillinfo(path): + """Parse a .fillinfo.txt sidecar: one line, ' '. Returns (fill, sor) + as (str, int), or None if the file is missing/malformed.""" + if not path.is_file(): + return None + parts = path.read_text().split() + if len(parts) != 2: + return None + fill, sor = parts + if not sor.isdigit(): + return None + return fill, int(sor) + + def main(argv): args = parse_args(argv) @@ -184,6 +240,47 @@ def main(argv): slot_ms = int(round(args.slot_length_min * 60_000)) wanted = [r for r in args.runs.split(",") if r] if args.runs else [] + if args.mc and args.hybrid: + log("ERROR: --mc and --hybrid are exclusive -- MC residual files carry no time label to window on") + return 1 + if args.mc and args.slot_length_min: + log(f"NOTE: --mc -- ignoring --slot-length-min {args.slot_length_min}, one window per run") + + if args.hybrid and args.slot_length_min_coarse is None: + log("ERROR: --hybrid requires --slot-length-min-coarse (no default -- pass 0 explicitly if you " + "really want a single coarse window per fill's post-fine tail)") + return 1 + slot_ms_coarse = int(round(args.slot_length_min_coarse * 60_000)) if args.hybrid else None + fine_ms = int(round(args.fine_window_min * 60_000)) + + # --hybrid: read every selected run's .fillinfo.txt sidecar (written by discoverResiduals.sh + # when its batch file carries the optional fill/SOR columns) and derive each FILL's start as the + # earliest SOR among that fill's runs seen HERE, under --lists-dir -- i.e. among the runs this + # invocation is actually processing (see --runs), same scoping RUNS_FILTER already uses elsewhere + # in this pipeline. A run missing its sidecar is a hard error, not a silent fall-back to per-run + # windowing: that would silently change the physics (see the module-level --hybrid help text). + fill_of_run = {} + fill_start = {} + if args.hybrid: + missing_fillinfo = [] + for idx in index_files: + run = idx.name[: -len(".residuals_lists.txt")] + if wanted and run not in wanted: + continue + info = read_fillinfo(idx.parent / f"{run}.fillinfo.txt") + if info is None: + missing_fillinfo.append(run) + continue + fill, sor = info + fill_of_run[run] = fill + fill_start[fill] = min(fill_start.get(fill, sor), sor) + if missing_fillinfo: + log(f"ERROR: --hybrid needs a .fillinfo.txt sidecar (fill + SOR) for every selected " + f"run, missing for: {' '.join(missing_fillinfo)}") + log(" Re-run discoverResiduals.sh with the columns in the " + "batch file for these runs.") + return 1 + # Rebuilt from scratch each time so it always reflects THIS invocation's slot length and margin # rather than a stale earlier split. window_dir = Path(args.window_dir) if args.window_dir else \ @@ -221,25 +318,40 @@ def main(argv): _text = slot_file.read_text() t_read += time.monotonic() - _t _t = time.monotonic() - parsed = [] - for line in _text.splitlines(): - m = SIG_RE.search(line) - if m: - parsed.append((int(m.group(1)), int(m.group(2)), line)) - if not parsed: - log(f" WARNING: no parseable residual signature in {slot_file}, skipping") - continue + if args.mc: + files = [l.strip() for l in _text.splitlines() if l.strip()] + # A time label means data: windowing it as MC would silently drop the per-TF time cut + # and merge a whole run into one map. + if any(SIG_RE.search(l) for l in files): + log(f"ERROR: --mc, but {slot_file} holds time-labelled (data) residual files -- " + f"drop --mc for data") + return 1 + if not files: + log(f" WARNING: no residual files in {slot_file}, skipping") + continue + wins = [(0, 0, 0, 0)] + buckets, grazed, orphans = [files], [0], [] + else: + parsed = [] + for line in _text.splitlines(): + m = SIG_RE.search(line) + if m: + parsed.append((int(m.group(1)), int(m.group(2)), line)) + if not parsed: + log(f" WARNING: no parseable residual signature in {slot_file}, skipping " + f"(MC lists need --mc)") + continue - t_calc += time.monotonic() - _t - _t = time.monotonic() - run_first = min(p[0] for p in parsed) - run_last = max(p[1] for p in parsed) - wins = plan_windows(run_first, run_last, slot_ms, args.overlap_margin_ms) - buckets, grazed, orphans = assign(parsed, wins, run_first, slot_ms) + run_first = min(p[0] for p in parsed) + run_last = max(p[1] for p in parsed) + fine_cutoff_ts = fill_start[fill_of_run[run]] + fine_ms if args.hybrid else None + wins = plan_windows(run_first, run_last, slot_ms, args.overlap_margin_ms, + fine_cutoff_ts=fine_cutoff_ts, slot_ms_coarse=slot_ms_coarse) + buckets, grazed, orphans = assign(parsed, wins) n_win = len(wins) t_calc += time.monotonic() - _t - for w, ((ws, we, _), files) in enumerate(zip(wins, buckets)): + for w, ((ws, we, _margin, w_slot_ms), files) in enumerate(zip(wins, buckets)): if grazed[w]: log(f" (skipped {grazed[w]} file(s) grazing this window)") if not files: @@ -251,7 +363,10 @@ def main(argv): out.write_text("\n".join(files) + "\n") t_write += time.monotonic() - _t manifest_lines.append(f"{run}\t{w}\t{n_win}\t{ws}\t{we}\t{','.join(files)}") - log(f" window {w}/{n_win} [{ws},{we}] {(we - ws) // 1000}s {len(files)} file(s)") + kind = f" ({'FINE' if w_slot_ms == slot_ms else 'COARSE'})" if args.hybrid else "" + if args.mc: + kind = " (MC, no time cut)" + log(f" window {w}/{n_win} [{ws},{we}] {(we - ws) // 1000}s{kind} {len(files)} file(s)") n_slots_run += 1 # Safety net for the overlap margin: every file must still land in at least one window. It diff --git a/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh b/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh index f63b5cd7b..80a7dbb93 100755 --- a/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh +++ b/DATA/production/configurations/mapCreationGrid/submitMapCreation.sh @@ -15,6 +15,10 @@ # # Slot windowing: stageSlotsForGrid.py cuts each run into windows of ~SLOT_LENGTH_MIN and writes ONE # manifest line per window, carrying the window bounds and exactly the files that can contribute to it. +# With MAKE_HYBRID_MAPS=true (pp), it instead cuts SLOT_LENGTH_MIN-length windows for the first +# FINE_WINDOW_MIN minutes of each FILL and SLOT_LENGTH_MIN_COARSE-length windows for the rest of it -- +# see the config block below. Either way the manifest line format is unchanged, and mapCreationJob.sh +# does not need to know or care which regime produced a given window. # A subjob index therefore identifies a (run, window) pair on its own, so this is a SINGLE submission -- # no loop over TimeDivisionIndex, no TIME_DIVISION placeholders substituted into the payload, and one # stage1Workdir pointer instead of one per index. (With a fixed window length, runs of different length @@ -99,7 +103,7 @@ BAD_RANGES_NAME="Merge_BadIntervals_LHC26_PbPb_DCA-0P300.txt" # Set to "" to disable bad-ranges filtering entirely for this submission # (staticMapCreatorCPM.C treats an empty badRangeList argument that way) -- a # deliberate opt-out, not the same as forgetting to set it. -BAD_RANGES_LOCAL="" # optional: local path to the bad-time-ranges file to (re-)upload this run (real +BAD_RANGES_LOCAL="" # optional: local path to the bad-time-ranges file to (re-)upload this run (real # source for the PbPb26 list above: # /lustre/alice/tpcdata/Run3/SCDprodTests/lists/badRange/PbPb26/). Leave empty to # skip the upload and just use the copy already at @@ -121,7 +125,33 @@ SLOT_LENGTH_MIN=5 # target length of ONE map, in minutes. Each run is cut i # one map per run (no splitting). This replaces the old TIME_DIVISION count: with # a fixed length, a 20 min run gives 4 maps and a 60 min run gives 12, instead of # the same count regardless of run length. Stage 2 does not need this value -- it - # reuses stage 1's manifest, which already carries the windows. + # reuses stage 1's manifest, which already carries the windows. With + # MAKE_HYBRID_MAPS=true, this is reused as the FINE map length (see below). +MAKE_HYBRID_MAPS=false # true/false. When true, submits HYBRID time sampling instead of a uniform + # SLOT_LENGTH_MIN everywhere: SLOT_LENGTH_MIN-length ("fine") windows for the + # first FINE_WINDOW_MIN minutes of each FILL, SLOT_LENGTH_MIN_COARSE-length + # ("coarse") windows for the rest of that fill. The cutoff is anchored to the + # FILL's own start (earliest startTimestampMs among that fill's runs present + # under listsDir), not each run's own start -- a pp fill routinely spans several + # runs, and the space-charge build-up this targets is a fill-level effect, not a + # per-run one. Mirrors the old SLURM-era makeStaggerd.py/combinedSampling.py + # hybrid logic. Needs every selected run's discoverResiduals.sh batch-file line + # to have carried the optional columns -- if any + # selected run is missing that sidecar, stageSlotsForGrid.py fails fast, before + # anything is submitted, listing exactly which run(s). +SLOT_LENGTH_MIN_COARSE=60 # only used when MAKE_HYBRID_MAPS=true: target length of one COARSE map, in + # minutes, for the part of a fill at or after FINE_WINDOW_MIN. Same "0 = one + # window" and trailing-remainder-absorption semantics as SLOT_LENGTH_MIN, applied + # independently within the coarse region -- a fine/coarse boundary never leaves a + # runt window on either side of it. +FINE_WINDOW_MIN=60 # only used when MAKE_HYBRID_MAPS=true: duration, in minutes, of the fine + # sampling window measured from the FILL's start. +MC_MODE=false # true/false. true for lists built by discoverResiduals.sh --mc: MC residual files + # carry no time label, so each run becomes ONE map over all of its files, with + # window bounds 0 0 (switches off the macro's per-TF time cut). SLOT_LENGTH_MIN is + # ignored, MAKE_HYBRID_MAPS=true is an error. BAD_RANGES_NAME still applies, by the + # anchor run's orbits -- set it to "" unless you want the data's bad ranges cut + # from the MC too. # --------------------------------------------------------------------------- MACRO_ALIEN_DIR="alien://${MACRO_ALIEN_DIR_BARE}" @@ -165,14 +195,28 @@ fi # lists/config) silently overwrite the first's manifest, breaking any already-running job that re-reads # it and desyncing stage 2's later subjob-index -> slot lookup. MANIFEST_NAME="mapSlots.${JOBNAME}.tsv" +STAGE_ARGS=( + --lists-dir "${LISTS_DIR}" + --alien-dir "${MACRO_ALIEN_DIR_BARE}" + --manifest-name "${MANIFEST_NAME}" + --runs "${RUNS_FILTER}" + --slot-length-min "${SLOT_LENGTH_MIN}" + --overlap-margin-ms "${OVERLAP_MARGIN_MS}" +) +if [[ "${MAKE_HYBRID_MAPS}" == "true" ]]; then + STAGE_ARGS+=( + --hybrid + --slot-length-min-coarse "${SLOT_LENGTH_MIN_COARSE}" + --fine-window-min "${FINE_WINDOW_MIN}" + ) + echo "== MAKE_HYBRID_MAPS=true -- fine=${SLOT_LENGTH_MIN}min for the first ${FINE_WINDOW_MIN}min of each fill, coarse=${SLOT_LENGTH_MIN_COARSE}min for the rest ==" +fi +if [[ "${MC_MODE}" == "true" ]]; then + STAGE_ARGS+=(--mc) + echo "== MC_MODE=true -- one map per run, no time cut (BAD_RANGES_NAME='${BAD_RANGES_NAME}') ==" +fi echo "== Building + uploading slot manifest from ${LISTS_DIR} (as ${MANIFEST_NAME}) ==" -N=$("${SCRIPT_DIR}/stageSlotsForGrid.py" \ - --lists-dir "${LISTS_DIR}" \ - --alien-dir "${MACRO_ALIEN_DIR_BARE}" \ - --manifest-name "${MANIFEST_NAME}" \ - --runs "${RUNS_FILTER}" \ - --slot-length-min "${SLOT_LENGTH_MIN}" \ - --overlap-margin-ms "${OVERLAP_MARGIN_MS}") +N=$("${SCRIPT_DIR}/stageSlotsForGrid.py" "${STAGE_ARGS[@]}") echo "N=${N} slot(s) == map(s) to be produced" [[ "${N}" -gt 0 ]] || { echo "ERROR: no slots found under ${LISTS_DIR}" >&2; exit 1; } diff --git a/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh b/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh index 4f34df062..b08799f9c 100755 --- a/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh +++ b/DATA/production/configurations/mapCreationGrid/submitPostProcess.sh @@ -145,6 +145,7 @@ GRID_SUBMIT_ARGS=( --jobname "${JOBNAME}" --topworkdir "${TOPWORKDIR}" --prodsplit "${N}" + --cores 1 # smoothing/QA macros are single-threaded and the FastTransform runs with 1 thread (postProcessJob.sh); grid_submit.sh's 8-core default trips the JobAgent CPU monitor --packagespec "${PACKAGESPEC}" --asuser "${ASUSER}" )