[avehtari]

This PR was assisted by Claude

Summary

loo has its own is_constant() (R/helpers.R) with an absolute tolerance: abs(max(x) - min(x)) < .Machine$double.eps. Values of small magnitude are then treated as constant, so the result depends on the units of x. This PR makes the tolerance relative to the largest absolute value in x, as stan-dev/posterior#469 did for posterior::is_constant().

Problem

E_loo() uses is_constant() in the function-specific Pareto-k (.E_loo_khat_i()). When x is treated as constant, pareto_k is the k-hat of the ratios alone, and the k-hat of h * r is not computed. For small-magnitude x, pareto_k therefore changes with the scale of x:

library(loo)
LLmat <- example_loglik_matrix()
chain_id <- rep(1:2, each = nrow(LLmat) / 2)
psis_mat <- psis(-LLmat, r_eff = relative_eff(exp(LLmat), chain_id), cores = 1)
set.seed(123)
x <- matrix(rnorm(length(LLmat)), nrow = nrow(LLmat))
k <- E_loo(x, psis_mat, type = "mean", log_ratios = -LLmat)$pareto_k
k_s <- E_loo(1e-20 * x, psis_mat, type = "mean", log_ratios = -LLmat)$pareto_k
sum(abs(k_s - k) > 1e-10)   # master: 19 of 32 differ (up to 0.335); this PR (with posterior#469): 0

Change

• is_constant() in R/helpers.R: x is constant if max(x) - min(x) <= tol * max(abs(x)), or if all values are equal (also all 0 or all -Inf). The code is the same as in posterior#469.
• An infinite spread, for example c(-Inf, 2), is not constant, the same as on master. x with NA gives NA, the same as on master.
• Other behavior change: for values of magnitude above 1, values that differ by about 1 ulp are now constant, for example c(1, 1 + .Machine$double.eps).

E_loo() also calls posterior::pareto_khat() on h * r. With a posterior without stan-dev/posterior#469, that call returns NA for small-magnitude values, so E_loo() needs both changes to give a pareto_k that does not depend on the scale of x.

The other constant check in loo, the tail check in do_psis_i() (R/psis.R), works on log ratios shifted by their maximum. Log ratios do not change when the weights are scaled, so that check is not changed.

Tests

• tests/testthat/test_0_helpers.R: is_constant uses a tolerance relative to the magnitude of x: zeros, small constant values, -Inf, 1-ulp differences, small values that differ, infinite spread, NA.
• tests/testthat/test_E_loo.R: E_loo pareto_k does not depend on the scale of x, for s = 1e-20 and 1e-200. The test is skipped when the installed posterior has the absolute tolerance (posterior 1.7.1 on CRAN), because posterior::pareto_khat() then returns NA for h * r.

With posterior master (which has #469), the new tests give 6 failed expectations without this change and pass with it. With posterior 1.7.1 from CRAN, the is_constant() tests pass and the E_loo() test is skipped.

Tested versions

• loo: this branch, based on master fd910fef.
• posterior: 1.7.1 (CRAN), and master 48243f8.
• R 4.6.1, Linux x86_64.

Copyright and Licensing

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

• Code: MIT (https://opensource.org/licenses/MIT)
• Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.65%. Comparing base (d18de1f) to head (86b2c4e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #407   +/-   ##
=======================================
  Coverage   92.65%   92.65%           
=======================================
  Files          31       31           
  Lines        3240     3242    +2     
=======================================
+ Hits         3002     3004    +2     
  Misses        238      238           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 86b2c4e is merged into master:

Status	Name	Current	Incoming	Interval
✔️	loo_matrix	1.51s	1.51s	[-0.68%, +0.64%]
✔️	loo_function	1.38s	1.38s	[-0.46%, +0.48%]

Further explanation regarding interpretation and methodology can be found in the documentation.

[florence-bockting]

Thank you. The function is_constant is an exact copy of the function in posterior.

I assume you want to have the changes taking effect immediately, so I approve the PR. In general, I would recommend to use directly posterior::is_constant. But we can make this in one go for loo v3.0.0 (related Issue #408).