Skip to content

[ZEPPELIN-6773] Reduce duplicate GitHub Actions caches - #5617

Open
miinhho wants to merge 2 commits into
apache:masterfrom
miinhho:ZEPPELIN-6773-ci-cache-optimization
Open

miinhho wants to merge 2 commits into
apache:masterfrom
miinhho:ZEPPELIN-6773-ci-cache-optimization

Conversation

@miinhho

@miinhho miinhho commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

What is this PR for?

Reduce duplicate cache storage and unnecessary cache regeneration as the repository approaches the default 10 GB cache limit.

  • Replace mixed Maven/download caches with a shared Maven-only cache per OS and architecture.
  • Save Maven caches only from successful default-branch pushes: the full-reactor Java 11 build on Linux and the Windows build. Other jobs restore the baseline and download missing dependencies normally.
  • Exclude locally built Zeppelin artifacts by selecting archive roots that recursive tar cannot use to re-include the excluded directory.
  • Key Playwright browsers by the locked version and runner platform, rather than the entire npm lockfile.
  • Cache only Livy's requested Spark archive, keyed by Spark and Hadoop versions rather than unrelated POM changes.
  • Reuse identical cache steps and settings through YAML anchors and aliases.

Build/test commands and matrices remain unchanged. The new cache generations require initial downloads; existing entries remain until eviction or expiration.

Observed cache storage

After applying these changes and populating the default-branch caches in the fork, total cache storage was 7.34 GB across 11 entries, compared with the original 10.87 GB across 15 entries: 3.52 GB (32.4%) lower.

What type of PR is it?

Improvement

Todos

  • Validate the core and frontend workflows in the fork, including Linux and Windows Maven cache creation and Maven cache restoration in a subsequent attempt.
  • Compare the populated default-branch cache storage after deleting the test-branch caches.

What is the Jira issue?

How should this be tested?

  1. From the repository root, validate both workflows:
actionlint -shellcheck= -pyflakes= .github/workflows/core.yml .github/workflows/frontend.yml
  1. Run the affected core and frontend workflows with cold caches, then repeat with warm caches. Builds and tests should succeed, and consumers should restore the shared Maven baseline.
  2. Inspect saved Maven archives. Third-party dependencies and the Maven wrapper should be present; locally built Zeppelin artifacts and Playwright browsers should be absent.
  3. Confirm that PR runs do not save Maven caches, while eligible default-branch builds save a missing primary key.
  4. Confirm that unrelated POM or npm dependency changes do not change the Spark or Playwright cache keys. Their respective version changes should change the keys.

Questions:

  • Does the license files need to update? No.
  • Is there breaking changes for older versions? No.
  • Does this needs documentation? No.

@miinhho

miinhho commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

I will open this pr in forked repo and test cache size. I will re-open this when I confirm cache size reduced

@miinhho miinhho closed this Nov 13, 2026
@miinhho miinhho reopened this Nov 13, 2026
@miinhho
miinhho force-pushed the ZEPPELIN-6773-ci-cache-optimization branch from cc013ea to 7eb6880 Compare November 13, 2026 21:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant