Skip to content

fix(asana): page through stories, subtasks and tags and stop on null next_page - #9366

Open
ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:fix/asana-pagination
Open

ShivamSharma43 wants to merge 1 commit into
apache:mainfrom
ShivamSharma43:fix/asana-pagination

Conversation

@ShivamSharma43

Copy link
Copy Markdown
Contributor

Summary

The Asana collectors didn't page through list endpoints correctly:

  • collectStory, collectSubtask and collectTag sent limit=100 but had no PageSize and no GetNextPageCustomData, so the ApiCollector made one request per task and never sent offset. Only the first 100 stories, subtasks and tags of each task were collected. Stories include system events, so busy tasks go past 100.
  • collectTask and collectUser returned nil, nil from GetNextPageCustomData when next_page was null. That doesn't end the collection: the collector requests another page with CustomData set to nil, the query has no offset, and Asana returns page 1 again. When the last page is full, the collection starts over from the first page and saves the same raw rows again.

Asana's pagination docs: limit must be between 1 and 100, and "If there are no more pages available, next_page will be null and no offset will be provided."

Changes:

  • tasks/shared.go (new):
    • setAsanaPageQuery sets limit from the pager and, after the first page, offset.
    • getNextAsanaPageOffset returns next_page.offset, or api.ErrFinishCollect when next_page is null.
  • The task and user collectors use these helpers instead of their own closures.
  • The story, subtask and tag collectors get PageSize: 100 and use the same helpers, so they page through each task's list one page at a time.
  • tasks/shared_test.go (new): checks the query params, and checks that the next-page function returns the offset and stops when next_page is null.

Request URLs (apart from offset), raw tables and extractors are unchanged.

Does this close any open issues?

Closes #9365

Screenshots

N/A

Other Information

  • go test ./plugins/asana/tasks/: pass.
  • go build, go vet, gofmt and golangci-lint run ./plugins/asana/...: clean. The asana e2e tests need E2E_DB_URL, which I didn't run locally; they cover the extractors and converters, not the collectors.
  • No open PR or issue touches the asana plugin.
  • Suggested label: pr-type/bug-fix.
  • AI assistance disclosure: this change was prepared with the help of Claude Code (Claude Opus 5.5), and I reviewed the diff. The commit carries a Generated-by: trailer per the ASF generative tooling guidance.

🤖 Generated with Claude Code

…next_page

The story, subtask and tag collectors sent limit=100 without a PageSize
or GetNextPageCustomData, so only the first 100 stories, subtasks and
tags of each task were collected. The task and user collectors returned
nil, nil when next_page was null, which made the collector request
page 1 again without an offset whenever the last page was full.

Add getNextAsanaPageOffset, which ends the collection when next_page is
null, and setAsanaPageQuery, which sets limit and offset, and use them
in all five collectors.

Closes apache#9365

Generated-by: Claude Code (Claude Opus 5.5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 17:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Asana] Story, subtask and tag collectors stop at 100 per task; task and user collectors restart on a full last page

2 participants