Repository navigation
fix(asana): page through stories, subtasks and tags and stop on null next_page - #9366
Open
ShivamSharma43 wants to merge 1 commit into
Open
ShivamSharma43 wants to merge 1 commit into
ShivamSharma43 wants to merge 1 commit into
Conversation
…next_page The story, subtask and tag collectors sent limit=100 without a PageSize or GetNextPageCustomData, so only the first 100 stories, subtasks and tags of each task were collected. The task and user collectors returned nil, nil when next_page was null, which made the collector request page 1 again without an offset whenever the last page was full. Add getNextAsanaPageOffset, which ends the collection when next_page is null, and setAsanaPageQuery, which sets limit and offset, and use them in all five collectors. Closes apache#9365 Generated-by: Claude Code (Claude Opus 5.5) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Shivam Sharma <192242476+ShivamSharma43@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Asana collectors didn't page through list endpoints correctly:
collectStory,collectSubtaskandcollectTagsentlimit=100but had noPageSizeand noGetNextPageCustomData, so theApiCollectormade one request per task and never sentoffset. Only the first 100 stories, subtasks and tags of each task were collected. Stories include system events, so busy tasks go past 100.collectTaskandcollectUserreturnednil, nilfromGetNextPageCustomDatawhennext_pagewas null. That doesn't end the collection: the collector requests another page withCustomDataset tonil, the query has nooffset, and Asana returns page 1 again. When the last page is full, the collection starts over from the first page and saves the same raw rows again.Asana's pagination docs:
limitmust be between 1 and 100, and "If there are no more pages available,next_pagewill be null and no offset will be provided."Changes:
tasks/shared.go(new):setAsanaPageQuerysetslimitfrom the pager and, after the first page,offset.getNextAsanaPageOffsetreturnsnext_page.offset, orapi.ErrFinishCollectwhennext_pageis null.PageSize: 100and use the same helpers, so they page through each task's list one page at a time.tasks/shared_test.go(new): checks the query params, and checks that the next-page function returns the offset and stops whennext_pageis null.Request URLs (apart from
offset), raw tables and extractors are unchanged.Does this close any open issues?
Closes #9365
Screenshots
N/A
Other Information
go test ./plugins/asana/tasks/: pass.go build,go vet,gofmtandgolangci-lint run ./plugins/asana/...: clean. The asana e2e tests needE2E_DB_URL, which I didn't run locally; they cover the extractors and converters, not the collectors.pr-type/bug-fix.Generated-by:trailer per the ASF generative tooling guidance.🤖 Generated with Claude Code