Function paginate_pages 

pub async fn paginate_pages<T, F, Fut, P>(
    provider: &str,
    ecosystem: &str,
    noun: &str,
    name: &str,
    max_pages: u32,
    fetch_page: F,
    parse_page: P,
) -> Result<Vec<T>>
where
    F: FnMut(u32) -> Fut,
    Fut: Future<Output = Result<Bytes>>,
    P: FnMut(&Bytes) -> Result<Vec<T>>,

Drives a paginated-fetch loop against a per_page=100-shaped REST endpoint: page 1 alone, then subsequent pages in batches of up to CONCURRENCY pages.

Page 1 is always fetched by itself before any batching starts, for two reasons: most repos/projects fit in one page, so this keeps the common case at exactly the one request it took before this function gained concurrency; and an error on page 1 (bad auth, tripped rate limit, unknown project) is surfaced from a single request instead of fanning a doomed request out to CONCURRENCY pages at once.

Once page 1 is confirmed full, pages 2+ are fetched in batches of CONCURRENCY, stopping once a partial/empty page is seen or max_pages is reached. Pages within a batch are fetched concurrently, but always processed in page order — the pages dispatched after the batch’s partial page are simply discarded once found, not avoided, since by the time a batch’s first result comes back the rest of that batch’s requests are already in flight and cannot be un-sent. This bounds, but does not eliminate, extra requests: at most CONCURRENCY - 1 pages beyond the true last page may be fetched and discarded, only when that last page doesn’t land on a batch boundary. A caller that dedups “first item wins” on page order depends on out-of-order processing never happening — hence ordered buffered, not buffer_unordered.

provider/ecosystem/noun/name are forwarded to warn_if_pagination_truncated to name the API, the caller, and what is being paginated in the truncation warning.

Errors

Propagates the first error seen among fetch_page’s results (page 1’s own error, or the first in page order within a batch — any other in-flight futures in that batch are dropped), or the error from parse_page when a page’s body cannot be parsed.