Why YouTube Comment Scraping Misses Replies: Two-Tier Pagination and Completeness Verification

When YouTube comment scraping misses replies, the first step is to check whether only commentThreads was read. The replies included in a thread list are not necessarily complete; threads under a video must be paginated, and replies beneath each top-level comment may also require separate pagination. Reaching the end of the first tier does not mean all replies have been retrieved.

This is not a new issue. Baskaya noted the inconsistency between reply counts and returned content in a Stack Overflow historical discussion. Older answers contain guesses from the era of Google+ integration and cannot explain today's issues. What remains genuinely useful is the question it raises: are you counting threads, counting comments, or counting the subset of replies attached to an API response?

This article explains this distinction based on official documentation as of 2026-9-5 and outlines verification methods for ingestion results. It discusses official YouTube API contracts, is not a test report for EveryInfra, and does not promise the recovery of inaccessible content.

Why Replies in commentThreads Are Incomplete

According to YouTube resource documentation, a commentThread contains a top-level comment and potentially attached replies. Top-level comments are located in snippet.topLevelComment, and reply lists are located in replies.comments; the latter may only be a subset. snippet.totalReplyCount represents the reply count for that top-level comment and cannot be directly compared against the length of the entire thread list.

Suppose a synthesized thread declares 12 replies, but the current object only attaches 5. The 5 here is neither the count the scraper can declare as finished nor a reason to pad the returned array to 12 elements. The correct action is to continue checking the reply endpoint for that parent comment rather than repeatedly requesting the same thread object in the hope that it will automatically provide more next time.

Another easily overlooked distinction is thread IDs versus top-level comment IDs. They should be read and recorded separately from their respective resources; just because two strings happen to be identical in a single sample does not mean the program can assume they are always interchangeable. Looking identical in a data model does not mean having the same purpose in an API contract.

Maintaining Pagination Progress Separately for Top-Level Comments and Replies

The official commentThreads.list supports reading threads by videoId and using the response nextPageToken to fetch the next page. The upper limit of maxResults is a maximum of 100 items per page; this is not a request that yields all comments for a video at once, nor does it mean a thread has a maximum of only 100 replies.

For a given top-level comment, follow the rules in comments.list by passing that comment ID into parentId and handling its own reply query nextPageToken. The pagination token here belongs to the reply query and cannot be fed back into the video thread query, nor can a token from another parent comment be reused.

Implementation can be divided into two task layers: video tasks discover top-level comments, and parent comment tasks handle fetching additional replies. Each parent comment maintains its own continuation position and termination reason. This ensures that a discussion with an exceptionally high number of replies does not turn other completed threads into a full restart.

Request conditions should also be stored alongside progress, such as whether searchTerms was used, what sorting method was selected, and what the target video is. If keywords were restricted from the start, subsequent completeness declarations can only cover that query scope; filter conditions cannot be omitted in export files to make them read as "all comments for this video".

Recording Completion Conditions in Results, Not Just Logs

It is recommended to separate comment content from scraping progress. Content should be deduplicated by platform and comment ID, preserving the video, parent comment association, and observation time, while progress describes where the current task stands. These are application-side design suggestions, and the following fields are not raw return fields from YouTube or EveryInfra.

A minimal progress record can answer: whether the thread list has reached the last page, which parent comments have reached the end of their replies, which stopped due to request failures or task budgets, and what the last confirmation time was. Relying solely on success: true cannot express that "the top level is finished, but one parent comment still has two unretrieved pages".

Deduplication should occur when saving content. If a process exits after writing a page, recovery might fetch the same page again; updating seen records by ID is more reliable than judging duplicates by comment text. Two people writing identical short phrases still constitute two comments, and editing the text of a single comment should not immediately turn it into two independent user opinions.

Progress must be advanced only after content is successfully saved. Recording the next page before saving the current one risks skipping data if a failure occurs midway. Allowing re-reads with deduplication is generally easier to verify than allowing silent page skipping. When cursor duplication occurs or no new IDs appear for an extended period, execution should stop and flag an anomaly; never treat repeatedly retrying the same page as valid progress.

Unfixable Discrepancies in Comment Counts

First check object hierarchies, filter conditions, and two-tier pagination, then look at error types. The official thread list documentation distinguishes between commentsDisabled, insufficient permissions, and missing videos. None of these should be wrapped as "the video currently has no comments". An empty array resulting from a successful query is not the same evidence as an empty array returned because an exception was swallowed by the client.

For quantity discrepancies, observation windows should also be recorded: the count seen on the page, thread metadata, and pagination results may not be retrieved from the exact same moment. A count difference alone cannot confirm which content is missing, let alone generate body text for it. When analyzing readable text, use actual acquired and attributable comments as samples while publicly disclosing incomplete ranges.

We recommend phrasing status checks in verifiable terms, such as "the thread page for this query has ended, and two reply tasks within parent comments remain incomplete." This is more honest than "completeness rate 98%", as the latter requires a reliable total denominator using matching definitions. Without a denominator, do not manufacture a seemingly precise percentage.

For general HTTP or parameter issues, refer to API Error Handling in the documentation. Retries can handle some temporary failures, but they cannot expand permissions or turn a partial reply object into a complete dataset.

Checking Scrapers with Four Small Samples

Checking four scenarios on accessible test objects or synthetic fixtures makes it easier to isolate errors than running a large batch of videos from the start.

1. A thread includes fewer replies than its declared quantity: the program should establish a reply task rather than immediately declaring completion.
2. A reply list has a next page: the program must continue processing that parent comment and not just paginate the video thread list.
3. The same page is read twice: the final unique comment count should not double, and parent-child relationships must remain consistent.
4. A parent comment request fails: other results can be retained, but exports must indicate partial incompleteness and must not overwrite older valid results with empty replies.

Acceptance testing should also verify output utility. When performing topic categorization, a reply like "this is wrong" loses almost all interpretability if detached from its parent comment. Preserving parent comment associations helps humans or models return to the discussion context. Documentation on Douyin Comment and Reply Relationships also discusses this modeling problem, but request parameters from different platforms cannot be copied across each other.

The above are suggested tests, not real video experiments executed by this article. If you already have JSON exported from an API, you can offline-verify these structures and states before deciding whether authorization to re-fetch data is needed.

Delivering a Bounded Comment Sample

"How many items were scraped" is only part of the result. An export suitable for analysis should also explain targets, filters, collection times, parent-child relationships, deduplication methods, and incomplete items. Analysis reports should inherit these boundaries and must not drop known upstream gaps just because the process entered the summary stage.

When choosing encapsulated data APIs, use the directory verification method in Getting Started with Unified Data APIs to confirm item-by-item what fields and pagination capabilities are actually provided. Do not directly apply official parameters from this article to another set of APIs, nor assume that "supporting comments" means all levels and historical content are covered.

The key to YouTube comment collection is not continuously increasing per-page limits, but ensuring that each reply's attribution, each pagination task's progress, and each termination reason remain explainable. Only then can scraping results serve as a basis for analysis rather than a file unable to account for what it missed.