Subgraphs: Paginate results

Read every pool, holder, or vote in a collection for a report or a sync job. Order by a stable field and ask for the rows after the last id you received, so each request is bounded and a worker can resume where it stopped.

Prerequisites

A Subgraphs API key.
A Subgraph endpoint that exposes the entity collection you want to scan.
A stable ordering field such as id, createdAtTimestamp, or another indexed field in the Subgraph schema.

Choose a pagination method

For a few pages, first and skip are usually enough. For long scans, filter past the last value of a stable field, such as where: { id_gt: $lastId } with orderBy: id. Each page then starts where the previous one ended, however far into the collection you are:

Each page asks for the rows after the last id of the page before it. Save that id after you process a page, and stop when a page returns fewer rows than you requested.

Request

This request reads one page of 100 pools after $lastId. For the first page, set $lastId to a value that sorts before every id, such as the zero address in these examples.

curl "https://subgraphs.graphops.xyz/v1/alias/<ALIAS>" \
  -H "Authorization: Bearer <GRAPHOPS_API_KEY>" \
  -H "Content-Type: application/json" \
  --data '{
    "query": "query PoolsPage($first: Int!, $lastId: ID!) { pools(first: $first, where: { id_gt: $lastId }, orderBy: id, orderDirection: asc) { id token0 { symbol } token1 { symbol } } _meta { block { number hash } } }",
    "variables": { "first": 100, "lastId": "0x0000000000000000000000000000000000000000" },
    "operationName": "PoolsPage"
  }'

let lastId = "0x0000000000000000000000000000000000000000";

const response = await fetch(
  "https://subgraphs.graphops.xyz/v1/alias/<ALIAS>",
  {
    method: "POST",
    headers: {
      Authorization: "Bearer <GRAPHOPS_API_KEY>",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      query: `
        query PoolsPage($first: Int!, $lastId: ID!) {
          pools(
            first: $first
            where: { id_gt: $lastId }
            orderBy: id
            orderDirection: asc
          ) {
            id
            token0 { symbol }
            token1 { symbol }
          }
          _meta { block { number hash } }
        }
      `,
      variables: { first: 100, lastId },
      operationName: "PoolsPage",
    }),
  },
);

const result = await response.json();
const pools = result.data.pools;
lastId = pools.at(-1)?.id ?? lastId;

import { GraphQLClient, gql } from "graphql-request";

const client = new GraphQLClient(
  "https://subgraphs.graphops.xyz/v1/alias/<ALIAS>",
  {
    headers: {
      Authorization: "Bearer <GRAPHOPS_API_KEY>",
    },
  },
);

const query = gql`
  query PoolsPage($first: Int!, $lastId: ID!) {
    pools(
      first: $first
      where: { id_gt: $lastId }
      orderBy: id
      orderDirection: asc
    ) {
      id
      token0 { symbol }
      token1 { symbol }
    }
    _meta { block { number hash } }
  }
`;

const result = await client.request(query, {
  first: 100,
  lastId: "0x0000000000000000000000000000000000000000",
});
const nextLastId = result.pools.at(-1)?.id;
console.log(nextLastId);

import requests

last_id = "0x0000000000000000000000000000000000000000"

response = requests.post(
    "https://subgraphs.graphops.xyz/v1/alias/<ALIAS>",
    headers={
        "Authorization": "Bearer <GRAPHOPS_API_KEY>",
        "Content-Type": "application/json",
    },
    json={
        "query": """
        query PoolsPage($first: Int!, $lastId: ID!) {
          pools(
            first: $first
            where: { id_gt: $lastId }
            orderBy: id
            orderDirection: asc
          ) {
            id
            token0 { symbol }
            token1 { symbol }
          }
          _meta { block { number hash } }
        }
        """,
        "variables": {"first": 100, "lastId": last_id},
        "operationName": "PoolsPage",
    },
    timeout=30,
)
response.raise_for_status()
pools = response.json()["data"]["pools"]
next_last_id = pools[-1]["id"] if pools else last_id
print(next_last_id)

Response

{  "data": {    "pools": [Up to $first rows, ordered by id      {        "id": "0x0001...",        "token0": { "symbol": "USDC" },        "token1": { "symbol": "WETH" }      },      {        "id": "0x3af1...",Last id: your next $lastId        "token0": { "symbol": "WBTC" },        "token1": { "symbol": "WETH" }      }    ],    "_meta": {      "block": {        "number": 18000000,Log with each page        "hash": "0x..."      }    }  }}

An illustrative page. Pass the last id as $lastId for the next request, and stop when a page has fewer rows than $first.

Run the full scan

Send the first request with $lastId set before every id.
Process the page, then save the last id you processed.
Send the next request with that id as $lastId.
Stop when a page returns fewer rows than $first.

Keep these in mind for long scans:

Keep page sizes bounded, and back off after 429 responses.
Prefer id_gt filters over high skip values.
Persist the last processed id so a worker can resume after a retry or restart.
Log _meta.block.number with each page. Historical block queries are not supported, so pages cannot be pinned to one block and later pages can reflect newer data.