Load a protocol's history into your database before you follow the live chain. This guide runs a bounded block range, checks the output, and then shows how to split a long history into ranges you can retry independently and continue live from the same consumer.
Prerequisites
- A Substreams API key.
- A package and output module that support the target network.
- A destination that can commit output idempotently, so a retried range does not duplicate rows.
Run a bounded range
Set a start block and a stop block. This example streams filtered_events for blocks 18000002 up
to the stop block 18000008 on Ethereum mainnet.
The CLI tab is a complete command. The Go, Rust, and TypeScript tabs are complete consumers.
Prerequisites
Install the Substreams CLI.
Set GRAPHOPS_API_KEY to your GraphOps API key.
Substreams CLI
# The CLI sends SUBSTREAMS_API_KEY to GraphOps in the x-api-key header.
export SUBSTREAMS_API_KEY="${GRAPHOPS_API_KEY}"
# Backfill USDC Transfer events (--params) from 18000002 up to, not including, 18000008.
substreams run \
-e "https://1.evm.substreams.graphops.xyz" \
"https://spkg.io/v1/packages/ethereum-common/v0.4.0" \
"filtered_events" \
--params "filtered_events=evt_addr:0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48 && evt_sig:0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef" \
--start-block 18000002 \
--stop-block 18000008 \
--max-retries 0 \
--force-protocol-version 2 \
-o jsonl
Output
{ "@module": "filtered_events", "@block": 18000002,Historical block: 18,000,002 "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "5b94c39ad7eecbf043f62a4a76c5abee87195207df98ae16eb3f289b2267f141", "number": "18000002", "timestamp": "2023-08-26T16:21:59Z" }, "events": [ { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",Ethereum mainnet USDC contract; hexadecimal in CLI output, base64 in Protobuf JSON "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",ERC-20 Transfer signature "0x000000000000000000000000e673bda19a529d9ff13aeddae0e08c0fda8b5488",Sender address, left-padded to 32 bytes "0x00000000000000000000000088e6a0c2ddd26feeb64f039a2c41296fcb3f5640"Recipient address, left-padded to 32 bytes ], "data": "0x00000000000000000000000000000000000000000000000000000000401332c0",Transfer amount: 1,075,000,000 base units, 1,075 USDC "index": 1, "blockIndex": 172, "ordinal": "2582" }, "txHash": "29ec08dc5eed0e1cd411b29c52607661155087ffa107bae7f5e4a79f5306be1d"Transaction containing this log }, { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000a40bb1c47f6dd27142a2bd7c93bfa98db9d1f5c5", "0x000000000000000000000000b4e16d0168e52d35cacd2c6185b44281ec28c9dc" ], "data": "0x0000000000000000000000000000000000000000000000000000000011348e5d", "index": 2, "blockIndex": 315, "ordinal": "5139" }, "txHash": "a8b07b56f7bb4ea52eecf5a125c75f143955a8e57e1ea17d2d8a48830e7d67db" } ] }}{ "@module": "filtered_events", "@block": 18000003, "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "90c06a9fad66de217ce5862a92a2d45a8f42f9abf8ec17d5bfb85ffdc7836ca4", "number": "18000003", "timestamp": "2023-08-26T16:22:11Z" }, "events": [ { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000180cd4303582d348a54fcb0436d5665f082a23d1", "0x000000000000000000000000a9d1e08c7793af67e9d92fe308d5697fb81d3e43" ], "data": "0x00000000000000000000000000000000000000000000000000000000aad0a340", "blockIndex": 222, "ordinal": "4219" }, "txHash": "05c9c59145387faa91370e56d717d42856bd20f8f19306814d5b7afbe837efe3" }, { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x0000000000000000000000001c727a55ea3c11b0ab7d3a361fe0f3c47ce6de5d", "0x000000000000000000000000c3dcc088aef4373e665b00fc8be45a4b6a4b24a6" ], "data": "0x000000000000000000000000000000000000000000000000000000000686bcda", "blockIndex": 230, "ordinal": "4543" }, "txHash": "eb8505dd3dc2964c1b43ed40371fb9c907b751ccb9294b99db032270889edc42" } ] }}{ "@module": "filtered_events", "@block": 18000004, "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "7f7889c3686d7f560dd690aeb2a10b49e3a76844543cf78cc0dfe4687e2985d2", "number": "18000004", "timestamp": "2023-08-26T16:22:23Z" }, "events": [ { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x0000000000000000000000000000000000000000000000000000000000000000", "0x00000000000000000000000043954785980c50768373dc1dafd7ba3a2052036f" ], "data": "0x000000000000000000000000000000000000000000000000000000e8d433aa6e", "index": 1, "blockIndex": 191, "ordinal": "3060" }, "txHash": "b2c1d8c8e7989adab51dafdc9be24fd05243342b8d08828c4325662de629c690" } ] }}{ "@module": "filtered_events", "@block": 18000005, "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "127c6b9794874d7011770b3aa2a0a640e1db201c2d677d01b2b3b19e7ad427c7", "number": "18000005", "timestamp": "2023-08-26T16:22:35Z" }, "events": [ { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000c880d8be4e94f644f1daa7d3558b50ff5a1c2db7", "0x000000000000000000000000ae2d4617c862309a3d75a0ffb358c7a5009c673f" ], "data": "0x0000000000000000000000000000000000000000000000000000002e90edd000", "blockIndex": 64, "ordinal": "1110" }, "txHash": "1691b8e89e13f483c549d9b61efbc54ecc54015b8186726911c8458f0993e13d" }, { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000b86fc5e985facdcae6034beefc6cbdcd8bbd8cb4", "0x00000000000000000000000027f713adefc708469d826ab8f5b1e4672f1a070e" ], "data": "0x0000000000000000000000000000000000000000000000000000000059682f00", "blockIndex": 203, "ordinal": "3669" }, "txHash": "440106b494b7eb657d19e5f82a164ce4cc62e5252058fab5fb4ee6b92600543c" } ] }}{ "@module": "filtered_events", "@block": 18000006, "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "ed05b59286d4f2db801e78a23ce4584d869d857b70b2758822844427caa55a85", "number": "18000006", "timestamp": "2023-08-26T16:22:47Z" } }}{ "@module": "filtered_events", "@block": 18000007, "@type": "sf.substreams.ethereum.v1.Events", "@data": { "clock": { "id": "db63e26b655ad31d60b2173c02127180ed46b28506f785a2682e058ca61afdc1", "number": "18000007", "timestamp": "2023-08-26T16:22:59Z" }, "events": [ { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000b4eae88d1643e172518dde0ef213c20628798c65", "0x00000000000000000000000038c5a4f22e2e290ba9bbb9e3d9aaf8582f79f4d1" ], "data": "0x0000000000000000000000000000000000000000000000000000000006b78840", "blockIndex": 94, "ordinal": "1448" }, "txHash": "21d8e836f79606db39fcd51f7e563ea9c12cfb7a2bb3e36f23f4eb89b93045ea" }, { "log": { "address": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48", "topics": [ "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x000000000000000000000000579752cff8fee7af09446b2133ee2f9ff10c4fbf", "0x0000000000000000000000001715a3e4a142d8b698131108995174f37aeba10d" ], "data": "0x000000000000000000000000000000000000000000000000000000001d0221cd", "blockIndex": 179, "ordinal": "3398" }, "txHash": "d78a180e669534a650d842f13d032610e29bf4a7731fb90a6dff73d752ac049c" } ] }}Check the output
The CLI prints a filtered_events message for every block in the range, in block order, with no
events field for blocks without matching USDC Transfers. It exits when it reaches
the stop block. Each message has the same envelope as the
quickstart output: the block number in
@block and the module output in @data.
Before you scale up, compare the matching events in the requested range with your destination and confirm that rerunning the same range leaves the row count unchanged.
Split a long history
A single stream can cover a long range, but one failure late in the range means repeating it. For large histories, plan the work as bounded ranges:
Choose range boundaries
Pick a range size you can rerun in an acceptable time. Adjacent ranges share a boundary block number; the stop block of one range is the start block of the next.
Run and commit each range
Run each range with its own start and stop block. Write output idempotently, keyed by block number, block hash, ordinal, or a package-specific ID.
Record progress
After a range commits, store the range, package reference, module name, endpoint, and final cursor in your backfill state. A failed range can then be retried alone.
Continue live
When the history is loaded, start a stream at the last stop block and omit --stop-block. The
stream continues with new blocks as they arrive.
Run ranges within your plan's concurrency. If you see rate-limit or resource errors, run fewer ranges at once. Compare output volume with the current credit rates.