Load a protocol's history into your database before you follow the live chain. This guide runs a bounded block range, checks the output, and then shows how to split a long history into ranges you can retry independently and continue live from the same consumer.
Prerequisites
- A Substreams API key.
- A package and output module that support the target network.
- A destination that can commit output idempotently, so a retried range does not duplicate rows.
Run a bounded range
Set a start block and a stop block. This example streams map_block_meta for blocks 18000000 up
to the stop block 18001000 on Ethereum mainnet.
The CLI tab is a complete command. The Go, Rust, and TypeScript tabs show the values a standard Substreams client library needs for the same range.
export SUBSTREAMS_API_KEY="<GRAPHOPS_API_KEY>"
substreams run \
-e "https://1.evm.substreams.graphops.xyz" \
"https://github.com/streamingfast/substreams-explorers/releases/download/v1.0.1/ethereum-explorer-v0.1.2.spkg" \
"map_block_meta" \
--start-block 18000000 \
--stop-block 18001000 \
-o jsonconst endpoint = "https://1.evm.substreams.graphops.xyz"
const packageRef = "https://github.com/streamingfast/substreams-explorers/releases/download/v1.0.1/ethereum-explorer-v0.1.2.spkg"
const moduleName = "map_block_meta"
const startBlock = 18000000
const stopBlock = 18001000
// Run a finite range with a standard Substreams Go client or sink.
// Checkpoint progress by cursor and block number.const ENDPOINT: &str = "https://1.evm.substreams.graphops.xyz";
const PACKAGE_REF: &str = "https://github.com/streamingfast/substreams-explorers/releases/download/v1.0.1/ethereum-explorer-v0.1.2.spkg";
const MODULE_NAME: &str = "map_block_meta";
const START_BLOCK: i64 = 18_000_000;
const STOP_BLOCK: i64 = 18_001_000;
// Run a finite range with a standard Substreams Rust client.
// Checkpoint progress by cursor and block number.const backfill = {
endpoint: "https://1.evm.substreams.graphops.xyz",
packageRef: "https://github.com/streamingfast/substreams-explorers/releases/download/v1.0.1/ethereum-explorer-v0.1.2.spkg",
moduleName: "map_block_meta",
startBlock: 18_000_000,
stopBlock: 18_001_000,
headers: {
"x-api-key": "<GRAPHOPS_API_KEY>",
},
};
console.log(backfill);Check the output
The CLI prints one map_block_meta message per block in block order, then exits when it reaches
the stop block. Each message has the same envelope as the
quickstart output: the block number in
@block and the module output in @data.
Before you scale up, confirm that your destination received one set of rows per block and that rerunning the same range leaves the row count unchanged.
Split a long history
A single stream can cover a long range, but one failure late in the range means repeating it. For large histories, plan the work as bounded ranges:
Choose range boundaries
Pick a range size you can rerun in an acceptable time. Adjacent ranges share a boundary block number; the stop block of one range is the start block of the next.
Run and commit each range
Run each range with its own start and stop block. Write output idempotently, keyed by block number, block hash, ordinal, or a package-specific ID.
Record progress
After a range commits, store the range, package reference, module name, endpoint, and final cursor in your backfill state. A failed range can then be retried alone.
Continue live
When the history is loaded, start a stream at the last stop block and omit --stop-block. The
stream continues with new blocks as they arrive.
Run ranges within your plan's concurrency. If you see rate-limit or resource errors, run fewer ranges at once. Compare output volume with the current credit rates.