Resume with cursors

Restart a consumer after a disconnect or deploy without losing or duplicating data. Save each block's cursor in the same transaction as its output, then reconnect with the latest committed cursor. The stream continues at the next block.

Prerequisites

• A durable database, checkpoint table, or state store.
• A consumer that receives BlockScopedData messages and their cursors.
• Idempotent output writes.

Commit, then resume

For each BlockScopedData message:

1. Write the block's output.
2. Save the message's cursor in the same transaction.
3. Commit, then process the next message.

When the consumer restarts, read the latest committed cursor and pass it to the stream. The CLI tab resumes with --cursor. The Go, Rust, and TypeScript tabs sketch the two functions a consumer built on a standard Substreams client library needs: one that commits output and cursor together, and one that reads the cursor back on startup.

export SUBSTREAMS_API_KEY="<GRAPHOPS_API_KEY>"

substreams run \
  -e "https://1.evm.substreams.graphops.xyz" \
  "https://github.com/streamingfast/substreams-explorers/releases/download/v1.0.1/ethereum-explorer-v0.1.2.spkg" \
  "map_block_meta" \
  --start-block 18000000 \
  --cursor "<LAST_COMMITTED_CURSOR>" \
  -o json

func commitBlock(output []byte, cursor string) error {
	// In production, commit output and cursor in one database transaction.
	// If the cursor commit fails, replaying the block must be safe.
	return nil
}

func resumeCursor() string {
	return "<LAST_COMMITTED_CURSOR>"
}

fn commit_block(output: &[u8], cursor: &str) -> Result<(), String> {
    // In production, commit output and cursor in one durable transaction.
    // If the cursor commit fails, replaying the block must be safe.
    Ok(())
}

fn resume_cursor() -> String {
    "<LAST_COMMITTED_CURSOR>".to_string()
}

async function commitBlock(output: Uint8Array, cursor: string) {
  // In production, commit output and cursor in one durable transaction.
  // If the cursor commit fails, replaying the block must be safe.
  await Promise.resolve({ output, cursor });
}

async function resumeCursor() {
  return "<LAST_COMMITTED_CURSOR>";
}

Check the resume

After a restart, the first block the stream sends should be the block after the last one you committed. If it is earlier, the consumer saved the cursor before its output or did not save the latest cursor; if it is later, a cursor was saved for output that never committed.

Make replays safe

• Save the cursor after the output commits, never before.
• Make writes idempotent. A retry can replay data from the last committed cursor.
• Store the endpoint, package reference, and module name with the cursor, so you resume the same stream that produced it.
• Handle BlockUndoSignal separately from normal output; see Cursors and reorgs.
• Reconnect with exponential backoff and jitter after retryable disconnects.

Next steps