Troubleshooting

Find the symptom your stream or sink shows, then work through its checks in order. Each entry names the likely cause first. If the checks do not resolve it, contact GraphOps support(opens in a new tab) with the details listed at the end of this page.

Authentication failsCopy Anchor Link

The stream is missing a valid API key, or the key is sent in the wrong place.

1. Confirm the key was created for the organization that owns Substreams access and includes Substreams.
2. Confirm the key has not been revoked or rotated.
3. Confirm you use a query key, which starts with gqk_. Substreams refuses a service account API key (gsk_) as unauthenticated.
4. For the CLI and sinks, confirm SUBSTREAMS_API_KEY is exported in the shell that runs the command.
5. For raw gRPC clients, confirm every stream sends x-api-key: <GRAPHOPS_API_KEY> metadata.
6. Remove extra quotes or whitespace from the stored secret value.
7. Compare your setup with Authentication.

Stream is refused or ends with a valid keyCopy Anchor Link

The gRPC status tells you which kind of problem it is:

Most refusals include the GraphOps error body, {"code", "message", "docsUrl"}, inside the status message, sometimes after a prefix such as permission denied:. Its code names the reason, and gateway errors and API key errors explain each code. A new stream refused for exceeding the stream limit has no code. For PERMISSION_DENIED:

1. Check your credit balance in Billing. If it is zero or below (credits_exhausted), add credits to bring it above zero. Access is restored automatically.
2. If the balance is positive, GraphOps support may have paused the organization or disabled the key (usage_access_paused, organization_disabled or api_key_disabled). Adding credits does not restore access. Contact GraphOps support(opens in a new tab), or create a new key if only the key is disabled.
3. For stream_access_blocked, GraphOps blocked streaming for the organization. Contact support.
4. For stream_session_key_mismatch, use the same API key for the whole stream, or start a new stream after rotating a key.
5. For api_key_route_not_allowed, confirm the endpoint matches the chain you stream.
6. For api_key_target_not_allowed, the key's Substreams access is limited and does not include the network of the endpoint you stream from. Stream from an allowed network, or add this one to the key's limits.

Package cannot be resolvedCopy Anchor Link

Standard Substreams tooling cannot load the package name, version, URL, or local .spkg reference.

1. Confirm the package exists on substreams.dev(opens in a new tab) or at the URL you provided.
2. Pin an exact package version rather than a moving tag such as latest.
3. Confirm an unpublished .spkg file is readable by the process running the consumer.
4. Confirm the package supports the network endpoint you selected.

Module not foundCopy Anchor Link

The package does not expose the requested output module.

1. Check the package page or manifest for the available modules.
2. Match the module name exactly, including case.
3. Confirm the module is a map module you can stream, not a store module.
4. Regenerate client Protobuf types if the package output changed.

Stream disconnectsCopy Anchor Link

Long-running gRPC streams can disconnect because of network, client, or service conditions.

1. Reconnect with exponential backoff and jitter.
2. Resume from the latest committed cursor.
3. Confirm the cursor is stored durably, not only in process memory.
4. Reduce concurrency if many consumers reconnect at once.
5. Log the endpoint, package reference, module name, cursor, and block number for each disconnect.

Duplicate data after reconnectCopy Anchor Link

The consumer resumed from an earlier cursor, or replayed a block after a failed commit.

1. Make output writes idempotent.
2. Save the cursor only after the output commits.
3. Store the cursor and output in one transaction when possible.
4. Deduplicate by block number, block hash, ordinal, or a package-specific ID.

See Resume with cursors for the commit order.

Undo signal not handledCopy Anchor Link

The stream sent a BlockUndoSignal, but the consumer did not revert its local state.

1. Implement undo handling before you use reversible data for production state.
2. Revert local state to the last valid block named in the signal.
3. Save the signal's cursor after the rollback.
4. Keep enough local history to undo recent writes.

See Cursors and reorgs for the recovery model.

Backfill is slow or expensiveCopy Anchor Link

The range is large, the output volume is high, or concurrency exceeds your organization's limits.

1. Split the backfill into smaller ranges you can retry independently.
2. Reduce concurrency.
3. Use a narrower output module when possible.
4. Track blocks processed, bytes out, bytes read, and bytes written.
5. Check rates and limits in Usage and limits.

You need GraphOps to host the sinkCopy Anchor Link

Self-service Substreams assumes you run the sink. Hosted sinks are an enterprise option.

1. Check whether a standard sink in your infrastructure fits; see Run a sink.
2. If not, contact GraphOps from the platform about hosted sink options.

Need supportCopy Anchor Link

When you contact GraphOps support(opens in a new tab), include:

• Organization name or ID.
• Endpoint URL and chain ID.
• Package reference and version.
• Module name.
• Start block, stop block, and latest committed cursor.
• Approximate timestamp.
• gRPC status or error message.
• Whether the consumer saw BlockScopedData, BlockUndoSignal, or progress messages.

Never include your API key.

Next stepsCopy Anchor Link