# Troubleshooting

Source: https://docs.graphops.xyz/substreams/troubleshooting

Find the symptom your stream or sink shows, then work through its checks in order. Each entry
names the likely cause first. If the checks do not resolve it, contact
[GraphOps support](https://platform.graphops.xyz/organizations/current/overview?support=open)
with the details listed at the end of this page.

## Authentication fails

The stream is missing a valid API key, or the key is sent in the wrong place.

1. Confirm the key was created for the organization that owns Substreams access and includes Substreams.
2. Confirm the key has not been revoked or rotated.
3. Confirm you use a query key, which starts with `gqk_`. Substreams refuses a service account API key
(`gsk_`) as unauthenticated.
4. For the CLI and sinks, confirm `SUBSTREAMS_API_KEY` is exported in the shell that runs the command.
5. For raw gRPC clients, confirm every stream sends `x-api-key: <GRAPHOPS_API_KEY>` metadata.
6. Remove extra quotes or whitespace from the stored secret value.
7. Compare your setup with [Authentication](/substreams/authentication).

## Stream is refused or ends with a valid key

The gRPC status tells you which kind of problem it is:

| Status               | Meaning                                                                                                                              | What to do                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `UNAUTHENTICATED`    | The API key is missing, malformed, revoked or rotated, is not a query key, or does not include Substreams.                           | Work through [Authentication fails](#authentication-fails).                                                         |
| `PERMISSION_DENIED`  | The key is valid, but GraphOps refuses the stream. A running stream can also end with this status, for example when credits run out. | Work through the checks below.                                                                                      |
| `UNIMPLEMENTED`      | The endpoint host is not a GraphOps Substreams endpoint.                                                                             | Check the endpoint against [Authentication](/substreams/authentication).                                            |
| `RESOURCE_EXHAUSTED` | The organization has more streams or concurrent work open than its limits allow.                                                     | Close streams you no longer need and reduce parallel workers. See [usage and limits](/substreams/usage-and-limits). |
| `UNAVAILABLE`        | GraphOps could not check access right now.                                                                                           | Retry with backoff.                                                                                                 |

Most refusals include the GraphOps error body, `{"code", "message", "docsUrl"}`, inside the status
message, sometimes after a prefix such as `permission denied:`. Its `code` names the reason, and
[gateway errors](/platform/errors/gateway) and [API key errors](/platform/errors/api-keys) explain
each code. A new stream refused for exceeding the stream limit has no code. For
`PERMISSION_DENIED`:

1. Check your credit balance in **Billing**. If it is zero or below
(`credits_exhausted`), add credits to bring it above zero. Access is restored automatically.
2. If the balance is positive, GraphOps support may have paused the organization or disabled the key
(`usage_access_paused`, `organization_disabled` or `api_key_disabled`). Adding credits does not
restore access. Contact
[GraphOps support](https://platform.graphops.xyz/organizations/current/overview?support=open),
or create a new key if only the key is disabled.
3. For `stream_access_blocked`, GraphOps blocked streaming for the organization. Contact support.
4. For `stream_session_key_mismatch`, use the same API key for the whole stream, or start a new
stream after rotating a key.
5. For `api_key_route_not_allowed`, confirm the endpoint matches the chain you stream.
6. For `api_key_target_not_allowed`, the key's Substreams access is
[limited](/platform/api-keys#limit-a-key) and does not include the network of the endpoint you
stream from. Stream from an allowed network, or add this one to the key's limits.

## Package cannot be resolved

Standard Substreams tooling cannot load the package name, version, URL, or local `.spkg` reference.

1. Confirm the package exists on [substreams.dev](https://substreams.dev/packages) or at the URL you
provided.
2. Pin an exact package version rather than a moving tag such as `latest`.
3. Confirm an unpublished `.spkg` file is readable by the process running the consumer.
4. Confirm the package supports the network endpoint you selected.

## Module not found

The package does not expose the requested output module.

1. Check the package page or manifest for the available modules.
2. Match the module name exactly, including case.
3. Confirm the module is a map module you can stream, not a store module.
4. Regenerate client Protobuf types if the package output changed.

## Stream disconnects

Long-running gRPC streams can disconnect because of network, client, or service conditions.

1. Reconnect with exponential backoff and jitter.
2. Resume from the latest committed cursor.
3. Confirm the cursor is stored durably, not only in process memory.
4. Reduce concurrency if many consumers reconnect at once.
5. Log the endpoint, package reference, module name, cursor, and block number for each disconnect.

## Duplicate data after reconnect

The consumer resumed from an earlier cursor, or replayed a block after a failed commit.

1. Make output writes idempotent.
2. Save the cursor only after the output commits.
3. Store the cursor and output in one transaction when possible.
4. Deduplicate by block number, block hash, ordinal, or a package-specific ID.

See [Resume with cursors](/substreams/guides/resume-with-cursors) for the commit order.

## Undo signal not handled

The stream sent a `BlockUndoSignal`, but the consumer did not revert its local state.

1. Implement undo handling before you use reversible data for production state.
2. Revert local state to the last valid block named in the signal.
3. Save the signal's cursor after the rollback.
4. Keep enough local history to undo recent writes.

See [Cursors and reorgs](/substreams/concepts/cursors-and-reorgs) for the recovery model.

## Backfill is slow or expensive

The range is large, the output volume is high, or concurrency exceeds your organization's limits.

1. Split the backfill into smaller ranges you can retry independently.
2. Reduce concurrency.
3. Use a narrower output module when possible.
4. Track blocks processed, bytes out, bytes read, and bytes written.
5. Check rates and limits in [Usage and limits](/substreams/usage-and-limits).

## You need GraphOps to host the sink

Self-service Substreams assumes you run the sink. Hosted sinks are an enterprise option.

1. Check whether a standard sink in your infrastructure fits; see
[Run a sink](/substreams/guides/run-a-sink).
2. If not, contact GraphOps from the platform about hosted sink options.

## Need support

When you contact [GraphOps support](https://platform.graphops.xyz/organizations/current/overview?support=open), include:

* Organization name or ID.
* Endpoint URL and chain ID.
* Package reference and version.
* Module name.
* Start block, stop block, and latest committed cursor.
* Approximate timestamp.
* gRPC status or error message.
* Whether the consumer saw `BlockScopedData`, `BlockUndoSignal`, or progress messages.

Never include your API key.

## Next steps

* [Resume with cursors](/substreams/guides/resume-with-cursors): Make restarts safe so disconnects do not lose or duplicate data.
* [Backfill a range](/substreams/guides/backfill-a-range): Plan a large history as bounded, retryable ranges.
* [Usage and limits](/substreams/usage-and-limits): Check the limits that slow or reject streams.