Skip to main content

📟 POI Radio

The source code for the POI Radio is available on GitHub and Docker builds are automatically published as GitHub Packages. POI Radio is also published as a Crate on Crates.io.

Introduction​

The POI Radio is an optional component of the Graph Protocol Indexer Stack that leverages the Graphcast Network to cross-check subgraph data integrity with other participating Indexers.

In order for an Indexer to earn indexing rewards from the Protocol, valid Proofs of Indexing (POIs) must be posted on-chain regularly, proving that the Indexer has the correct data. Posting an invalid POI exposes the Indexer to the liability of a Dispute and being slashed by the Protocol. This Radio provides Indexers peace of mind that their POIs are being continuously cross-checked against other participating Indexers. In the event that there is a POI discrepancy, the Radio acts as an early warning system, notifying the Indexer within minutes of the discrepancy occuring.

All POIs that are generated by the Radio are normalized, meaning they are hashed with a 0x0 Indexer Address and can be compared between Indexers, all without being valid POIs that could be submitted on-chain for rewards. The Radio will group and weight all normalized POIs by the aggregate stake in GRT that is attesting to each. The normalized POI with the largest aggregate attesting stake is considered canonical and will be used for comparisons with your local Indexer POIs.

For security reasons, POI Radio needs an independent Graphcast ID linked to your Indexer account to operate. Your Graphcast ID is an Ethereum account that has been authorized to sign POI attestations on behalf of your Indexer. Follow instructions for registering your Graphcast ID below.

Getting Started​

Before you can run any Radio on Graphcast, you need to register a Graphcast ID for your on-chain Indexer address. You can learn what a Graphcast ID is and how to register one here.

Once you complete those steps you will have a Graphcast ID that is authorized to sign messages on behalf of your Indexer. You can then use that Graphcast ID to run a POI Radio instance.

Basic Configuration​

The POI Radio is configured using environment variables. You will need to prepare the following environment variables:

NameDescription and examples
PRIVATE_KEYPrivate key to the Graphcast ID wallet (Precendence over mnemonics).
Example: 0x0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
GRAPH_NODE_STATUS_ENDPOINTURL to a Graph Node Indexing Status endpoint.
Example: http://index-node:8030/graphql
REGISTRY_SUBGRAPHURL to the Graphcast Registry subgraph for your network.
Mainnet: https://api.thegraph.com/subgraphs/name/hopeyen/graphcast-registry-mainnet
Goerli: https://api.thegraph.com/subgraphs/name/hopeyen/graphcast-registry-goerli
NETWORK_SUBGRAPHURL to the Graph Network subgraph
Mainnet: https://gateway.thegraph.com/network
Goerli: https://gateway.testnet.thegraph.com/network
GRAPHCAST_NETWORKThe Graphcast Messaging fleet and pubsub namespace to use.
Mainnet: mainnet
Goerli: testnet

Run with Docker​

  1. Pull the POI Radio image
docker pull ghcr.io/graphops/poi-radio:latest
  1. Run the image, providing the required environment variables. Here's a sample mainnet configuration:
docker run \
    -e GRAPHCAST_NETWORK="mainnet" \
    -e REGISTRY_SUBGRAPH="https://api.thegraph.com/subgraphs/name/hopeyen/graphcast-registry-mainnet" \
    -e NETWORK_SUBGRAPH="https://gateway.thegraph.com/network" \
    -e PRIVATE_KEY="GRAPHCAST_ID_PRIVATE_KEY" \
    -e GRAPH_NODE_STATUS_ENDPOINT="http://graph-node:8030/graphql" \
    -e RUST_LOG="warn,hyper=warn,graphcast_sdk=info,poi_radio=info" \
    ghcr.io/graphops/poi-radio:latest

(or) Run with docker-compose​

You can append this service definition to your docker-compose manifest and customise the definitions:

services:
  # ... your other service definitions
  poi-radio:
    image: ghcr.io/graphops/poi-radio:latest
    container_name: poi-radio
    restart: unless-stopped
    environment:
      GRAPHCAST_NETWORK: "mainnet"
      REGISTRY_SUBGRAPH: "https://api.thegraph.com/subgraphs/name/hopeyen/graphcast-registry-mainnet"
      NETWORK_SUBGRAPH: "https://gateway.thegraph.com/network"
      PRIVATE_KEY: "GRAPHCAST_ID_PRIVATE_KEY"
      GRAPH_NODE_STATUS_ENDPOINT: "http://graph-node:8030/graphql"
      RUST_LOG: "warn,hyper=warn,graphcast_sdk=info,poi_radio=info"
    logging:
      driver: local

(or) Run as part of StakeSquid's docker-compose setup​

POI Radio is included as an optional component in both the mainnet and testnet versions of StakeSquid's guide. The only prerequisite is to have a Graphcast ID registered. To enable the POI Radio, you can edit the .env file as follows - uncomment the GRAPHCAST_ID_PRIVATE_KEY line and replace the placeholder value with the private key of your Graphcast ID wallet.

(or) Run using a pre-built binary​

We also provide pre-built binaries for Ubuntu and MacOS, which you can find in the Assets section on each release in the releases page on Github. Simply download the binary, make it executable (chmod a+x ./poi-radio-{TAG}-{SYSTEM}) and then run it (using ./poi-radio-{TAG}-{SYSTEM}).

Advanced Configuration​

In the configuration table below is the full list of environment variables you can set, along with example values.

See Basic Configuration above. The following environment variables are optional:

Name (Optional variables)Description and examples
MNEMONICMnemonic to the Graphcast ID wallet (first address of the wallet is used; Only one of PRIVATE_KEY or MNEMONIC is needed). Example: claptrap armchair violin...
COLLECT_MESSAGE_DURATIONSeconds that the POI Radio will wait to collect remote POI attestations before making a comparison with the local POI. Example: 120 for 2 minutes.
COVERAGEToggle for topic coverage level. Possible values: "comprehensive", "on-chain", "minimal". Default is set to "on-chain" coverage.
TOPICSComma separated static list of content topics (subgraphs) to subscribe to. Example: QmWmyoMoctfbAaiEs2G46gpeUmhqFRDW6KWo64y5r581Vz,QmUwCFhXM3f6qH9Ls9Y6gDNURBH7mxsn6JcectgxAz6CwU,QmQ1Lyh3U6YgVP6YX1RgRz6c8GmKkEpokLwPvEtJx6cF1y
WAKU_HOSTInterface onto which to bind the bundled Waku node. Example: 0.0.0.0
WAKU_PORTP2P port on which the bundled Waku node will operate. Example: 60000
WAKU_NODE_KEYStatic Waku Node Key.
BOOT_NODE_ADDRESSESPeer addresses to use as Waku boot nodes. Example: "addr1, addr2, addr3"
SLACK_TOKENSlack Token to use for notifications. Example: xoxp-0123456789-0123456789-0123456789-0123456789
TELEGRAM_TOKENTelegram Bot Token to use for notifications. Example: 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
TELEGRAM_CHAT_IDThe ID of the Telegram chat to send messages to. Example: -1001234567890
SLACK_CHANNELName of Slack channel to send messages to (has to be a public channel). Example: poir-notifications
WAKU_LOG_LEVELWaku node logging configuration. Example: INFO (is also the default)
RUST_LOGRust tracing configuration. Example: graphcast_sdk=debug,poi_radio=debug, defaults to info for everything
DISCORD_WEBHOOKDiscord webhook URL for notifications. Example: https://discord.com/api/webhooks/123456789012345678/AbCDeFgHiJkLmNoPqRsTuVwXyZaBcDeFgHiJkLmN
METRICS_PORTIf set, the Radio will expose Prometheus metrics on this (off by default). Example: 3001
METRICS_HOSTIf set, the Radio will expose Prometheus metrics on this (off by default). Example: 0.0.0.0
SERVER_HOSTIf SERVER_PORT is set, the Radio will expose an API service on the given host and port. Default: 0.0.0.0
SERVER_PORTIf set, the Radio will expose an API service on the given port (off by default). Example: 8080
LOG_FORMATOptions: pretty - verbose and human readable; json - not verbose and parsable; compact - not verbose and not parsable; full - verbose and not parsible. Default value: pretty.
PERSISTENCE_FILE_PATHRelative path. If set, the Radio will periodically store states of the program to the file in json format (off by default).
DISCV5_ENRSComma separated ENRs for Waku Discv5 bootstrapping. Defaults to empty list.
DISCV5_PORTDiscoverable UDP port. Default: 9000

COVERAGE is used to specify the topic coverage level. It controls the range of topics (subgraph ipfs hashes) the Indexer subscribes to in order to process data and participate in the network.

There are three coverage levels available:

  • comprehensive: Subscribe to on-chain topics, user-defined static topics, and subgraph deployments syncing on graph node. This level is useful for Indexers who want to compare NPOIs for all deployments syncing on their graph node even if they don't have an active allocations open (their stake will not be taken into account in attestation).
  • on-chain: Subscribe to on-chain topics and user-defined static topics. This is the default coverage level and is suitable for indexers who only want to compare data for deployments with active allocations.
  • minimal: Only subscribe to user-defined static topics. This level is for Indexers who want to limit their participation to specific topics of interest.

WAKU_HOST and WAKU_PORT specify where the bundled Waku node runs. If you want to run multiple Radios, or multiple instances of the same Radio, you should run them on different ports.

If you want to customize the log level, you can toggle RUST_LOG environment variable. Here's an example configuration to get more verbose logging:

RUST_LOG="warn,hyper=warn,graphcast_sdk=debug,poi_radio=debug"

The PERSISTENCE_FILE_PATH configuration variable allows the Radio to maintain operational continuity across sessions. When set, it triggers the Radio to periodically store its state, including local attestations and remote messages, in a JSON-formatted file at the specified path. This facilitates seamless session transitions and minimizes data loss. In the event of a system disruption, the state can be reloaded from this file, ensuring the Radio can resume operation effectively.

Discv5 is an ambient node discovery network for establishing a decentralized network of interconnected Graphcast Radios. Discv5, when used in Graphcast Radios, serves as a dedicated peer-to-peer discovery protocol that empowers Radios to form an efficient, decentralized network. Without Discv5, the traffic within the Graphcast network would largely rely on centrally hosted boot nodes, leading to a less distributed architecture. However, with Discv5, Radios are capable of directly routing messages among themselves, significantly enhancing network decentralization and reducing reliance on the central nodes. If you want to learn more about Discv5, check out the official spec.

Monitoring the Radio​

Notifications​

If the Radio operator has set up a Slack, Discord and/or Telegram bot integration and the Radio finds a POI mismatch, it sends alerts to the designated channels. The operator can also inspect the logs to see if the Radio is functioning properly, if it's sending and receiving messages, if it's comparing normalised POIs, if there is a found POI mismatch, etc.

Prometheus & Grafana​

The POI Radio exposes metrics that can then be scraped by a Prometheus server and displayed in Grafana. In order to use them you have to have a local Prometheus server running and scraping metrics on the provided port. You can specify the metrics host and port by using the environment variables METRICS_PORT and METRICS_HOST. We also provide a Grafana dashboard config JSON file which you can use to visualise the metrics in Grafana.

HTTP Server​

The Radio spins up an HTTP server with a GraphQL API when SERVER_HOST and SERVER_PORT environment variables are set. The supported routes are:

  • /health for health status
  • /api/v1/graphql for GET and POST requests with GraphQL playground interface

The GraphQL API now includes several advanced queries:

  • radioPayloadMessages
  • localAttestations
  • comparisonResults
  • stakeRatio
  • senderRatio

Below are some example queries:

Query {
  radioPayloadMessages{
    identifier
    nonce
    blockNumber
    network
    signature
  }
  localAttestations{
    deployment
    blockNumber
    attestation{
      npoi
    }
  }
  comparisonResults(deployment:"Qm...."){
    deployment
    blockNumber
    resultType
    localAttestation{
      npoi
    }
    attestations{
      npoi
    }
  }
  stakeRatio(filter: {deployment: "__", blockNumber: "___"}){
    deployment
    blockNumber
    compareRatio
  }
  senderRatio{
    deployment
    blockNumber
    compareRatio
  }
}

You can customize the returned data from the stakeRatio and senderRatio queries by providing optional filters as arguments:

  • deployment - If provided, only attestations for the specified deployment will be included in the comparison.
  • block - If provided, only attestations for the specified block number will be included in the comparison.
  • filter - A more complex filter that can include deployment, block_number, and result_type fields. This filter is used to further refine the set of attestations included in the comparison. Here's an example of a query with filters:
Query {
  stakeRatio(filter: {deployment: "Qm....", blockNumber: 12345, result_type: "Type_Name"}){
    deployment
    blockNumber
    compareRatio
  }
}

In this example, the stakeRatio query will return the stake ratios only for attestations from deployment "Qm...." and block number 12345, and only for the specified result type.

Note: The result_type field of the filter corresponds to the resultType field in the comparisonResults query. This field represents the type of comparison result.

The API also includes the senderRatio and stakeRatio endpoints, which return more detailed insights into the state of the Radio.

senderRatio provides an overview of the consensus status of the attestations from remote messages. It gives a ratio string that signifies the number of indexers with the same npoi as the local Radio. The results are presented as x/y!/z where:

  • x, y, and z are sorted by descending stake weights
  • ! indicates the entry that corresponds to the local result.

For example, 2/0! means there are two indexers attesting with a higher sum of stake weight and no other indexer shares the same nPOI as the local Radio. 8! means there are eight other indexers agreeing with the local Radio.

stakeRatio offers similar functionality to senderRatio but the results are based on the stake weight. It orders the attestations by stake weight, then computes the ratio of unique senders.

Query {
  senderRatio{
    deployment
    blockNumber
    compareRatio
  }
  stakeRatio{
    deployment
    blockNumber
    compareRatio
  }
}

These queries provide a clear aggregation of the attestations from remote messages, giving a concise understanding of the Radio's state. The optional filters - deployment, block, and filter - can be used to refine the results.

How it works​

Fetching active allocations​

The POI Radio is responsible for reading active allocations of the Radio operator's corresponding Indexer. It periodically polls the Graph Node for new blocks on all relevant networks and constructs Graphcast topics on each allocation identified by subgraph deployment IPFS hash.

tip

The relevant networks are those corresponding to the subgraphs that have active allocations.

The Radio fetches new active allocations at a regular interval to ensure that it is processing the latest information. Chainheads for these networks are updated with data from the Graph Node, and the Radio ensures that it is always using the latest chainhead when processing messages.

Gathering and comparing normalised POIs​

At a given interval, the Radio fetches the normalised POI for each deployment. This interval is defined in blocks different for each network. It then saves those nPOIs, and as other Indexers running the Radio start doing the same, messages start propagating through the network. The Radio saves each message and processes them on a given interval.

The messages include a nonce (UNIX timestamp), block number, signature (used to derive the sender's on-chain Indexer address) and network. Before saving an entry to the map, the Radio operator verifies through the Graph network subgraph for the sender's on-chain identity and amount of tokens staked, which is used during comparisons later on.

At another interval, the Radio compares the local nPOIs with the collected remote ones. The remote nPOIs are sorted so that for each subgraph (on each block), the nPOI that is backed by the most on-chain stake is selected. This means that the combined stake of all Indexers that attested to it is considered, not just the highest staking Indexer. The top nPOI is then compared with the local nPOI for that subgraph at that block to determine consensus.

If there is a mismatch and if the Radio operator has set up a Slack, Discord and/or Telegram bot integration, the Radio will send alerts to the designated channels.

After a successful comparison, the attestations that have been checked are removed from the store.

Developing the POI Radio​

Building the image using the Dockerfile locally​

If you want to make any changes to the POI Radio codebase, you can use this option.

Prerequisites​
  1. Clone this repo and cd into it
  2. Create a .env file that includes at least the required environment variables. To see the full list of environment variables you can provide, check out the Configuration section.
Running the POI Radio inside a Docker container​
docker-compose up -d

Building POI Radio locally​

To have full control over the POI Radio code and run it directly on your machine (without Docker) you can use this option.

Prerequisites​

  1. Clone this repo and cd into it
  2. Make sure you have the following installed:
  • Rust
  • Go
  • Build tools (e.g. the build-essentials package for Debian-based Linux distributions or Xcode Command Line Tools for MacOS)
  • C compiler (e.g. the clang package for Debian-based Linux distribution or Xcode Command Line Tools for MacOS)
  • OpenSSL (e.g. the libssl-dev package for Debian-based Linux distribution or openssl for MacOS)
  • PostreSQL libraries and headers (e.g. the libpq-dev package for Debian-based Linux distribution or postgresql for MacOS)
  1. You have Graph Node syncing your indexer's on-chain allocations.
  2. You have created a .env file that includes at least the required environment variables. To see the full list of environment variables you can provide, check out the Configuration section.

Running the POI Radio natively​

cargo run