Quick Start

Make sure you have all the Prerequisites before starting.

Install Taskfile

Launchpad has a large number of tooling dependencies that will run on your local machine. The most important dependency is Taskfile.

Follow the installation instructions for your environment and install Taskfile on your local machine before continuing.

Use launchpad-starter for your new infra repo

Next, we are going to create the repository that will contain your new infrastructure's configuration.

First, prepare a new empty repository to hold your infrastructure repo. This could be a new repository on GitHub, GitLab, BitBucket, etc.

Next, we're going to clone launchpad-starter, and then replace the existing origin remote with your new remote repository. This allows us to retain the commit history of launchpad-starter. A shared commit history will make future rebases against the upstream launchpad-starter much easier.

# Clone the starter into my-new-infra and cd into it
git clone https://github.com/graphops/launchpad-starter my-new-infra
cd my-new-infra

# Set your own remote as origin
git remote remove origin
git remote add origin git@github.com:you/your-infra.git

# Push to your new repo
git push origin main

All work on your infrastructure will take place in this new repo. We recommend carefully version controlling all changes you make to your infrastructure configuration.

Setup the launchpad dependencies

Next, we should install all of the local tooling dependencies (like Helm or Kubectl) that we will need.

We can easily do that by running the launchpad:setup command.

# You may need to use sudo for this command
task launchpad:setup

# For now, this will just run launchpad:deps, which will install all the local tooling dependencies

Connect your Local environment to your Kubernetes cluster

To connect your local machine to a Kubernetes cluster, you can follow these general steps:

Get Cluster Configuration: Make sure your kubeconfig has been added to ~/.kube/config file. If you don't have this file, you may need to ask the administrator that created the cluster for the configuration.

Verify Configuration: Open the config file in a text editor to verify that it contains the correct cluster details, including server URL, certificates, and context information.

Switch Context if working with multiple Kubernetes clusters: A context in Kubernetes is a combination of a cluster, a user, and a namespace. Use the kubectl config use-context command to set your desired context. For example:

kubectl config use-context <context-name>

Test Connection: Run a simple kubectl command to test if your local machine can connect to the cluster:

kubectl get pods

This command should list the pods in the default namespace of your cluster.

Remember that each cluster might have specific setup steps or requirements, especially if it's managed by a cloud provider. Always refer to the documentation provided by the cluster administrator or the cloud provider for detailed instructions on connecting your local machine to the cluster.

🎉 Milestone: Local environment configured!

• We now have our own private git repo containing the declarative configuration for our cluster deployments
• We have installed all the tooling dependencies on our local machine, which will be used to control the cluster
• Next: Copy sample.helmfile.yaml to helmfile.yaml and edit it to customize and deploy Namespaces on the Kubernetes cluster

Customize your helmfile.yaml

If you don't have an helmfile.yaml file yet, you can start with copying the existing sample one (sample.helmfile.yaml):

cp sample.helmfile.yaml helmfile.yaml

and edit it with your editor of choice. Under that file, there is an helmfiles: section declaring Namespaces to be deployed on the cluster, looking like this:

helmfiles:
  - path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-stable/latest
    selectorsInherited: true
    values:
      - helmDefaults:
          <<: *helmDefaults

  - path: git::https://github.com/graphops/launchpad-namespaces.git@monitoring/helmfile.yaml?ref=monitoring-stable/latest
    selectorsInherited: true

  - path: git::https://github.com/graphops/launchpad-namespaces.git@ingress/helmfile.yaml?ref=ingress-stable/latest
    selectorsInherited: true
    values:
      - kubeVersion: *kubeVersion

You can override values, and select which Namespaces you are interested in. Refer to Namespaces documentation available here for more examples on how to configure them, or to see which ones are available: Namespaces.

Syncing your helmfile.yaml with the cluster

Next we need to install key non-Graph components of our stack, including monitoring and logging systems.

Let's see what the releases:apply-base task is actually doing by running task help -- releases:apply-base:

task: releases:apply-base
  
Apply current helmfile state filtered by all base layer services
  
commands:
  
• task releases:apply -- launchpad.graphops.xyz/layer=base

As you can see, releases:apply-base just calls releases:apply filter for all namespaces with the label launchpad.graphops.xyz/layer=base.

You can list all the releases present in the helmfile.yaml, and their labels, by running task releases:list:

NAME                            NAMESPACE               ENABLED INSTALLED       LABELS                                                                                  CHART                                           VERSION       
openebs                         storage                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:storage              openebs/openebs                                 3.8.0         
openebs-zfs-localpv             storage                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:storage              openebs-zfs-localpv/zfs-localpv                 2.3.0         
openebs-zfs-storageclass        storage                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:storage              graphops/resource-injector                      0.2.0         
openebs-zfs-snapclass           storage                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:storage              graphops/resource-injector                      0.2.0         
postgres-operator               postgres-operator       true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:postgres-operator    postgres-operator-charts/postgres-operator      1.10.0        
ingress-nginx                   ingress                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:ingress              ingress-nginx/ingress-nginx                     4.7.1         
cert-manager                    ingress                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:ingress              jetstack/cert-manager                           v1.12.3       
cert-manager-resources          ingress                 true    true            launchpad.graphops.xyz/layer:base,launchpad.graphops.xyz/namespace:ingress              graphops/resource-injector                      0.2.0         
sealed-secrets                  sealed-secrets          true    true            launchpad.graphops.xyz/namespace:sealed-secrets                                         sealed-secrets/sealed-secrets                   2.1

First, update configuration in your helmfile.yaml for the base namespaces. You will likely need to configure storage and ingress with your own values.

In particular, the storage namespace may be a requirement even for other base namespaces, so lets install that one first by running task releases:apply -- launchpad.graphops.xyz/namespace=storage

Next, let's go ahead and install all the remaining cluster services. You will be prompted to install each namespace, with a summary of changes to be made.

task releases:apply-base

🎉 Milestone: Kubernetes and core systems running!

• We connected to our hosts, configured them, and installed Kubernetes
• We installed core cluster services like Prometheus, Grafana, Loki and others
• Next: Deploy blockchain nodes and the Graph Indexing stack

tip

You can now use task indexer:forward-grafana to securely access your remote cluster's Grafana instance at http://localhost:3001

Deploy blockchain namespaces as desired

note

If you have existing external blockchain nodes that you would like to use instead of deploying them into your cluster, you can skip this section, but make sure that you can access those nodes securely (e.g. via an internal network, or using HTTPS and authentication).

Launchpad comes with Namespace definitions for a number of blockchain networks, including Ethereum Mainnet, Ethereum Goerli Testnet, Gnosis Chain Mainnet, Polygon mainnet, Abitrum Mainnet, Avalanche Mainnet, Celo Mainnet and others. Using those Namespaces, you can easily deploy blockchain nodes for the networks you want to index into your cluster.

(optional, eth-goerli) Install Erigon, Nimbus and Proxyd for Ethereum Goerli

Update your helmfile.yaml to add the ethereum Namespace and ensure the values are correct:

helmfiles:
  - path: git::https://github.com/graphops/launchpad-namespaces.git@ethereum/helmfile.yaml
    selectorsInherited: true
    values:
    - flavor: goerli
      helmDefaults:
        <<: *helmDefaults
    - erigon:
        values:
          statefulNode:
            volumeClaimSpec:
              storageClassName: "<<your storage class>>"
            jwt:
              existingSecret:
                name: jwt
                key: jwt
      nimbus:
        values:
          nimbus:
            volumeClaimSpec:
              storageClassName: "<<your storage class>>"
            jwt:
              existingSecret:
                name: jwt
                key: jwt

You will need to generate a JWT and create a kubernetes secret accordingly, to be used by Erigon and Nimbus.

Deploy by syncing your cluster with the declarative helmfile.yaml:

task releases:apply -- eth-goerli

Install the Graph Goerli Indexer Stack

Update your helmfile to include the graph Namespace, and configure the values accordingly:

helmfiles:
  - path: git::https://github.com/graphops/launchpad-namespaces.git@graph/helmfile.yaml
    selectorsInherited: true
    values:
    - helmDefaults:
        <<: *helmDefaults
      flavor: "goerli"
    - graph-network-indexer:
        values:
          indexerDefaults:
            config:
              indexer-address: "<<your indexer ethereum address>>"
          indexerAgent:
            config:
              public-indexer-url: "<<your public index URL>>"
      graph-operator-mnemonic:
        values:
          resources:
            ### RECOMMENDED, safe to commit
            sealed-secret:
              apiVersion: bitnami.com/v1alpha1
              kind: SealedSecret
              metadata:
                name: graph-operator-mnemonic
                namespace: graph-goerli
              spec:
                template:
                  metadata:
                    name: graph-operator-mnemonic
                    namespace: graph-goerli
                  type: Opaque
                encryptedData:
                  mnemonic: <<your encrypted mnemonic>> # Generate a SealedSecret encryptedData key with the "utils:seal-secrets" task, e.g.: task utils:seal-secrets -- -n graph-goerli -s graph-operator-mnemonic -k mnemonic -v "your mnemonic words"
      graph-database:
        values:
          resources:
            postgres-cr-primary-subgraph-data:
              spec:
                volume:
                  storageClass: "<<your storage class>>"
            postgres-cr-indexer-metadata:
              spec:
                volume:
                  storageClass: "<<your storage class>>"

Proceed to deploy:

task releases:apply -- graph-goerli

🎉 Milestone: Graph Indexer running and accessible

• We (optionally) configured and deployed blockchain nodes into our cluster
• We configured and deployed the Graph Indexing stack into our cluster
• Next: Use the remote-toolbox to allocate to subgraphs and begin serving requests

Updates

Updating launchpad-namespace changes into your stack

As new versions of key components in the stack are released, we will update launchpad-namespaces's templated definitions and the various release streams available. You can selectively inherit these updates with ease by changing the git ref as a means to track what release stream you may want, or to pin to any particular major, minor or patch version.

following latest:

Your ?ref= would look like this, for the storage namespace: ?ref=storage-latest, or alternatively: ?ref=storage-stable/latest. The path for this Namespace, under helmfiles, would then look like:

- path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-latest

following a specific major version:

Your ?ref= would look like this, for the storage namespace: ?ref=storage-v1. The path for this Namespace, under helmfiles, would then look like:

- path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-v1

following a specific minor version:

Your ?ref= would look like this, for the storage namespace: ?ref=storage-v1.2. The path for this Namespace, under helmfiles, would then look like:

- path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-v1.2

pinning to an exact version:

Your ?ref= would look like this, for the storage namespace: ?ref=storage-v1.2.2. The path for this Namespace, under helmfiles, would then look like:

- path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-v1.2.2

following the latest canary:

Your ?ref= would look like this, for the storage namespace: ?ref=storage-canary/latest. The path for this Namespace, under helmfiles, would then look like:

- path: git::https://github.com/graphops/launchpad-namespaces.git@storage/helmfile.yaml?ref=storage-canary/latest

We would recommend that you either follow the latest stable releases, or pin to a specific version.

note

For full implementation details and other comprehensive notes about launchpad-namespaces please visit the github repo.

Pulling in starter changes

From time to time, you may want to update your infra repo with the latest changes from our starter.

Launchpad comes with a built in task to do this:

task launchpad:pull-upstream-starter