Ultimate Spacemesh Setup - Introduction (1/7)

Ubuntu, Docker, Grafana and more!

Github: Hake Hardware

I should qualify this article by adding that this is my ultimate Spacemesh setup. It’s unlikely that you have the exact hardware and knowledge that I do, so I urge you to take what is useful from these tutorials, improve where you can, and have fun learning about some really cool technology. My hardware mostly consists of older tech that is being repurposed for Spacemesh, however I’m not going to focus very much on the hardware in these articles. Spacemesh is designed to be usable across a wide range of hardware, but the gist of it is you need a decent GPU to create the postdata (this is done once per node), a hard drive to store the postdata, and a CPU with enough power to complete the proofing process during the cycle gap. RAM you probably want to prioritize more of it over speed, as each node you run will require ~4-4.5GiB during the proofing process. You can find the system requirements for Spacemesh here: System Requirements.

One last thing before we dive in, this article expects you understand the basics of Spacemesh. Although this will be a very detailed guide, and I will make every effort to explain the “why”, most of this guide actually focuses on non-Spacemesh tech.

One more last thing - every article has an associated YouTube video. I will link those videos at the start of the article.

What is the Ultimate Setup?

Alright, so what exactly are we building? And even more importantly what makes this the ultimate setup? Well to answer that let’s explore some problems I have that I want to solve. I have a lot of nodes. Right now I have 3 hosts with 15 nodes each, that is 45 nodes. It’s a huge pain to monitor, update, and keep running. I need something that gives me a high level overview of the health of my nodes without being overly complex. Previously I used Proxmox, first with VMs, then with LXCs, and although it worked, it was not efficient. I would create a new VM or LXC for each node, which made updates and maintenance a total pain. In this tutorial we will be using Docker containers, along with Docker Compose to build an extremely lightweight and efficient setup. Docker is the heart of this setup, it not only will run our nodes, but we will install additional Docker containers that add the ability to monitor our containers and host in another tool called Grafana - which is an analytics and monitoring solution where you can make really cool dashboards. So lets briefly go over the high level architecture of this system.

PoST Creation

In order to smesh, you need to create postdata. I use a dedicated host with a good GPU to create my postdata instead of creating it on the host which has my go-Spacemesh nodes. There are a lot of ways to do this, but what has worked best for me is to install all of my HDDs on my Main host (the one where the nodes will be) and then create a Samba file server to share those HDDs with my GPU host. In this way, the GPU host will create the postdata over the network onto my HDD. It works something like this:

Because the postdata creation is so slow, the network usage is actually quite small. With an RTX4090 you are only generating postdata at about 7MiB/s, so unless you have a huge GPU farm, it’s unlikely to saturate the network. You could also physically attach the HDDs to your GPU host and plot directly to them, then move them over to the Main host when the drive is filled. However, depending on your setup this may require you to power off the Main host each time a new drive is added.

There will be an article that covers how I do this using Ubuntu Server + postcli.

The Main Host

Now let’s talk about the Main host. This is where your go-spacemesh nodes will sit. Like the GPU host, we are using Ubuntu Server here. You can use Ubuntu Desktop if you want a GUI, it doesn’t matter much. Because I will be using my Main host as a dedicated Spacemesh host, I will be doing everything over SSH with my laptop. You can of course just log directly into the Main host if you have a monitor hooked up.

The first thing we will do after getting Ubuntu updated is install Docker. Docker allows us to containerize our nodes, which makes things really neat and clean since they are effectively isolated from one-another. Docker can do some amazing things - and one of those things is create a network bridge that all of our containers will communicate over. Docker also works with a special file called Docker Compose, which allows us to specify everything about our containers in yml format. We will be creating one of these files to deploy all of our containers. We will also be using Portainer, which is actually a container within Docker. Portainer is basically just a GUI front end for Docker. It makes things a lot easier to manage. All of our HDDs are physically installed on the main host, so those will be bound to each of our go-spacemesh Docker containers when we start smeshing. Our config files and state will also live in their own folders on our host, which like the HDDs will be bound to our Docker containers.

For our nodes we will enable metrics, which will allow us to get some basic metrics about the state and peers connected to our node. In order to monitor our host (such as CPU usage, RAM, Network usage, etc..), we will create a Node Exporter service that runs on our host and publishes metrics. For container level metrics (such as how much RAM each container is using) we will use cAdvisor.

Prometheus is a service we will use to ingest all of our metrics from our Spacemesh nodes, Node Exporter, and cAdvidor. Prometheus will also run as a Docker container on our Main host, and will eventually be hooked up to Grafana as a data source in order to produce our dashboards.

When we are done it will look something like this:

What is Next

Now that we have a basic understanding of what we are building, the next few articles will dive into how to build it. They are written with the assumption that you are starting from scratch, but you should be able to bounce around if you are just looking to do something specific. Here is the general outline of the articles:

1. Article 1 - Introduction (you are here)

2. Article 2 - Foundation

   1. Setup Ubuntu Server

   2. Setup Docker

   3. Setup Portainer

   4. Setup Prometheus

3. Article 3 - Metrics

   1. Setup Node Exporter

   2. Setup cAdvisor

4. Article 4 - Grafana

   1. Setup Grafana

5. Article 5 - Nodes!

   1. The Folder Structure

   2. Mount your Drives (w/ PoST Data)

   3. Mount your Drives (no PoST Data)

   4. Stacks on Stacks

   5. Spacemesh Dashboard

6. Article 6 - What’s Next?

   1. GRPC

   2. Public & Private Nodes

   3. Updates

   4. Logs

7. Article 7 - Hulk Smesh

   1. Architecture

   2. Samba

   3. Mount Samba Drive

   4. PostCLI

   5. Generating postdata

   6. More Drives

   7. No More Drives

Now this may seem like a lot, but it’s fairly straight forward. Prior to Spacemesh I had very little experience with Linux, and no experience with Docker or any of the other tech like Grafana. That brings me to my next point, there is a good chance I won’t always do something the ‘right’ or ‘best’ way. It’s probably because I don’t know better. I’m new to this, so use this guide at your own risk. I am not a Linux guru. If you know how to do something better, by all means modify this guide to your own desire. If you want to contribute and provide some tips, that is also appreciated. You can leave a comment here or find me in the official Spacemesh discord, my name is ‘repost’. Let’s get started!