Ultimate Spacemesh Setup - Smesh Things (7/7)

Finally.

Github: Hake Hardware

I know, you want to create postdata and get smeshing already. Fine. If you insist. I’ll start out by saying that I use a dedicated PC with a good GPU to create my postdata directly onto my Main host. I do this by sharing the drive with Samba. This means if I have 10 HDD on my Main host, I can generate postdata on all of them without having to move drives around. Since generating postdata is actually pretty low bandwidth, the network isn’t really at risk of getting saturated. You can generate postdata directly on the host if you have a GPU installed, just skip the Samba stuff.

Architecture

So like I mentioned, I have a separate GPU host which generates postdata over the network. It works like this:

It takes a little bit of set up, but its really pretty easy! Lets jump in to it.

Samba

Let’s start on the Main host by install Samba. Add the packages:

sudo apt update && sudo apt install samba -y

Stop the Samba service

sudo systemctl stop smbd

Navigate to Samba Config

cd /etc/samba

Back up Config

sudo mv smb.conf smb.conf.old

Create new Config

sudo nano smb.conf

Copy the following

[global]
server string = Bravo Server
workgroup = WORKGROUP
security = user
map to guest = BAD USER
name resolve order = bcast host

[postdata01]
path = /media/postdata01
force user = smbuser
force group = smbgroup
create mask = 0664
force create mode = 0664
directory mask = 0775
force directory mode = 0775
public = yes
writable = yes

Alright so we have a lot going on here. The ‘server string’ you can change if you want. Leave everything else as is under ‘global’. The second section (postdata01) is our drive that we want to share. Whatever you put in the ‘[ ]’ will be what the drive is named. For path, you want to point to where your HDD is mounted. If you follow my guides that is going to be /media/postdataXX. Leave everything else as is. Next we need to create the user and group that we specify in the config

Group:

sudo groupadd --system smbgroup

We are creating basically a system user - with no home directory or shell access

sudo useradd --system --no-create-home --group smbgroup -s /bin/false smbuser

How change the ownership of the directory where the drive is mounted. This is temporary while we generate data. We will change this back after.

sudo chown -R smbuser:smbgroup /media/postdata01

Give Write Access

sudo chmod -R g+w /media/postdata01

Start Samba

sudo systemctl start smbd

Make sure it started okay

sudo systemctl status smbd

You should see it ‘running’ now. Now take note of your IP address. You can get it with:

hostname -I | awk '{print $1}'

From here we are going to move to our GPU Host.

Mount Samba Drive

On the GPU host, I am assuming you have Ubuntu installed. Also this is only for NVIDIA cards. Make sure your system is fully updated:

sudo apt update && sudo apt upgrade -y

Install the cifs-utils dependency:

sudo apt install cifs-utils -y

Then edit your fstab:

sudo nano /etc/fstab

Then we want to add an entry for the network drive at the bottom

//<YOUR IP>/postdata01       /media/postdata       cifs username=smbuser,password=pass,iocharset=utf8  0       0

You will need to add your IP, and change the folder depending on what your set up is. This will mount the network drive /media/postdata on the GPU Host. Keep that in mind when we run postcli.

Press CTRL+X and then ‘Y’ to save changes. Then mount the drive:

sudo mount -av

You should see that it has been successfully mounted. Now let’s get postcli started up.

PostCLI

Install dependencies

sudo apt install git clinfo nvtop nvidia-driver-525 tmux unzip iotop python3.10-venv python3-dev -y

Now at this point its a good idea to reboot your system. In fact. You have to.

sudo reboot now

Confirm your GPU is listed

clinfo -l

Make the directory where we will install postcli, then enter it

mkdir postcli && cd postcli

Get the latest release URL from Releases

wget <Latest Release URL Here>

Unzip

unzip postcli-Linux.zip

Clean up after yourself

rm postcli-Linux.zip

Make executable

chmod +x postcli

Confirm you can see your GPU again, but this time with postcli

./postcli -printProviders

Okay we are ready to generate postdata. Run your command:

sudo ./postcli -provider 0 -commitmentAtxId <highest ATX> -labelsPerUnit 4294967296 -maxFileSize 2147483648 -datadir /media/postdata -numUnits <your numuints>

(NOTE: If you run into an error that you need to specify the node id, your key.bin was probably already created by go-spacemesh. Scroll down a bit and there will be a command to get your node id from your key.bin - then there is an example on how to add the ‘-id’ part to the command.)

Alright lets break this down. You need to use the current highest ATX. This can’t change once you start generating postdata. If the highest ATX changes while you are generating postdata it isn’t a problem, just keep using the one you are already using. You can either query your node for the highest ATX:

grpcurl --plaintext -d "{}" <YOUR IP>:9092 spacemesh.v1.ActivationService.Highest

Then you can use this command to convert it:

echo -n 'OUTPUT FROM GRPC' | base64 -d | xxd -c 32 -g 32

Should be something like this

Or you can go to my website and get it here: Smeshi

Make sure your datadir goes to where you mounted the drive. Lastly, enter the numUnits that are appropriate for your drive. That’s really it. Your drive should start smeshing. If you need to stop it you can, but note that you will need the node ID to restart it. You can get the node id by inspecting the key.bin and getting the last 60 characters:

sudo cat /media/postdata/key.bin | tail -c 64 ; echo

Then you can use it in your command like this:

sudo ./postcli -provider 0 -commitmentAtxId <highest ATX> -labelsPerUnit 4294967296 -maxFileSize 2147483648 -datadir /media/postdata -numUnits <your numuints> -id <NODE ID>

If you want to use dual GPUs you can also do that by specifying the toFile and fromFile. For instance if I mount the network drive to two GPU hosts, I can generate postdata with both GPUs by running the postcli command on each GPU host and specifying incremental files. Lets say I am doing the minimum of 4 numUnits. That is a total of 128 files (4x32 at maxFileSize 2147483648). So for GPU Host 1 I would do:

sudo ./postcli -provider 0 -commitmentAtxId <highest ATX> -labelsPerUnit 4294967296 -maxFileSize 2147483648 -datadir /media/postdata -numUnits <your numuints> -id <NODE ID> -fromFile 0 -toFile 63

And GPU Host 2:

sudo ./postcli -provider 0 -commitmentAtxId <highest ATX> -labelsPerUnit 4294967296 -maxFileSize 2147483648 -datadir /media/postdata -numUnits <your numuints> -id <NODE ID> -fromFile 64 -toFile 127

Note that we start at 0 and end at 127, this is because 0 is the first file. Also note that you MUST specify an ‘-id’ to generate postdata with toFile and fromFile. You can create it by just running the commmand initially without toFile and fromFile to create the key.bin. Then stop it and grab the node ID and rerun the commands with the to and from.

Alright - I went a bit deeper than I wanted. I have dedicated videos on postcli, so check those out if you get confused.

Smeshing

Once the postdata is completed, you can safely unmount the drive from the GPU host with:

sudo umount //<your ip>/postdata01

Then on your Main host, depending on if you have more drives to generate postdata, or if that was your last drive, you can either leave samba installed and update the config for the next drive, and restart the service, or you can remove samba altogether, along with the user and group.

More Drives

So if you have more drives, you will want to update the config file so that the share now points to the next drive. So we started with postdata01, if we wanted to update to postdata02 we would do the following:

Edit the config

sudo nano /etc/samba/smb.conf

Change to postdata02

[global]
server string = Bravo Server
workgroup = WORKGROUP
security = user
map to guest = BAD USER
name resolve order = bcast host

[postdata02]
path = /media/postdata02
force user = smbuser
force group = smbgroup
create mask = 0664
force create mode = 0664
directory mask = 0775
force directory mode = 0775
public = yes
writable = yes

Save with CTRL+X then ‘Y’. Then restart the Samba service

sudo systemctl restart smbd

Now go back to your GPU Host and remount.

No More Drives

If you are done with generating postdata, then you can safely remove Samba. First, lets uninstall Samba:

sudo apt-get remove --purge samba

Then we delete the user and group we created

sudo userdel smbuser && sudo groupdel smbgroup

You also want to make sure you update the ‘smeshing-start’ in the config from ‘false’ to ‘true’.

"smeshing-start": true

And lastly, you need to restart only that node. You can do this by going into Portainer, clicking “Containers” then finding the container for the node where the postdata exists, and then clicking the checkbox for just that node. Then at the top click ‘restart’. This will restart the node with the new config.

The node will generate the initial proof and you should be able to easily monitor this with Node Exporter or cAdvisor.

And that is it! You are all set. Hopefully this was helpful. Happy smeshing.