TRX1 Dev Blog #9 (September 2021)

In this episode, you will learn about NodeOp tools and decentralized Midgard connector.

THORChain Monitoring Bot: NodeOp tools

NodeOp tools is a solution for monitoring important events that occur to the THORNodes by receiving personalized notifications in the Telegram bot.

It might be helpful for THORNode operators in order to maintain health and reliable operation of the nodes and the whole network. Also, curious users can use NodeOp tools to keep up-to-date with the status of the network.

Once you have added the nodes to the tracking list, you can select the categories of alerts you are interested in and configure their settings.

NodeOp tools use both aiothornode library and internal Midgard connection. But most of the information comes via a WebSockets channel from the https://thorchain.network/ website’s backend. Shoutout to its author for such an awesome and useful tool! 👍

The following notification categories are currently available:

Slash point accrual
Software version changes and new version detector
The service goes offline/online (the services are RPC, Midgard, Bifrost and THORNode itself)
Node churning in/out
Node bond changes
IP address changes
Block height stuck/caught up for each blockchain connected

Let’s see how NodeOp tools works in the wild.

NodeOp Guide

First, open THORChain Monitoring Telegram bot 👉https://t.me/thor_infobot 👈

From the main menu navigate to 🤖 NodeOp tools.

Then you need to add one or many or even all the nodes to your watch list. Proceed to Add nodes.

As you can see on the screenshot below, you can pick specific nodes by their public address (the last 4 characters of the address, as is customary in our community) or you can add all connected nodes or all active nodes at once. Node bond size in Rune is inside brackets. The nodes are sorted in descending order:

You can remove nodes from your watch list in the Edit nodes menu.

Clicking on the corresponding address will remove the node from the watch list.

Menu for removing nodes from the watch list

You can also clear the entire watch list if you no longer want to receive any notifications. However, you can temporarily pause sending notifications to you without clearing your watch list. To do this, go into the settings.

There you can pause/unpause all NodeOp alerts.

While we’re at the settings, let’s take a closer look at them. Bond, IP address and Churning are simple. You just turn them on or off like this:

Every time your node changes its IP address, its Rune bond or churn in/out the active validator set you will receive a notification.

Version’s part is a little bit trickier. It has two steps. First, you are asked about new version notifications and after that you can turn on or off notifications for tracking software upgrade of the nodes in your watch list.

Version notifications settings have 2 steps

After the second step you will be back in the Settings menu.

In the Offline submenu, you manage notifications for monitoring online status of THORNodes and their services. When you choose “ON” you will be asked for a time period. So you get a notification if one of the services on any of the nodes will be offline during this period of time. You will also be informed when it goes back to work. So far it is not possible to toggle individual service.

Block height submenu works in a similar manner. If the block height on one of the connected blockchain clients lags behind the actual block height by about the selected time interval, you will receive a message about it. When it catches up with everyone else, you will also be notified. If the threshold interval is less than the typical block time for the blockchain, it will be increased to 150% of the typical time (15 minutes for BTC).

Last but not least are the Slash point notifications. There are three steps in their setup. First step is that you turn them on. The second step is to set a slash-point limit.

The third step is to set an interval of time.

Now it’s done. If during the selected time interval the node accumulates so many or more points, the bot will send you a notification about this fact.

Let’s see a couple of examples of the notifications of various kinds:

That’s it. Give it a try. Feedback is welcome.

THORChain Monitoring Bot: what else new?

As you may know, most of THORChain-related services are usually connected to the public Midgard hosted at https://midgard.thorchain.info/v2/doc.

The Midgard Public API queries THORChain and any chains linked via the Bifröst and prepares information about the network to be readily available for public users.

Sometimes, especially at times of high load, the public Midgard can be a point of failure. Do you remember THORStarter IDO for example?

Using the principles of decentralization, we want to ensure that the bot has stable availability of information, regardless of the state of the central access point.

Each node can run its own Midgard. Therefore, an algorithm was developed that allows the bot to retrieve data from all available Midgards. In September, this approach was integrated into all subsystems of the bot. The THORChain monitoring bot continues to work even during peak loads on the system.

Miscellaneous

Other little tweaks made to improve the bot’s performance.

In the LP yield module added a fallback method for detection of the user’s pools for non-THOR addresses
Bug fix: corrected calculations for large swap transaction notification
Updated the thresholds for TX alerts
The stable coin list is configurable from now
Database optimizations