Telegram Open Network: theory and practice from the network validator

In the past few months, all the attention of the world blockchain community has been riveted to the launch of one of the largest cryptocurrency projects - Telegram Open Network (TON).
What is the TON blockchain really like? Is the TON network truly decentralized? What are its real scalability? How to become a network validator?

The answers to these and other questions were tried to find by the team of the Mercuryo project, which has been an active participant in the test network since the beginning of September 2019.

On November 15, 2019, Telegram services moved to testnet 2 and the third phase of testing started. Our team continued to participate in testing, becoming the first validators on the network after TON.

To participate in the validation process, the user is required not only to have a sufficient number of coins (GRAM tokens), but also a constantly running full network node (TON Blockchain Full Node).

Theoretically, any user can become a validator subject to the condition that he owns the minimum necessary share of the asset (in Gram coins) in the master chain, but in practice a number of questions arise that our team will answer in this article.

In addition, we want to share experience on the use of tonlib-cli , as currently there is practically no documented information, unlike the basic version described in detail in HowTo.

TON Blockchain

The main component of the Telegram Open Network is a flexible blockchain system, hereinafter referred to as TON Blockchain, which, according to the developers themselves, is capable of processing millions of transactions per second, supporting Turing Complete Smart Contracts, updated official blockchain specifications, multicurrency transfers, as well as micropayment channels for off-chain (Off-Chain) payment networks.

“The TON Blockchain architecture is unique because it has specific features such as a“ self-healing ”vertical blockchain mechanism and Instant Hypercube Routing, which allows the blockchain to be both fast, reliable scalable and sustainable. ”

As mentioned above, TON Blockchain is the conventional name for a decentralized network (a set of block chains) or a 2D blockchain consisting of three main types of blockchains.

Master blockchain or Masterchain is a unique chain of blocks containing general information about the protocol and the current values ​​of its parameters, a set of validators and their shares, a set of currently active workchains and their “shards”, as well as a set of hashes of the last blocks masterchains and shardchaynov.

Working blockchains or Workchains - a set (up to 232) of blockchains that are “workhorses” that contain asset transfer transactions and smart contracts. At the same time, individual workchains can have their own “rules”, account address formats, transaction formats, various virtual machines (VMs) for smart contracts, different basic tokens or cryptocurrencies, etc. But all of them must satisfy some basic criteria of interoperability in order to ensure relatively simple interaction among themselves. Thus, the TON Blockchain is inherently heterogeneous, just like the EOS and Polkadot blockchains.

Shard blockchains or Shardchains - a subset of blockchains (up to 260) within a set of workchains, ensuring the operation of the sharding system and having the same rules and block format as the workchains. Shardchains contain only a subset of accounts, depending on the first few (most significant) bits of the address of each particular account. Since all shardchains have a common format and rules for building blocks, the TON blockchain in this respect is homogeneous and meets the requirements described in one of the Ethereum scaling proposals.



Each block of the shardchain (as well as the masterchain) is actually not just a block, but a small blockchain. As a rule, this “block blockchain” or “vertical blockchain” consists of exactly one block, so it can be considered just a block of the corresponding “ordinary” blockchain (or “horizontal block chain”). However, if it becomes necessary to correct incorrect blocks, a new block is entered into the “vertical chain of blocks”, containing either the replacement of the existing “horizontal” block, or the “block difference”, containing only a description of those parts of the previous version of the block that need to be replaced. This TON-specific mechanism for replacing detected invalid blocks without the need for a hard fork is called a 2D blockchain , or simply 2-blockchain.

Consensus Algorithms and Network Protection Mechanism

TON offers a blockchain based on the Infinite Sharding Paradigm (Proof of Stake or PoS). According to the developer documentation:

“Almost all implementations of blockchains using sharding are based on the top-down model: first we imagine one blockchain, and then we decide how to divide it into several parts interacting with each other (shardchains) to increase efficiency and increase scalability.

TON's approach to sharding is based on the bottom-up principle, which means that the original blockchain is extremely scalable, and each individual shardchain contains only one account or smart contract. At the next level, we have a huge number of “account chains”, each of which describes the transitions between the states of only one account and sends each other messages containing information about transactions. At the same time, it is impractical to have hundreds of millions of blockchains, updates (i.e. new blocks) in each of them appear rather rarely, therefore, for their more effective implementation, we grouped these “account chains” into “shardchains”, each block of which is essentially is a collection of blocks of chains of accounts that have been tied to this particular shard. Thus, “account chains” are actually just virtual or logical blocks within the really existing “shardchains”. This mechanism sheds light on many of the design decisions of the TON blockchain and we call it the “Infinite Sharding Paradigm”.

The TON consensus network consists of various types of nodes: validators, nominators, phishers, and collators.



Validators are PoS nodes and block manufacturers. Fishers monitor the consensus network in order to find an error or identify a supposedly malicious consensus node, and if the phisher unambiguously confirms that the node is such, it receives a reward in the form of confiscation of part of the share of this validator.

The task of collators is to prepare shardchain blocks and provide them for validation to PoS nodes, for which they receive their part of the reward for creating the block. At the same time, collators are essentially additional participants in the consensus, since validators almost always generate blocks on their own.

Nominators lend their assets ( Gram tokens) to the validators for profit. In fact, nominees are not included in the infrastructure of validators, but only share their large initial share of the asset between them in exchange for a proportional percentage of the total remuneration. Thus, the scheme and the amount of remuneration that the nominees receive depends entirely on the results of the work of the validators, while the nominees “vote” for the validators, lending them Gram tokens. The nominators can be either individual token holders or pools that manage the funds of individual TON users and at the same time act as validators, acting as delegates through the TON smart contract. In this case, the total remuneration of such a pool is distributed between its participants in proportion to their contributions.

The very process of generating new blocks proceeds as follows: a certain number of validators select a master chain (shards) suitable for validation using a special algorithm, then a smaller subset of validators is selected for each such shard in the order determined by a pseudo-random method with an interval of approximately every 1024 blocks.

Thus, for each block, there is a pseudo-randomly selected set of validators to determine whose candidate block has the highest priority. Validators and other nodes verify the validity of the proposed block candidates. If the validator automatically (not intentionally) signs an invalid candidate for blocks, then he is punished with the loss of part or all of his remuneration, or with the suspension from participation in the selection of validators for some time.

Next, validators need to reach a consensus based on the BFT (Byzantine Resiliency Protocol) algorithm, similar to the pBFT or Honey Badger BFT protocol. Then, after consensus is reached, a new block is created, while transaction fees are distributed among the validators.

It should be noted that each validator can be selected to participate in several subsets of validators; therefore, it is assumed that all validation and consensus algorithms are run in parallel.

After all the new blocks of shards of the chain are generated or the timeout is over, a message appears that a new block of the master chain has been created, which includes hashes of the last blocks of all shards based on the pBFT consensus of all validators.

TON Testnet: hands-on experience in the Telegram Open Network

The Mercuryo project team has been an active participant in the test network since September 2019, and during the testing period we have gained some experience that we would like to share with you.

Network Access Methods

Interaction with the TON network, one way or another, boils down to using TL specifications that describe how to interact with the API. Specification files are available here .

There are three types of APIs:

ton_api - for interacting with Full Node validator-engine-console
lite_api - for working with lite-client
tonlib - everything about the wallet is collected here and this is the only publicly available tonlib-cli API

Wallet Creation

The easiest way to create a wallet is to use the Test Gram Wallet, which is available on the official website for the Windows, macOS and Linux operating systems.




There are also several ways to interact through the command line interface: basic and using tonlib-cli . Unfortunately, at the moment there is no compatibility between them.

Here we will consider only those tools that the TON developers themselves offer. If the basic version is documented in detail in HowTo , then information on using tonlib-cli is practically absent.

As mentioned above, in TON there are 3 APIs for different tasks. The functions associated with the operation of the wallet are responsible for tonlib .

To start working with tonlib-cli , in addition to the command line interface itself, you must have a configuration file for connecting to the public TON network liteserver, which is available here .

Connection is carried out by the team

tonlib-cli -c ton-lite-client-test1.config.json -v 0

where -v 0 is the parameter responsible for the output of debugging information.

List of commands:

To create a wallet address, use the genkey command and a list of mnemonic phrases that may be necessary to restore access to the address in case of loss of the private key.

Key List

The keys command displays a list of keys. For further operations when executing other commands, it is necessary to use their serial number, i.e. for the first key there will be id 0 .

Address initialization

After creating the address, it must be registered on the network. To do this, you must first replenish it. Initially, a special smart contract was used for this - testgiver , but now it’s easier and more convenient to use a special bot in the @test_ton_bot telegram.

Immediately after replenishment, the status of the account is defined as uninited_accountState and will only change after you send GRM test tokens from this address.

If you already have tokens on your balance and you need to activate another wallet, then you can use the transferf command and then, along with the wallet replenishment, it will be initialized.



You can find out the status of the wallet using the getstate 0 command .



Get transaction history using the command

gethistory <num_of_key>

where <num_of_key> is the key sequence number

Network basis

As with most existing blockchains, TON is based on servers that store a complete history of all blockchains that have ever been created on the network.

To run a full node in the TON test network, 8 productive cores, 4-8 GB of RAM are enough, at the time of writing, the data occupied about 50GB of hard disk, but it is better to have a margin of at least 100GB. It should be noted that it is better to use an SSD drive, as A large number of IOPS are required for recording, otherwise synchronization with the network will be very slow.

As a working OS, it is best to use Ubuntu 18.04, as most of the community tests are conducted on it.

Installation Guides

README.txt
FullNode-HOWTO.txt
Validator-HOWTO.txt

Validator system

It is known that the TON blockchain consists of shardchain and masterchain blocks, which are created and verified by special designated nodes called validators. Validators receive some reward for their “work”: maintaining the health of the TON blockchain, while income is distributed proportionally within the validator community.

At first glance, everything is clear, but in practice, a number of questions arise in connection with this:

• Is there a network restriction on the maximum validator steak?

The limit on the size of a share for one validator can always be checked with the getconfig 17 command, which will show the actual sizes of allowable steaks:



The screenshot shows that at the moment the minimum share size is 10,000 GRAM. However, if a validator does not get more than 100,000 GRAM for a round of voting, he is not entitled to participate in the elections. At the same time, the maximum number of tokens per validator cannot exceed 10,000,000 GRAM and in order for the vote to take place, the minimum size of the total steak must exceed 1,000,000 GRAM.

• How are validators selected?

To apply for participation in the validation process, you must have a minimum of 10,000 GRAM. The election process algorithm is described in detail in the smart contract elector-code.fc
Most likely, the contract will be different in the main network, so the current version is applicable only for the test network.

A share of 10,000 GRAM does not mean that you can become a validator, as receiving test tokens could be easily automated by requests to testgiver .

At the moment, almost all validators, when participating in the voting, set max-factor in the amount of 2.7 and a steak in the amount of 120,000 GRAM, since there are most such bets, because of their weight the minimum steak rises to 120,000 / 2.7 = 45,000 GRAM (unlike 100,000 according to official documentation). But even with such a minimal steak, your chances are practically zero, since the three top validators have a max-factor of 2, which raises the minimum share to 60,000 GRAM, which allows you to become a validator in the testnet.

If all the current validators increased their max factor or reduced the size of the steak, then it would be possible to become a validator with a minimum steak, given that the maximum number of validators (1000 nodes) would not be exceeded.

• If the validator system is centralized, then the whole blockchain too?

There are no checks, i.e. no one centrally controls the validators; nominators themselves determine the risks when choosing a validator.

• What types of fines are provided for validators?

There is no information at the moment, most likely there will be a consensus mechanism in the document, because even the out-of-sync nodes received awards in the testnet.

• Smart contracts

To create TON smart contracts, two special programming languages ​​are used: Fift and FunC . If Fift has at least general documentation , then finding information about FunC is almost impossible (even in the conditions of a development contest it is indicated that it can only be obtained by analyzing its source code).

During testing, it was possible to find out that the FunC code base is not so voluminous (compared to Fift ) and allows you to learn it much faster, so working with FunC is much easier than with Fift .

• Actual / urgent / open questions

Slow sync
https://github.com/ton-blockchain/ton/issues/100

Permissions for validation-engine
+0 = usual lite-client queries
+1 = full node statistics queries
+2 = full code configuration modification queries
+4 = potentially dangerous queries (such as private key export or signing arbitrary strings)
+8 = reserved for future extensions (does nothing at the moment)

• How to make PIPE work with lite-client?

By default, lite-client output is sent to stderr, so to process it, you must first redirect the output from stderr to stdout:

$ lite-client 2>> (grep ...)

• What are the options for programmatic network access?

https://github.com/ton-blockchain/ton/issues/76

• What server configuration is required for the validator?

The use of a dual-processor server is officially recommended (at least 8 cores per processor). The software is not very demanding on RAM, so 16 GB is enough. You must use an SSD as the primary drive, the minimum recommended size of which is 512 GB. An 8TB HDD is sufficient for storing archived data.

It is imperative that you have a high-speed Internet connection: with a predicted average load of 100 Mbit / sec, you must be able to handle peak loads of up to 1Gbit / sec.

It is recommended to use XFS as the file system, since information about each block is stored in a separate file. It is known that, for example, ext4 does not work very well with a large number of small files and can lead to a situation where you simply do not have free inodes with enough disk space.

• How do I know that a node is synchronized?

The log will complete the sync message, or using validator-engine-console -c "getstats" unixtime and masterchainblocktime should be almost the same.

• How many validators can be online?

Getconfig 16
max_validators: 1000 max_main_validators: 100 min_validators: 5

• How to find out current active validators?

Getconfig 34

Previous getconfig 32 validator set

• Time for which validators are selected?

The whitepaper indicates that the validators are selected for a month, but in the testnet this time is much less and you can find it out from the getconfig 15 config.

After restarting testnet, the time intervals for validators have changed:

ConfigParam (15) = (validators_elected_for: 65536 elections_start_before: 32768 elections_end_before: 8192 stake_held_for: 32768)

From which it follows that a group of validators is selected for 65536 seconds.

• How are the validators selected for the next round?

The application process is described in detail in the Validator-HOWTO . Validators are selected using a smart contract, the current address of which can always be found using the getconfig 1 command. Then you need to track the start of the vote.



If the answer is result: [0], then the vote is not active, but if the response contains timestamp, then you can apply for participation. Even if you are not going to participate, applications from other selection participants can always be viewed:

> runmethod -1: A4C2C7C05B093D470DE2316DBA089FA0DD775FD9B1EBFC9DC9D04B498D3A2DDA participant_list

• Is it possible to block TON?

Despite the complex design of the network stack based on overlay networks, UDP and TCP are still used as TON transport protocols. It is known that today Telegram locks are quite unsuccessful, because It is possible to change IP addresses, use proxies and update settings through push.However, TON will not have such opportunities: it is not possible to quickly move nodes, while validators will not want to risk their shares. Therefore, most likely, in the near future, Telegram developers will introduce new solutions to bypass locks, for example, using ADNL Proxy.

Below is the traffic of one full node after processing 10 million packets. A list of 159 IP addresses running full testnet nodes is as follows:

126 DIGITAL OCEAN (presumably servers running TON)
13 AMAZON
4 GOOGLE
3 HETZNER
3 ALIBABA CLOUD
2 OVH
2 SELECTEL
2 ONLINE.NET
1 LINODE
1 hosteurope. de
1 contabo.de
and 1 person possibly hosting it at home in Italy telecomitalia.it

Therefore, it will not be difficult for the regulator to obtain a list of IP addresses of the network core. However, this is a problem not only of TON, but of any other Internet service.

In fact, the Telegram Open Network is an attempt to implement an independent decentralized Internet based on the concept of WEB 3.0 within the existing global network, the interaction with which is carried out through the already created information infrastructure of the Telegram messenger.

From a technical point of view, TON is a unique project, the technological stack of which was created from scratch, without using existing ready-made solutions. Today it is already possible to say that:

• , ;
• - (Fift FunC), ;
• , ;
• telegram-, TON;

TON

The Infinite Sharding Paradigm sounds impressive, but it needs to be carefully tested in practice, as as the number of shards increases, the number of cross-chains grows, which is likely to lead to a decrease in transaction processing speed, thus, in fact, there is an effective upper limit on throughput. TON plans to use what the creators themselves call “instant routing in the hypercube” to route cross-chains and blocks between char-chains. However, although a routing path can be found efficiently, this is not the final solution to the underlying problem of scaling the TON blockchain and increasing throughput.

While it is impossible to unequivocally say that the TON sharding approach is unsuccessful, nevertheless it should be noted that the “Paradigm of Infinite Sharding” is a bit misleading, and the number of shardchains should most likely remain within reasonable limits.

Decentralization and consensus

In fact, in order to be eligible to participate in consensus building, TON nodes must have a certain share of the asset (Gram tokens) on deposit. The disadvantage of this model is that it can lead to the fact that the rich will become even richer, since the share of the asset is the only determining factor and, accordingly, TON provides rich people with the means to finance the management of PoS nodes, which means that only large organizations with sufficient funding will be receive rewards from most sites without effort and technical know-how.

The consensus PoS nodes are responsible for producing blocks for each shardchain on the network. As a result, the number of such mini-blockchains will dynamically increase depending on the transaction load, while PoS nodes are pseudo-randomly assigned to control a specific sector of the shardchains, but the number of consensus nodes themselves does not grow proportionally. Initially, there will be only about a hundred PoS nodes that are responsible for creating, distributing, and synchronizing all blocks and shardchains. Thus, this creates a large load on PoS nodes both in terms of the required bandwidth and processing power.

Grams Wallet - Gram Token Storage App

In the light of the recently published set of rules on the use of the blockchain service for storing native TON Grams Wallet tokens , which will be integrated directly into the Telegram messenger, many members of the blockchain community have concerns that the Telegram FZ-LLC management intends to block suspicious accounts and unconditionally execute any instructions from regulators in all jurisdictions (even authoritarian countries).

However, with a more detailed study of the document, it becomes clear that this User Agreement does not contain such measures or restrictions, but only that the user must be over 18 years old and does not plan to use the application for purposes that violate legal norms, regulatory conditions and regulations acting in specific jurisdictions and in case of violation of these rules and regulations, Telegram FZ-LLC is not responsible for the actions of such users.

Thus, this document is a standard disclaimer that can be encountered with any other open source software (such as the Linux distribution). In this case, special attention should be paid to one very important section (p. 4, p. 4.3), which refers to the fact that the company does not manage the Telegram Open Network blockchain:

“We do not control the TON Blockchain and are unable to cancel or modify the data of transactions already completed.”

Our open source

Go-binding library

As already mentioned, the only way to interact with the TON network is to use the TONLIB library.

Since the Mercuryo project is developing in Go language, we decided to share our achievements with the community by publishing a wrapper library that we use ourselves.
https://github.com/mercuryoio/tonlib-go

The tonlib api specification is still far from its final state, but now you can use it to perform almost any action related to wallet functionality:

• Key operations (create / delete / export / import / change password);
• Sending messages / gram / boc files in a transaction;
• Getting the status of the wallet and information about it;
• Getting a list of transactions by wallet;
• Tongo is a lightweight wallet utility;

At the moment, the priority in the development of the library are:

• Network monitoring. The ability to receive information on each transaction from all network blocks. We are really looking forward to support from tonlib itself;
• Interactions with contracts. Work is already underway;
• Extending the functionality of the tongo console assistant. We are trying to add something new with each release;
• Generation of interface structures according to tl specification. It will allow us to be more mobile and release updates with minimal delays;

We will continue a series of posts about testing the Telegram Open Network, Grams Wallet and will share our observations.