nLearning

Posted on May 12, 2022

Many layer 1 public blockchains (Solana, Near, Avalanche, Harmony) competing against Ethereum promise the combination of extremely high performance (transaction per second), low latency (fast finality), and low cost (low transaction fee). There are usually some catches, e.g. sacrificing security or decentralization according to Vitalik’s blockchain scaling trilemma.

Transaction spamming, or Tx spamming, is one of the persistent security attacks on these high performance blockchains that can wreak havoc for extended periods of time. The attackers might or might not gain any financial benefit, but can certainly make everybody’s life quite miserable at relatively low cost.

Definitions

Very-low-fee flooding (Source: Bitcoin wiki)

A simplistic attack that people often try is to send tens of thousands of zero- or very-low-fee transactions at once. This makes the “mempool” number shown by some websites go up massively, which sometimes causes people to panic, but in reality it is only a slight drain on the network’s bandwidth resources, and hardly a problem at all. Transactions with very low fees will never be confirmed, and will typically expire from nodes’ mempools ((a temporary queue of transactions to be processed) after a few days.

In addition, Tx spamming can also be considered as a form of Distributed Denial of Service (DDOS).

Spam Detection

Spam transactions might be malicious or unintentional (e.g. smart contract code error). It could even be a performance test by a curious tinkler. And it is not feasible to eradicate spam traffic. So how do we detect there is serious spamming in the network?

Disregard the short-term spikes of traffic, some of the symptoms that the network is under spam attacks are (in order of severity):

On-chain traffic and blockchain database size are growing faster than usual. This could be gradual and accumulating over a period of time, e.g. weeks or months.
It takes much longer to complete a normal transaction, e.g. asset transfer
On-chain data traffic spikes significantly, queues of unprocessed transactions are maxed out , validator nodes are under heavy load and stressed
Validator nodes loses consensus and can no longer produce valid blocks, chain halts

Although it is not possible to predict and prevent all spam attacks in advance, it is important for protocol designers and project developers to have an active monitoring and early warning system.

Potential solutions

Someone said that “there is no such thing as an illegal transaction in blockchain”. If a transaction is properly formed according to the specification, it will be accepted into the mempool of the API node, broadcasted to the network, potentially be packed into a block by a miner, validated and accepted into the blockchain.

The key question is how to prioritize productive transactions over spam transactions. There are technical and economic solutions to the Tx spamming problem. And here is a list of common solutions. Please note that many blockchains already implement some of the Tx prioritization, thus the solution can be a combination of implementing new features and adjusting parameters for existing features.

Priority Based on Transaction Fees
Priority Based on throughput
Priority Based on Reputation
After-the-Fact Filtering

Since the 3rd solution is not widely used and the 4th solution is only relevant for historical data, we will only focus on the first two solutions in this article.

Increase Minimum Transaction Fees

The first option, increase the minimum Tx fee, is a simple and natural solution. Since Tx spammers will send thousands, millions or even billions of low-fee or no-fee transactions, raising the minimum Tx fee by just a little might already become prohibitively expensive for the attacker. While such a small increase might not have any material impact on the legitimate users of the network.

There are arguments not to introduce a non-zero minimum transaction fee, for example, to simplify user onboarding of dApps with a wallet of zero balance. However, protocol designers can introduce a configurable max amount of “free” transactions a miner node will accept. This might work in some way, but Tx spammers are known to abuse these “free” transactions, and render the scheme almost useless.

Increase Transaction Processing Capacity

Consensus nodes or mining nodes, bear the majority of burden for processing transactions. Increasing their capacity can have a direct impact. Scaling up raw processing power, such as more CPU cores, more RAM, more and faster SSD storage, and higher capacity lower latency Internet connections.

Some consensus nodes in the blockchain network have additional functionality, such as RPC server for API access of the blockchain, as well as storing and providing historical data for blockchain browsers. These can become bottlenecks too, if they are not properly provisioned for load or if there are inefficiencies in I/O, database schema, or business logic. Fixing those problems can improve the usability of the dApps, who rely on these services, during a Tx spamming storm..

Yet another option is to prioritize transactions with smaller data size over larger size, if Tx fees were about equal. This way we can reduce the burden not only on the final blockchain database size, but also reduce the processing load on miner nodes.

Case study 1: Polygon, May-September 2021

What happened:

With a low 1 gwei in gas price , it takes about $1,000 to fill all blocks for an entire day. Therefore, two arbitrage trading bots have been spamming the Polygon network with about 2 million transactions daily, which is roughly 30% of the network’s total transaction count.

According to one research, the two arbitration bots might be making an average of $6,800 in daily profits from this operation.

What did the team do?

Team proposed to increase the minimum tx fee from 1 gwei to 30 gwei.

Aftermath

Shortly after the adjustment, the spam transactions dropped 75% from 2M to just 500k transactions per day. And the total daily transactions on the Polygon network almost halved from about 6M down to 3M.

Case study 2: Solana, September 2021

What happened:

On September 14th, the Solana network was offline for 17 hours. A type of DDOS using large amounts of spam transactions was the culprit.

“At 12:00 UTC, Grape Protocol launched their IDO on Raydium, and bots generated transactions which flooded the network. These transactions created a memory overflow, which caused many validators to crash forcing the network to slow down and eventually stall. The network went offline when the validator network could not come to agreement on the current state of the blockchain, which prevented the network from confirming new blocks.”

Network forensics data for Certus One’s primary validator indicates peaks of raw transaction data of >1 Gbps and 120k packets per second, essentially indistinguishable from a volumetric DDoS attack.

What did the team do

The community proposed a hard fork of the network. And within the hard fork, several fixes were implemented:

Ignore write locks on programs, so transactions can be processed in parallel
Fine-tune the RPC retry behavior, allowing applications to be more intelligent about retrying transactions.
Prioritizes vote transactions (essential for consensus algorithm), which prevents regular transactions from “drowning out” vote transactions.
Increase memory for consensus nodes, to prevent Out of Memory that stalled the nodes.

Aftermath

After two restarts, with support from the community, Solana blockchain restored its consensus. No funds were lost, and the network returned to full functionality in under 24 hours.

Case study 3: Harmony, June 2021

What happened

Starting from June 5th, Harmony blockchain has been receiving a large number of transactions on our beacon chain. These transactions have the following characteristics:

All these transactions are smart contract calls with some nested static calls in code execution.
Though these transactions come from quite a number of different addresses, the receiver addresses are limited to four contract addresses.
All transactions do not have any token transfer records, which is most likely just modifying some value within the contract storage.
The amount of transactions is huge. It peaked at ~150/block, which is about 40 transactions per second, >5 times the TPS on Ethereum.

By the time of 2:30 p.m PST June 21th, 2021, 4 addresses have received 18,311,525 transactions.

And the impacts are:

Make the blocks larger than ever, which affects the RPC sync service. Consequently, the shard 3 chain broke in consensus and had a downtime of ~2 hours.
The transaction history of a single account can become large, which makes the explorer db hard to catch up with. Since all public RPC points are also explorer nodes, this results in public RPC points experiencing some level of un-stability, and data inaccuracy related to explorer DB.

What did the team do

The Harmony team carried out three upgrades to the mainnet:

Increase the Gas limit to 1 gWei to increase the cost of spamming transactions.
A fix to adding an in-memory cache for explorer database (DB): Link. This serves as a temporary fix to ease the machine resources.
An explorer database schema change that includes a database migration and the new schema logic. The idea behind this fix is to shred the bulk address information into small entries each with an address and a transaction.

Aftermath

With all the three fixes deployed, the Harmony mainnet returns to normal operation by June 18th 2022.

Case study 4: NKN, March 2022

What happened:

Starting in early March, there are a large number of spam transactions with a 1e-8 NKN txn fee. Although it neither halted the blockchain, nor created any unusually high load for the mining nodes, it did create a few annoyances:

Prevented some legit transactions from being confirmed and packed into the blockchain. For example, new node generate ID transactions as well as group subscription transactions.
These in turn resulted in: new mining nodes cannot initiate (even though existing mining nodes are fine), and new nMobile and d-chat users cannot join public or private chat groups.

What did the team do

Actually all of the impacted transaction types already have the option of specifying a non-zero transaction fee. However, those were defaulted to zero in order to improve user experience. So the team implemented the following:

Change default tx fee to 0.01 NKN for generate id transaction type
Updated nMobile and nConnect apps to add user configurable Tx fee for subscribing to a topic
Added support on web and mobile wallet to replace pending transaction by new transaction with higher Tx fees

Aftermath

Even though the spamming did not stop until early May 2022, there was no impact on normal functionality of the NKN blockchain or major apps running on top of the NKN blockchain.

Best Practice

Are there any optimal all-in-one solutions for Tx spamming attacks? Unfortunately the answer is no. Maybe the best option already exists: Ethereum’s gas fee mechanism that asks each transaction to pay their fair share of computing and storage costs. But again we are back to square one, how can we get the high performance and low cost to the growing dApps?

The best practice is to have a complete system of monitoring, early warning, and mitigation in place, with a combination of toolkits we discussed in this essay. On top of that, developers should have timely and clear communication with the entire ecosystem, and empower the community to help make economic and technical decisions that can minimize the impact of transaction spamming.

And happy transacting!