Introducing Decentralized Pub/Sub Based on NKN

Posted on April 9, 2019

by Yilun Zhang, CTO of NKN.org

What is Pub/Sub

One basic function of NKN client (e.g. https://github.com/nknorg/nkn-client-js) is to provide a decentralized message system, which includes unicast, multicast and anycast. That’s pretty much enough if the message sender knows who are the receivers. However, in lots of common scenarios, the receiver should be logically decoupled from the sender. For example, when I send a message to a chatroom, I don’t want to know exactly who is in the chatroom, I just want whoever in the chatroom to receive my message. That’s where pub/sub comes into place.

Basically, pub/sub (short for publish/subscribe) is a model that decouples message sender (publisher) and receiver (subscribers). Publishers publish message to topic (we only consider topic based pub/sub here) without having to know who subscribe to the topic and will receive the message. Subscribers subscribe to topics and will receive messages sent to subscribed topics. Pub/sub is a basic building block for modern applications, and has been widely used from infrastructure level (e.g. load balancing) to application level (e.g. chat room/messenger).

The following diagram from Google Cloud (https://cloud.google.com/pubsub/docs/overview) shows the relationship between publisher, subscriber and topic.

Challenges of Decentralized Pub/Sub

Cloud providers like Google Cloud, AWS provide cloud based pub/sub, but their centralized nature makes them hard (if not impossible) to be used in decentralized applications.

On the other hand, building decentralized pub/sub is also challenging, as most existing decentralized systems (e.g. Ethereum) are not well suited for real time messages — if sending a single message would cost over a cent in USD and almost a minute to deliver, not to mention the scalability issue, how could you expect it to be practically useful?

To be more general, there are a few challenges in building decentralized pub/sub based on existing decentralized system (most likely blockchain based):

Message needs to be delivered in real time
Message delivery needs to be affordable
Message throughput needs to be horizontal scalable

If you come from a non-blockchain background, you are probably laughing at how trivial the above “challenges” are. Well the reality is, if we rely on on-chain transaction to transmit information, it’s still very hard to solve the above problems.

One solution to these problems is to use off-chain message delivery. That’s why we believe NKN is perfect for being the infrastructure of a decentralized pub/sub system: message delivery in NKN is instant (millisecond level end to end latency), free, and horizontal scalable (more nodes → higher throughput) because it’s purely off-chain.

Building Decentralized Pub/Sub

To build a pub/sub system, we need to solve two basic problems: how to store and retrieve topic-subscriber info, and how to deliver messages. While NKN naturally solves the second one, we still need to decide where the subscriber information is stored.

After quite a lot of discussion, we decide to store the topic-subscriber information on-chain. As the result, subscribe needs to be done in a transaction, which will be reliable but not horizontal scalable. Luckily, subscribe should be a much less frequent action compared to publish (which will be off-chain and horizontal scalable), so it shouldn’t be the bottleneck.

After some work and tests, we are now happy to say that our pub/sub is working quite nicely. Because publishing is mainly sending off-chain messages, it is integrated into NKN client (e.g. https://github.com/nknorg/nkn-client-js). Subscribing, on the other hand, is integrated into NKN wallet (e.g. https://github.com/nknorg/nkn-wallet-js) as it needs to sign and send transactions. Both are integrated into NKN SDK (e.g. https://github.com/nknorg/nkn-sdk-go) which has both NKN client and NKN wallet.

Using Pub/Sub

Details of how to use pub/sub can be found in the documentation of various NKN client/wallet/SDK implementations. The API is quite straightforward in general. For example, in the JavaScript implementation, subscribing to a topic is as simple as

wallet.subscribe(topic, bucket, duration)

The reason that we have bucket concept is to avoid (accidental) message flooding in the case of massive subscribers, and can be hidden by higher level APIs like SubscribeToFirstAvailableBucket. Similarly, publishing to a topic is as simple as

client.publish(topic, bucket, message)

and bucket count of a topic can be get by APIs like GetTopicBucketsCount. Clients subscribed to the topic can listen for messages with

client.on(‘message’, (src, payload, payloadType) => {
});

Use Cases and Summary

Pub/sub has been widely used in many systems and applications. In additional to the existing use cases, I’d like to dive into a new class that is more suitable and unique to decentralized applications.

Most centralized applications are not open sourced, and protocol is often bound with the application only. Decentralized applications, on the other hand, are most likely open sourced, and protocol is decoupled from the implementation to allow cross implementation communication. This greatly reduces the friction to design and implement cross application protocols. If multiple applications want to share the same information flow, a decentralized, application neutral and language neutral pub/sub platform would be essential.

Here are some examples:

Different service providers want to share the same service discovery mechanism
Multiple applications want to share the same rating system
Applications would like to pass data to downstreaming applications sharing the protocol

Our decentralized pub/sub could be used to achieve these goals. To be more general, this represents one of the most interesting properties (in my opinion) of decentralized applications — the separation of application, protocol, and data. NKN’s technology is uniquely positioned and purposely built to capitalize upon this opportunity. We will soon release something built on our decentralized pub/sub framework, so please stay tuned!

About NKN

NKN is the new kind of P2P network connectivity protocol & ecosystem powered by a novel public blockchain. We use economic incentives to motivate Internet users to share network connection and utilize unused bandwidth. NKN’s open, efficient, and robust networking infrastructure enables application developers to build the decentralized Internet so everyone can enjoy secure, low cost, and universally accessible connectivity.