{"id":7821,"date":"2019-04-09T13:51:00","date_gmt":"2019-04-09T13:51:00","guid":{"rendered":"https:\/\/nkn.org\/community\/blog\/introducing-decentralized-pub-sub-based-on-nkn\/"},"modified":"2020-12-12T16:00:37","modified_gmt":"2020-12-12T16:00:37","slug":"introducing-decentralized-pub-sub-based-on-nkn","status":"publish","type":"post","link":"https:\/\/nkn.org\/zh-hans\/community\/blog\/introducing-decentralized-pub-sub-based-on-nkn\/","title":{"rendered":"Introducing Decentralized Pub\/Sub Based on NKN"},"content":{"rendered":"<p><em>by Yilun Zhang, CTO of NKN.org<\/em><\/p>\n<figure>\n<p><img decoding=\"async\" data-width=\"1600\" data-height=\"900\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*-CcJQViswGj_JpSH\"><br \/>\n<\/figure>\n<h3>What is\u00a0Pub\/Sub<\/h3>\n<p>One basic function of NKN client (e.g. <a href=\"https:\/\/github.com\/nknorg\/nkn-client-js\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/nknorg\/nkn-client-js<\/a>) is to provide a <strong>decentralized message system<\/strong>, which includes unicast, multicast and anycast. That\u2019s pretty much enough if the message sender knows who are the receivers. However, in lots of common scenarios, the receiver should be logically decoupled from the sender. For example, when I send a message to a chatroom, I don\u2019t want to know exactly who is in the chatroom, I just want whoever in the chatroom to receive my message. That\u2019s where pub\/sub comes into place.<\/p>\n<p>Basically, <strong>pub\/sub (short for publish\/subscribe) is a model that decouples message sender (publisher) and receiver (subscribers)<\/strong>. Publishers publish message to <strong>topic<\/strong> (we only consider topic based pub\/sub here) without having to know who subscribe to the topic and will receive the message. Subscribers subscribe to topics and will receive messages sent to subscribed topics. <strong>Pub\/sub is a basic building block for modern applications<\/strong>, and has been widely used from infrastructure level (e.g. load balancing) to application level (e.g. chat room\/messenger).<\/p>\n<p>The following diagram from Google Cloud (<a href=\"https:\/\/cloud.google.com\/pubsub\/docs\/overview\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/cloud.google.com\/pubsub\/docs\/overview<\/a>) shows the relationship between publisher, subscriber and topic.<\/p>\n<figure class=\"wp-caption\">\n<p><img decoding=\"async\" data-width=\"1600\" data-height=\"1055\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*6UtbVI5L7kSbD5Xs\"><figcaption class=\"wp-caption-text\"><a href=\"https:\/\/cloud.google.com\/pubsub\/docs\/overview\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/cloud.google.com\/pubsub\/docs\/overview<\/a><\/figcaption><\/figure>\n<h3>Challenges of Decentralized Pub\/Sub<\/h3>\n<p>Cloud providers like Google Cloud, AWS provide cloud based pub\/sub, but their centralized nature makes them hard (if not impossible) to be used in decentralized applications.<\/p>\n<p>On the other hand, building decentralized pub\/sub is also challenging, as most existing decentralized systems (e.g. Ethereum) are not well suited for real time messages\u200a\u2014\u200aif sending a single message would cost over a cent in USD and almost a minute to deliver, not to mention the scalability issue, how could you expect it to be practically useful?<\/p>\n<p>To be more general, there are a few <strong>challenges<\/strong> in building decentralized pub\/sub based on existing decentralized system (most likely blockchain based):<\/p>\n<ul>\n<li>Message needs to be delivered in real time<\/li>\n<li>Message delivery needs to be affordable<\/li>\n<li>Message throughput needs to be horizontal scalable<\/li>\n<\/ul>\n<p>If you come from a non-blockchain background, you are probably laughing at how trivial the above \u201cchallenges\u201d are. Well the reality is, if we rely on on-chain transaction to transmit information, it\u2019s still very hard to solve the above problems.<\/p>\n<p>One solution to these problems is to use <strong>off-chain message delivery<\/strong>. That\u2019s why we believe <strong>NKN is perfect for being the infrastructure of a decentralized pub\/sub system<\/strong>: message delivery in NKN is <strong>instant<\/strong> (millisecond level end to end latency), <strong>free<\/strong>, and <strong>horizontal scalable<\/strong> (more nodes \u2192 higher throughput) because it\u2019s purely off-chain.<\/p>\n<h3>Building Decentralized Pub\/Sub<\/h3>\n<p>To build a pub\/sub system, we need to solve two basic problems: <strong>how to store and retrieve topic-subscriber info<\/strong>, and <strong>how to deliver messages<\/strong>. While NKN naturally solves the second one, we still need to decide where the subscriber information is stored.<\/p>\n<p>After quite a lot of discussion, we decide to <strong>store the topic-subscriber information on-chain<\/strong>. As the result, subscribe needs to be done in a transaction, which will be reliable but not horizontal scalable. Luckily, subscribe should be a much less frequent action compared to publish (which will be off-chain and horizontal scalable), so it shouldn\u2019t be the bottleneck.<\/p>\n<p>After some work and tests, we are now happy to say that <strong>our pub\/sub is working quite nicely<\/strong>. Because publishing is mainly sending off-chain messages, it is integrated into NKN client (e.g. <a href=\"https:\/\/github.com\/nknorg\/nkn-client-js\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/nknorg\/nkn-client-js<\/a>). Subscribing, on the other hand, is integrated into NKN wallet (e.g. <a href=\"https:\/\/github.com\/nknorg\/nkn-wallet-js\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/nknorg\/nkn-wallet-js<\/a>) as it needs to sign and send transactions. Both are integrated into NKN SDK (e.g. <a href=\"https:\/\/github.com\/nknorg\/nkn-sdk-go\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/nknorg\/nkn-sdk-go<\/a>) which has both NKN client and NKN wallet.<\/p>\n<h3>Using Pub\/Sub<\/h3>\n<p>Details of how to use pub\/sub can be found in the documentation of various NKN client\/wallet\/SDK implementations. The API is quite straightforward in general. For example, in the JavaScript implementation, subscribing to a topic is as simple as<\/p>\n<pre>wallet.subscribe(topic, bucket, duration)<\/pre>\n<p>The reason that we have <strong>bucket<\/strong> concept is to avoid (accidental) message flooding in the case of massive subscribers, and can be hidden by higher level APIs like <em>SubscribeToFirstAvailableBucket<\/em>. Similarly, publishing to a topic is as simple as<\/p>\n<pre>client.publish(topic, bucket, message)<\/pre>\n<p>and bucket count of a topic can be get by APIs like <em>GetTopicBucketsCount<\/em>. Clients subscribed to the topic can listen for messages with<\/p>\n<pre>client.on(\u2018message\u2019, (src, payload, payloadType) =&gt; {<br>});<\/pre>\n<h3>Use Cases and\u00a0Summary<\/h3>\n<p>Pub\/sub has been <strong>widely used<\/strong> in many systems and applications. In additional to the existing use cases, I\u2019d like to dive into <strong>a new class that is more suitable and unique to decentralized applications<\/strong>.<\/p>\n<p>Most centralized applications are not open sourced, and protocol is often bound with the application only. Decentralized applications, on the other hand, are most likely open sourced, and protocol is decoupled from the implementation to allow cross implementation communication. This greatly reduces the friction to design and implement cross application protocols. If multiple applications want to share the same information flow, a decentralized, application neutral and language neutral pub\/sub platform would be essential.<\/p>\n<p>Here are some examples:<\/p>\n<ul>\n<li>Different service providers want to share the same service discovery mechanism<\/li>\n<li>Multiple applications want to share the same rating system<\/li>\n<li>Applications would like to pass data to downstreaming applications sharing the protocol<\/li>\n<\/ul>\n<p>Our decentralized pub\/sub could be used to achieve these goals. To be more general, this represents one of the most interesting properties (in my opinion) of decentralized applications\u200a\u2014\u200a<strong>the separation of application, protocol, and data<\/strong>. NKN\u2019s technology is uniquely positioned and purposely built to capitalize upon this opportunity. We will soon release something built on our decentralized pub\/sub framework, so please stay tuned!<\/p>\n<h3>About NKN<\/h3>\n<p>NKN is the new kind of P2P network connectivity protocol &amp; ecosystem powered by a novel public blockchain. We use economic incentives to motivate Internet users to share network connection and utilize unused bandwidth. NKN\u2019s open, efficient, and robust networking infrastructure enables application developers to build the decentralized Internet so everyone can enjoy secure, low cost, and universally accessible connectivity.<\/p>\n<figure>\n<p><img decoding=\"async\" data-width=\"1000\" data-height=\"316\" src=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*3ydmKFrIsTDxMJs1\"><br \/>\n<\/figure>\n<p>Home:<a href=\"https:\/\/nkn.org\/zh-hans\/\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/nkn.org\/<\/a><\/p>\n<p>Email: <a href=\"mailto:contact@nkn.org\" target=\"_blank\" rel=\"noopener noreferrer\">contact@nkn.org<\/a><\/p>\n<p>Telegram:<a href=\"https:\/\/t.me\/nknorg\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/t.me\/nknorg<\/a><\/p>\n<p>Twitter:<a href=\"https:\/\/twitter.com\/NKN_ORG\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/twitter.com\/NKN_ORG<\/a><\/p>\n<p>Forum: <a href=\"https:\/\/forum.nkn.org\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/forum.nkn.org<\/a><\/p>\n<p>Medium:<a href=\"https:\/\/medium.com\/nknetwork\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/medium.com\/nknetwork<\/a><\/p>\n<p>Linkedin:<a href=\"https:\/\/www.linkedin.com\/company\/nknetwork\/\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/www.linkedin.com\/company\/nknetwork\/<\/a><\/p>\n<p>Github:<a href=\"https:\/\/github.com\/nknorg\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/github.com\/nknorg<\/a><\/p>\n<p>Reddit:<a href=\"https:\/\/www.reddit.com\/r\/nknblockchain\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/www.reddit.com\/r\/nknblockchain<\/a><\/p>\n<p>Discord:<a href=\"https:\/\/discord.gg\/yVCWmkC\" target=\"_blank\" rel=\"noopener noreferrer\"> https:\/\/discord.gg\/yVCWmkC<\/a><\/p>\n<p>YouTube:<a href=\"http:\/\/www.youtube.com\/c\/NKNORG\" target=\"_blank\" rel=\"noopener noreferrer\"> http:\/\/www.youtube.com\/c\/NKNORG<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>by Yilun Zhang, CTO of NKN.org What is\u00a0Pub\/Sub One basi [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"mc4wp_mailchimp_campaign":[]},"categories":[251,211],"tags":[435,391,718],"acf":[],"_links":{"self":[{"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/posts\/7821"}],"collection":[{"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/comments?post=7821"}],"version-history":[{"count":1,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/posts\/7821\/revisions"}],"predecessor-version":[{"id":7825,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/posts\/7821\/revisions\/7825"}],"wp:attachment":[{"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/media?parent=7821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/categories?post=7821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nkn.org\/zh-hans\/wp-json\/wp\/v2\/tags?post=7821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}