NSQ

Intro

Architecture

• directory service
  • other nodes have to register there
  • usually more directory services run at the same time
  • good idea to register to more directory services
• NSQ daemons
  • used to receive, store, and send messages
  • messages are persistent, i.e. are stored during restart
  • possible to use multiple NSQ daemons to handle the same message
• web admin interface
  • communicates with directory services
  • also with NSQ daemons if needed

Components

• nsqlookupd
• nsqd
• nsqadmin
• utilities
• client libraries

nsqd

• directory service
• knows about other services (address+port number)
• knows which service(s) has the specified topic
• possible to have multiple lookups
• clients are able to connect to multiple lookups as well
• (this is different from Zookeeper-like solution)

nsqlookupd

• receives messages
• store messages into queues
• delivers messages to clients
• supports persistent messages
• possible to have multiple nsqds having the same topic+channel
  • -> very high resilience

nsqadmin

• web interface to monitor the whole system
• primary communicates with nsqlookupds
• secondary can communicate with nsqds

Utilities

• there’s a lot of third-party utilities

Client libraries

go-nsq                Go
pynsq                 Python
nsqjs                 JavaScript
elixir_nsq            Elixir
ensq                  Erlang
nsq-j                 Java
JavaNSQClient         Java
TrendrrNSQClient      Java
nsqjava               Java
nsq.js                JavaScript
node-nsqueue          JavaScript
knsq                  Kotlin
NsqSharp              .NET
php-nsq               PHP
phpnsq                PHP
nsqphp                PHP
nsq-py                Python
gnsq                  Python
krakow                Ruby
ruby_nsq              Ruby
evnsq                 C++
libnsq                C
hsnsq                 Haskell
nsq-java              Java
nsq-client            JavaScript
NSQnet                .NET
perl-anyevent-nsq     Perl
NsqSpinner            Python
nsq-ruby              Ruby
uvnsq                 C++11
nsq-clojure           Clojure
nsqie                 Scala
nodensq               JavaScript
nsqueue               Rust
asyncnsq              Python
nsq-ocaml             OCaml
node-red-contrib-nsq  NodeRED Nodes
asyncnsq              Rust
tokio-nsq             Rust
ansq                  Python

Topology patterns

• Simple producer-consumer pattern with multiple consumers

schema

• Dtto but with support for persistent messages

schema

• Distributed architecture

• possible to use any load balancer on API level

schema

• Solution for higher I/O throughput

schema

• Multiple nsqlookupds

schema

Building and running

Nsq compiling

$ git clone https://github.com/nsqio/nsq
 
Cloning into 'nsq'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 11117 (delta 2), reused 7 (delta 1), pack-reused 11099
Receiving objects: 100% (11117/11117), 12.18 MiB | 505.00 KiB/s, done.
Resolving deltas: 100% (6658/6658), done.
Checking connectivity... done.

Links

NSQ documentation

• A realtime distributed messaging platform
• NSQ Topology Patterns

Presentations about NSQ

• Realtime Distributed Message Processing at Scale with NSQ

Czech articles about NSQ

• NSQ – systém pro doručování zpráv bez centrálního message brokera
• NSQ – systém pro doručování zpráv bez centrálního message brokera (dokončení)

Other links

• Idempotence (Wikipedia)