Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up

Example of High-Speed Subscriber Patterns in ZeroMQ

License

NotificationsYou must be signed in to change notification settings

ryansmccoy/zmq-high-speed-subs

Repository files navigation

Example of High-Speed Subscribers (Black Box Pattern) in ZeroMQ

The Simple Black Box Pattern

"A common use case for pub-sub is distributing large data streams like market data coming from stock exchanges. A typical setup would have a publisher connected to a stock exchange, taking price quotes, and sending them out to a number of subscribers. If there are a handful of subscribers, we could use TCP. If we have a larger number of subscribers, we'd probably use reliable multicast, i.e., PGM." - ZeroMQ Guide

Setup Environment & Run Example (Windows):

git clone https://github.com/ryansmccoy/zmq-high-speed-subscd zmq-high-speed-subsconda create -n zmq-high-speed-subs python=3.8 pyzmq pandas numpy -yactivate zmq-high-speed-subspip install -r requirements.txt

Setup Environment & Run Example (Linux):

git clone https://github.com/ryansmccoy/zmq-high-speed-subscd zmq-high-speed-subsconda create -n zmq-high-speed-subs python=3.8 pyzmq pandas numpy -ysource activate zmq-high-speed-subspip install -r requirements.txt

To run, open seperate terminal windows, run:

Publisher

$   activate zmq-high-speed-subs$   python zmq_high_speed_subs\publisher_fakefeed.pyOutput:    2019-07-31 06:21:45,898 INFO     [Publisher-1(30632)]    2019-07-31 06:21:45,967 INFO     [Publisher-1(30632)]    2019-07-31 06:21:45,986 INFO     [Publisher-1(30632)] Time Elapsed: 10.1 seconds    2019-07-31 06:21:46,018 INFO     [Publisher-1(30632)] Messages During Period:       55008    2019-07-31 06:21:46,026 INFO     [Publisher-1(30632)] Messages Per Second:  5443.9    2019-07-31 06:21:46,032 INFO     [Publisher-1(30632)]    2019-07-31 06:21:46,041 INFO     [Publisher-1(30632)] Total Messages Published:    32950459    2019-07-31 06:21:46,059 INFO     [Publisher-1(30632)]    2019-07-31 06:21:46,065 INFO     [Publisher-1(30632)]

Subscriber-to-Pusher

$   activate zmq-high-speed-subs$   python zmq_high_speed_subs/_10_manager.pyOutput:    2019-07-31 04:57:47,588 INFO     [ZMQSubscriber(31384)]    2019-07-31 04:57:47,588 INFO     [ZMQSubscriber(31384)] Time Elapsed:   10 seconds    2019-07-31 04:57:47,589 INFO     [ZMQSubscriber(31384)] Messages During Period: 7359215    2019-07-31 04:57:47,589 INFO     [ZMQSubscriber(31384)] Messages Per Second:    5502.83    2019-07-31 04:57:47,589 INFO     [ZMQSubscriber(31384)]    2019-07-31 04:57:47,589 INFO     [ZMQSubscriber(31384)] Total Time Elapsed:     1516.85 seconds    2019-07-31 04:57:47,590 INFO     [ZMQSubscriber(31384)] Total Messages: 7359215    2019-07-31 04:57:47,590 INFO     [ZMQSubscriber(31384)] Total Messages Per Second:      4851.64    2019-07-31 04:57:47,591 INFO     [ZMQSubscriber(31384)]    2019-07-31 04:57:47,591 INFO     [ZMQSubscriber(31384)] Current _Queue Size:    23    2019-07-31 04:57:47,592 INFO     [ZMQSubscriber(31384)]    2019-07-31 04:57:55,951 INFO     [ZMQPusher(45028)]    2019-07-31 04:57:55,951 INFO     [ZMQPusher(45028)] Time Elapsed:       10 seconds    2019-07-31 04:57:55,952 INFO     [ZMQPusher(45028)] Messages During Period:     55008    2019-07-31 04:57:55,952 INFO     [ZMQPusher(45028)] Messages Per Second:        5494.9    2019-07-31 04:57:55,952 INFO     [ZMQPusher(45028)]    2019-07-31 04:57:55,953 INFO     [ZMQPusher(45028)] Total Time Elapsed: 1524.16 seconds    2019-07-31 04:57:55,953 INFO     [ZMQPusher(45028)] Total Messages:     7414199    2019-07-31 04:57:55,953 INFO     [ZMQPusher(45028)] Total Messages Per Second:  4864.46    2019-07-31 04:57:55,954 INFO     [ZMQPusher(45028)]    2019-07-31 04:57:55,954 INFO     [ZMQPusher(45028)] Current _Queue Size:        0    2019-07-31 04:57:55,954 INFO     [ZMQPusher(45028)]    2019-07-31 04:57:56,005 INFO     [ZMQPuller (30656)]    2019-07-31 04:57:56,005 INFO     [ZMQPuller (30656)] Time Elapsed:      10 seconds    2019-07-31 04:57:56,006 INFO     [ZMQPuller (30656)] Messages During Period:    55000    2019-07-31 04:57:56,006 INFO     [ZMQPuller (30656)] Messages Per Second:       5484.78    2019-07-31 04:57:56,006 INFO     [ZMQPuller (30656)]    2019-07-31 04:57:56,007 INFO     [ZMQPuller (30656)] Total Time Elapsed:        1517.3 seconds    2019-07-31 04:57:56,007 INFO     [ZMQPuller (30656)] Total Messages Distributed to finished Workers:    7390000    2019-07-31 04:57:56,008 INFO     [ZMQPuller (30656)] Total Messages Per Second: 4870.48    2019-07-31 04:57:56,008 INFO     [ZMQPuller (30656)]    2019-07-31 04:57:56,010 INFO     [ZMQPuller (30656)]    Total Messagesin _Queue:        1699    2019-07-31 04:57:56,010 INFO     [ZMQPuller (30656)]    Currently Running Workers:       12    2019-07-31 04:57:56,011 INFO     [ZMQPuller (30656)]    2019-07-31 04:57:53,607 INFO     [Worker    (23848)]    2019-07-31 04:57:53,607 INFO     [Worker    (23848)] Worker Messages Time Elapsed:      1.71 seconds    2019-07-31 04:57:53,607 INFO     [Worker    (23848)] Worker Messages:   2501    2019-07-31 04:57:53,608 INFO     [Worker    (23848)] Worker Messages Per Second:        1458.61    2019-07-31 04:57:53,608 INFO     [Worker    (23848)]    2019-07-31 04:57:53,608 INFO     [Worker    (23848)] COMPLETED    2019-07-31 04:57:53,608 INFO     [Worker    (23848)]

If want to connect to database, rename .env.template to .env and enter your data base info:

# .envDB_HOST=localhostDB_DATABASE=zmq-exampleDB_TABLE=dataDB_USERNAME=pythonDB_PASSWORD=h1ghsp33d
[8]ページ先頭
©2009-2025 Movatter.jp