Repository files navigation

Library and command line tool not just for qdb.

This project implement a python qdb toolkit as dbnetget at first. With the progress of development, it is separated into multiple sub-projects asos-qdb-protocal,os-m3-engine, this project become a framework for similar works not just for qdb.

The main components are client libraries and command line tool.

Client is used for network processing(connect, close, send, receive, etc.) . It is protocol-independent and transparent to user. By now, there are sync/tornado-async clients, thread-safe client pool and tornado-async client pool are also convenient.

Command line tool's features can be extended by installing extra packages. SeeInstall.

Greatly appreciateBear Tian and his dbnetget!

• install package

  pip install os-dbnetget

• install extra packages

  subpackage	install command	enables
  m3	pip install os-dbnetget[m3]	Installm3 for command line tool support m3(multi-thread) engine
  tornado	pip install os-dbnetget[tornado]	InstallTornado for async client and command line tool support tornado engine
  rotate	pip install os-dbnetget[rotate]	Enable write data to rotate file

There are sync/async clients, generally speaking, you should not use them directly. The pool may be your first choice.

• native multi-thread, do not need extra packages
• thread safe
• retry when network error
• support multi connections with one endpoint

Example:

fromos_qdb_protocalimportcreate_protocalfromos_dbnetget.commands.qdbimportqdb_keyfromos_dbnetget.clients.sync_clientimportSyncClientPoolendpoints= ['host%02d:8012'%iforiinrange(1,10)]pool=SyncClientPool(endpoints)proto=create_protocal('test',qdb_key(b'test-key'))result=pool.execute(proto)pool.close()

• support tornado async
• retry when network error
• support multi connections with one endpoint

Example:

fromtornadoimportgenfromtornado.ioLoopimportIOLoopfromos_qdb_protocalimportcreate_protocalfromos_dbnetget.commands.qdbimportqdb_keyfromos_dbnetget.clients.tonado_clientimportTornadoClientPool@gen.coroutinedefmain():endpoints= ['host%02d:8012'%iforiinrange(1,10)]pool=TornadoClientPool(endpoints)proto=create_protocal('test',qdb_key(b'test-key'))result=yieldpool.execute(proto)yieldpool.close()IOLoop.current().run_sync(main)

• command line tool is on progress, the supported sub-commands:

  os-dbnetget -h

• each sub-command has is own features

  os-dbnetget [sub-command] -h

• some extra packages can be installed for enhancement, seeInstall

  for example, you can install m3 engine to improve processing capacity

  pip install os-dbnetget[m3]

  cat data.txt | os-dbnetget test --engine m3 --thread-num 50 -L endpoints.lst

$ tox

MIT licensed.