Repository files navigation

Package golang provides Go-like features for Python:

• gpython is Python interpreter with support for lightweight threads.
• go spawns lightweight thread.
• chan and select provide channels with Go semantic.
• func allows to define methods separate from class.
• defer allows to schedule a cleanup from the main control flow.
• error and package errors provide error chaining.
• b, u and bstr/ustr provide uniform UTF8-based approach to strings.
• gimport allows to import python modules by full path in a Go workspace.

Package golang.pyxprovides similar features for Cython/nogil.

Additional packages and utilities are alsoprovided to close other gapsbetween Python/Cython and Go environments.

Command gpython provides Python interpreter that supports lightweight threadsvia tight integration withgevent. The standard library of GPython is APIcompatible with Python standard library, but inplace of OS threads lightweightcoroutines are provided, and IO is internally organized vialibuv/libev-based IO scheduler. Consequently programs can spawn lots ofcoroutines cheaply, and modules like time, socket, ssl, subprocess etc -all could be used from all coroutines simultaneously, and in the same blocking wayas if every coroutine was a full OS thread. This gives ability to scale programswithout changing concurrency model and existing code.

Additionally GPython sets UTF-8 to be default encoding always, and puts go,chan, select etc into builtin namespace.

go spawns a coroutine, or thread if gevent was not activated. It is possible toexchange data in between either threads or coroutines via channels. chancreates a new channel with Go semantic - either synchronous or buffered. Usechan.recv, chan.send and chan.close for communication. nilchanstands for nil channel. select can be used to multiplex on severalchannels. For example:

ch1 = chan()    # synchronous channelch2 = chan(3)   # channel with buffer of size 3def _():    ch1.send('a')    ch2.send('b')go(_)ch1.recv()      # will give 'a'ch2.recv_()     # will give ('b', True)ch2 = nilchan   # rebind ch2 to nil channel_, _rx = select(    ch1.recv,           # 0    ch1.recv_,          # 1    (ch1.send, obj),    # 2    ch2.recv,           # 3    default,            # 4)if _ == 0:    # _rx is what was received from ch1    ...if _ == 1:    # _rx is (rx, ok) of what was received from ch1    ...if _ == 2:    # we know obj was sent to ch1    ...if _ == 3:    # this case will be never selected because    # send/recv on nil channel block forever.    ...if _ == 4:    # default case    ...

By default chan creates new channel that can carry arbitrary Python objects.However type of channel elements can be specified via chan(dtype=X) - forexample chan(dtype='C.int') creates new channel whose elements are Cintegers. chan.nil(X) creates typed nil channel.Cython/nogil APIexplains how channels with non-Python dtypes, besides in-Python usage, can beadditionally used for interaction in between Python and nogil worlds.

func decorator allows to define methods separate from class.

For example:

@func(MyClass)def my_method(self, ...):    ...

will define MyClass.my_method().

func can be also used on just functions, for example:

@funcdef my_function(...):    ...

defer allows to schedule a cleanup to be executed when current functionreturns. It is similar to try/finally but does not force the cleanup partto be far away in the end. For example:

wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)defer(wc.close)         │     try:                        │        ......                     │        ......                     │        ......                     │     finally:                        │        wc.close()

If deferred cleanup fails, previously unhandled exception, if any, won't belost - it will be chained with (PEP 3134) and included into traceback dumpeven on Python2.

For completeness there is recover and panic that allow to program withGo-style error handling, for example:

def _():   r = recover()   if r is not None:      print("recovered. error was: %s" % (r,))defer(_)...panic("aaa")

But recover and panic are probably of less utility since they can bepractically natively modelled with try/except.

If defer is used, the function that uses it must be wrapped with @funcdecorator.

In concurrent systems operational stack generally differs from execution codeflow, which makes code stack traces significantly less useful to understand anerror. Pygolang provides support for error chaining that gives ability to buildoperational error stack and to inspect resulting errors:

error is error type that can be used by itself or subclassed. Byproviding .Unwrap() method, an error can optionally wrap another error thisway forming an error chain. errors.Is reports whether an item in error chainmatches target. fmt.Errorf provides handy way to build wrapping errors.For example:

e1 = error("problem")e2 = fmt.Errorf("doing something for %s: %w", "joe", e1)print(e2)         # prints "doing something for joe: problem"errors.Is(e2, e1) # gives True# OpError is example class to represents an error of operation op(path).class OpError(error):   def __init__(e, op, path, err):      e.op   = op      e.path = path      e.err  = err   # .Error() should be used to define what error's string is.   # it is automatically used by error to also provide both .__str__ and .__repr__.   def Error(e):      return "%s %s: %s" % (e.op, e.path, e.err)   # provided .Unwrap() indicates that this error is chained.   def Unwrap(e):      return e.errmye = OpError("read", "file.txt", io.ErrUnexpectedEOF)print(mye)                          # prints "read file.txt: unexpected EOF"errors.Is(mye, io.EOF)              # gives Falseerrors.Is(mye. io.ErrUnexpectedEOF) # gives True

Both wrapped and wrapping error can be of arbitrary Python type - notnecessarily of error or its subclass.

error is also used to represent at Python level an error returned byCython/nogil call (seeCython/nogil API) and preserves Cython/nogil errorchain for inspection at Python level.

Pygolang error chaining integrates with Python error chaining and takes.__cause__ attribute into account for exception created via raise X from Y(PEP 3134).

Pygolang, similarly to Go, provides uniform UTF8-based approach to strings withthe idea to make working with byte- and unicode- strings easy and transparentlyinteroperable:

• bstr is byte-string: it is based on bytes and can automatically convert to/from unicode[*].
• ustr is unicode-string: it is based on unicode and can automatically convert to/from bytes.

The conversion, in both encoding and decoding, never fails and never loosesinformation: bstr→ustr→bstr and ustr→bstr→ustr are always identityeven if bytes data is not valid UTF-8.

Both bstr and ustr represent stings. They are two differentrepresentations of the same entity.

Semantically bstr is array of bytes, while ustr is array ofunicode-characters. Accessing their elements by [index] and iterating them yield byte andunicode character correspondingly[†]. However it is possible to yield unicodecharacter when iterating bstr via uiter, and to yield byte character wheniterating ustr via biter. In practice bstr + uiter is enough 99% ofthe time, and ustr only needs to be used for random access to stringcharacters. SeeStrings, bytes, runes and characters in Go for overview ofthis approach.

Operations in between bstr and ustr/unicode / bytes/bytearray coerce to bstr, whileoperations in between ustr and bstr/bytes/bytearray / unicode coerceto ustr. When the coercion happens, bytes and bytearray, similarly tobstr, are also treated as UTF8-encoded strings.

bstr and ustr are meant to be drop-in replacements for standardstr/unicode classes. They support all methods of str/unicode and inparticular their constructors accept arbitrary objects and either convert or stringify them. Forcases when no stringification is desired, and one only wants to convertbstr/ustr / unicode/bytes/bytearray, or an object with bufferinterface[‡], to Pygolang string, b and u provide way to make sure anobject is either bstr or ustr correspondingly.

Usage example:

s  = b('привет')     # s is bstr corresponding to UTF-8 encoding of 'привет'.s += ' мир'          # s is b('привет мир')for c in uiter(s):   # c will iterate through     ...             #     [u(_) for _ in ('п','р','и','в','е','т',' ','м','и','р')]# the following gives b('привет мир труд май')b('привет %s %s %s') % (u'мир',                  # raw unicode                        u'труд'.encode('utf-8'), # raw bytes                        u('май'))                # ustrdef f(s):   s = u(s)          # make sure s is ustr, decoding as UTF-8(*) if it was bstr, bytes, bytearray or buffer.   ...               # (*) the decoding never fails nor looses information.

gimport provides way to import python modules by full path in a Go workspace.

For example

lonet = gimport('lab.nexedi.com/kirr/go123/xnet/lonet')

will import either

• lab.nexedi.com/kirr/go123/xnet/lonet.py, or
• lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py

located in src/ under $GOPATH.

Cython package golang providesnogil API with goroutines, channels andother features that mirror corresponding Python package. Cython API is not onlyfaster compared to Python version, but also, due tonogil property, allows tobuild concurrent systems without limitations imposed by Python's GIL. All thatwhile still programming in Python-like language. Brief description ofCython/nogil API follows:

go spawns new task - a coroutine, or thread, depending on activated runtime.chan[T] represents a channel with Go semantic and elements of type T.Use makechan[T] to create new channel, and chan[T].recv, chan[T].send,chan[T].close for communication. nil stands for nil channel. selectcan be used to multiplex on several channels. For example:

cdef nogil:   struct Point:      int x      int y   void worker(chan[int] chi, chan[Point] chp):      chi.send(1)      cdef Point p      p.x = 3      p.y = 4      chp.send(p)   void myfunc():      cdef chan[int]   chi = makechan[int]()       # synchronous channel of integers      cdef chan[Point] chp = makechan[Point](3)    # channel with buffer of size 3 and Point elements      go(worker, chi, chp)      i = chi.recv()    # will give 1      p = chp.recv()    # will give Point(3,4)      chp = nil         # rebind chp to nil channel      cdef cbool ok      cdef int j = 33      _ = select([          chi.recvs(&i),         # 0          chi.recvs(&i, &ok),    # 1          chi.sends(&j),         # 2          chp.recvs(&p),         # 3          default,               # 4      ])      if _ == 0:          # i is what was received from chi          ...      if _ == 1:          # (i, ok) is what was received from chi          ...      if _ == 2:          # we know j was sent to chi          ...      if _ == 3:          # this case will be never selected because          # send/recv on nil channel block forever.          ...      if _ == 4:          # default case          ...

Python channels are represented by pychan cdef class. Pythonchannels that carry non-Python elements (pychan.dtype != DTYPE_PYOBJECT) canbe converted to Cython/nogil chan[T] via pychan.chan_*().Similarly Cython/nogil chan[T] can be wrapped into pychan viapychan.from_chan_*(). This provides interaction mechanismin betweennogil and Python worlds. For example:

def myfunc(pychan pych):   if pych.dtype != DTYPE_INT:      raise TypeError("expected chan[int]")   cdef chan[int] ch = pych.chan_int()  # pychan -> chan[int]   with nogil:      # use ch in nogil code. Both Python and nogil parts can      # send/receive on the channel simultaneously.      ...def mytick(): # -> pychan   cdef chan[int] ch   with nogil:      # create a channel that is connected to some nogil task of the program      ch = ...   # wrap the channel into pychan. Both Python and nogil parts can   # send/receive on the channel simultaneously.   cdef pychan pych = pychan.from_chan_int(ch)  # pychan <- chan[int]   return pych

error is the interface that represents errors. errors.New and fmt.errorfprovide way to build errors from text. An error can optionally wrap anothererror by implementing errorWrapper interface and providing .Unwrap() method.errors.Is reports whether an item in error chain matches target. fmt.errorfwith %w specifier provide handy way to build wrapping errors. For example:

e1 = errors.New("problem")e2 = fmt.errorf("doing something for %s: %w", "joe", e1)e2.Error()        # gives "doing something for joe: problem"errors.Is(e2, e1) # gives True

An error can be exposed to Python via pyerror cdef class wrapperinstantiated by pyerror.from_error(). pyerror preserves Cython/nogil errorchain for inspection by Python-level error.Is.

panic stops normal execution of current goroutine by throwing a C-levelexception. On Python/C boundaries C-level exceptions have to be converted toPython-level exceptions with topyexc. For example:

cdef void _do_something() nogil:   ...   panic("bug")   # hit a bug# do_something is called by Python code - it is thus on Python/C boundarycdef void do_something() nogil except +topyexc:   _do_something()def pydo_something():   with nogil:      do_something()

Seelibgolang.h andgolang.pxd for details of the API.See alsotestprog/golang_pyx_user/ for demo project that uses Pygolang inCython/nogil mode.

The following additional packages and utilities are also provided to close gapsbetween Python/Cython and Go environments:

In addition to go and channels, the following packages are provided to helphandle concurrency in structured ways:

• golang.context (py,pyx) provides contexts to propagate deadlines, cancellation andtask-scoped values among spawned goroutines[§].
• golang.sync (py,pyx) provides sync.WorkGroup to spawn group of goroutines workingon a common task. It also provides low-level primitives - for examplesync.Once, sync.WaitGroup, sync.Mutex and sync.RWMutex - that aresometimes useful too.
• golang.time (py,pyx) provides timers integrated with channels.
• golang.os.signal (py,pyx) provides signal handling via channels.

qq (import from golang.gcompat) provides %q functionality that quotes asGo would do. For example the following code will print name quoted in "without escaping printable UTF-8 characters:

print('hello %s' % qq(name))

qq accepts both str and bytes (unicode and str on Python2)and also any other type that can be converted to str.

Packagegolang.strconv provides direct access to conversion routines, forexample strconv.quote and strconv.unquote.

py.bench allows to benchmark python code similarly to go test -bench and py.test.For example, running py.bench on the following code:

def bench_add(b):    x, y = 1, 2    for i in xrange(b.N):        x + y

gives something like:

$ py.bench --count=3 x.py...pymod: bench_add.pyBenchmarkadd    50000000        0.020 µs/opBenchmarkadd    50000000        0.020 µs/opBenchmarkadd    50000000        0.020 µs/op

Packagegolang.testing provides corresponding runtime bits, e.g. testing.B.

py.bench produces output inGo benchmark format, and so benchmark resultscan be analyzed and compared with standard Go tools, for example withbenchstat.Additionally packagegolang.x.perf.benchlib can be used to load and processsuch benchmarking data in Python.

GPython mimics and supports most of Python command-line options, like gpython-c <commands> to run Python statements from command line, or gpython -m<module> to execute a module. Such options have the same meaning as instandard Python and are not documented here.

GPython-specific options and environment variables are listed below:

-X gpython.runtime=(gevent|threads)

Specify which runtime GPython should use. gevent provides lightweightcoroutines, while with threads go spawns full OS thread. gevent isdefault. The runtime to use can be also specified via $GPYTHON_RUNTIMEenvironment variable.