- Notifications
You must be signed in to change notification settings - Fork121
gopy generates a CPython extension module from a go package.
License
go-python/gopy
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
gopy generates (and compiles) aCPython extension module from ago package.
This is an improved version that works with current versions of Go (e.g., 1.15 -- should work with any future version going forward), and uses unique int64 handles to interface with python, so that no pointers are interchanged, making everything safe for the more recent moving garbage collector.
It also supports python modules having any number of Go packages, and generates a separate .py module file for each package, which link into a single common binding library. It has been tested extensively on reproducing complex Go code in large libraries -- most stuff "just works". For example, theGoGi GUI library is fully usable from python now (domake; make install in the python directory there, and try theexamples/widgets/widgets.py demo).
New features:
- Callback methods from Go into Python now work: you can pass a python function to a Go function that has a function argument, and it will call the python function appropriately.
- The first embedded struct field (i.e., Go's version of type inheritance) is used to establish a corresponding class inheritance in the Python
classwrappers, which then efficiently inherit all the methods, properties, etc.
Gopy now assumes that you are working with modules-based builds, and requires a validgo.mod file, and works only with Go versions 1.15 and above.
Currently usingpybindgen to generate the low-level c-to-python bindings, but support forcffi should be relatively straightforward for those using PyPy instead of CPython (pybindgen should be significantly faster for CPython apparently). You also needgoimports to ensure the correct imports are included.
$ python3 -m pip install pybindgen$ go install golang.org/x/tools/cmd/goimports@latest$ go install github.com/go-python/gopy@latest
(This all assumes you have already installedGo itself, and added~/go/bin to yourPATH).
Toinstall python modules, you will need the python install packages:
python3 -m pip install --upgrade setuptools wheel
IMPORTANT: many errors will be avoided by specifying the-vm option to gopy, with a full path if needed, or typically just-vm=python3 to use python3 instead of version 2, which is often the default for the plainpython command.
On linux, you may need to ensure that the linkerld will look in the current directory for library files -- add this to your.bashrc file (andsource that file after editing, or enter command locally):
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
As of version 0.4.0, windows is now better supported, and is passing tests (on at least one developers machine). You may still need to set some environment variables depending on your python installation, but a vanilla standard install is working.
Install Python from the main Python distribution:https://www.python.org/downloads/windows/ --do not install from the Microsoft Store app! -- while that is very convenient, it creates symbolic links to access the python executables, which is incompatible with go exec.Command to run it, despite too many hours of trying to get around that.
The standard python install does not create apython3.exe which gopy looks for -- follow instructions here:https://stackoverflow.com/questions/39910730/python3-is-not-recognized-as-an-internal-or-external-command-operable-program/41492852(just make a copy of python.exe to python3.exe in the relevant installed location).
If you get a bunch of errors during linking in the build process, setLIBDIR orGOPY_LIBDIR to path to python libraries, andLIBRARY orGOPY_PYLIB to name of python library (e.g., python39 for 3.9).
See theCONTRIBUTING guide for pointers on how to contribute togopy.
A presentation was given atdotgo-2015.A longer version of that talk is also availablehere.An article was also posted on theGopherAcademy Advent-2015.
Documentation is available ongodoc:https://godoc.org/github.com/go-python/gopy
Thepkg andexe commands are for end-users and create a full standalone python package that can be installed locally usingmake install based on the auto-generatedMakefile. Theoretically these packages could be uploaded tohttps://pypi.org/ for wider distribution, but that would require a lot more work to handle all the different possible python versions and coordination with the Go source version, so it is easier to just do the local make install on your system. Thegen andbuild commands are used for testing and just generate / build the raw binding files only.
IMPORTANT: many errors will be avoided by specifying the-vm option to gopy, with a full path if needed, or typically just-vm=python3 to use python3 instead of version 2, which is often the default for the plainpython command.
Here are some (slightly enhanced) docs from the help command:
$ gopyhelpgopy - Commands: pkg generate and compile Python bindingsfor Go, automatically including subdirs also creates all the python files needed to install module exe like pkg but makes a standalone executable with Go packages bultin this is particularly useful when using -main arg to start process on gen generate (C)Python language bindingsfor Go build generate and compile main thread -- python interpreter can run on another thread.Use"gopy help <command>"for more information about a command.$ gopyhelp pkgUsage: gopy pkg<go-package-name> [other-go-package...]pkg generates and compiles (C)Python language bindingsfora Go package, including subdirectories, and generates python module packaging suitable for distribution. if setup.py file does not yet existin the target directory,then it is created along with other default packaging files, using arguments. Typically you create initial default versions of these files andthen edit them, and after that, only regenerate the Go binding files.ex: $ gopy pkg [options]<go-package-name> [other-go-package...] $ gopy pkg github.com/go-python/gopy/_examples/hiOptions: -author="gopy": author name -desc="": short description of project (long comes from README.md) -email="gopy@example.com": author email -exclude="": comma-separated list of package names to exclude -main="": code string to runin the GoGoPyInit()functionin the cgo library -name="": name of output package (otherwise name of first package is used) -output="": output directoryfor root of package -symbols=true: include symbolsin output -url="https://github.com/go-python/gopy": home pagefor project -user="": username on https://www.pypa.io/en/latest/for package name suffix -version="0.1.0": semantic version number -- can use e.g., git to get this from tag and pass as argument -vm="python": path to python interpreter -dynamic-link=false: whether to link output shared library dynamically to Python -build-tags="": build tags to be passed to`go build`$ gopyhelp exeUsage: gopy exe<go-package-name> [other-go-package...]exe generates and compiles (C)Python language bindingsfora Go package, including subdirectories, and generates a standalone python executable and associated module packaging suitable for distribution. if setup.py file does not yet existin the target directory,then it along with other default packaging files are created, using arguments. Typically you create initial default versions of these files andthen edit them, and after that, only regenerate the Go binding files.The primary needfor an exe instead of a pkg dynamic library is when the main thread must be usedfor something other than running the python interpreter, such asfor a GUI library where the main thread must be usedfor running the GUI event loop (e.g., GoGi).ex: $ gopy exe [options]<go-package-name> [other-go-package...] $ gopy exe github.com/go-python/gopy/_examples/hiOptions: -author="gopy": author name -desc="": short description of project (long comes from README.md) -email="gopy@example.com": author email -exclude="": comma-separated list of package names to exclude -main="": code string to runin the Gomain()functionin the cgo library -- defaults toGoPyMainRun() but typically should be overriden -name="": name of output package (otherwise name of first package is used) -output="": output directoryfor root of package -symbols=true: include symbolsin output -url="https://github.com/go-python/gopy": home pagefor project -user="": username on https://www.pypa.io/en/latest/for package name suffix -version="0.1.0": semantic version number -- can use e.g., git to get this from tag and pass as argument -vm="python": path to python interpreter -dynamic-link=false: whether to link output shared library dynamically to Python -build-tags="": build tags to be passed to`go build`$ gopyhelp genUsage: gopy gen<go-package-name> [other-go-package...]gen generates (C)Python language bindingsfor Go package(s).ex: $ gopy gen [options]<go-package-name> [other-go-package...] $ gopy gen github.com/go-python/gopy/_examples/hiOptions: -build-tags="": build tags to be passed to`go build` -dynamic-link=false: whether to link output shared library dynamically to Python -main="": code string to runin the gomain()functionin the cgo library -name="": name of output package (otherwise name of first package is used) -no-make=false:do not generate a Makefile, e.g., when called from Makefile -no-warn=false: suppress warning messages, which may be expected -output="": output directoryfor bindings -package-prefix=".": custom package prefix used when generating import statementsfor generated package -rename=false: rename Go symbols to python PEP snake_case -vm="python": path to python interpreter$ gopyhelp buildUsage: gopy build<go-package-name> [other-go-package...]build generates and compiles (C)Python language bindingsfor Go package(s).ex: $ gopy build [options]<go-package-name> [other-go-package...] $ gopy build github.com/go-python/gopy/_examples/hiOptions: -build-tags="": build tags to be passed to`go build` -dynamic-link=false: whether to link output shared library dynamically to Python -main="": code string to runin the gomain()functionin the cgo library -name="": name of output package (otherwise name of first package is used) -no-make=false:do not generate a Makefile, e.g., when called from Makefile -no-warn=false: suppress warning messages, which may be expected -output="": output directoryfor bindings -package-prefix=".": custom package prefix used when generating import statementsfor generated package -rename=false: rename Go symbols to python PEP snake_case -symbols=true: include symbolsin output -vm="python": path to python interpreter
Note: you now need to make a go.mod file if you don't already have one in your environment, and get the package before building:
$ go mod init dummy.com/dum$ go get github.com/go-python/gopy/_examples/hi$ gopy build -output=out -vm=python3 github.com/go-python/gopy/_examples/hi$ ls outMakefile __init__.py __pycache__/ _hi.so* build.py go.py hi.c hi.go hi.py hi_go.h hi_go.so$cd out$ python3>>> from out import hi>>> dir(hi)['Add','Concat','Hello','Hi','NewPerson','Person','__doc__','__file__','__name__','__package__']>>> hi.Hello("you")hello you from go
You can also run:
gotest -v -run=TestHi...NOTE: following not yet working in new version:
gopy comes with a littlepython module allowing to wrap and compilegopackages directly from thepython interactive shell:
>>>importgopy>>>hi=gopy.load("github.com/go-python/gopy/_examples/hi")gopy>inferringpackagename...gopy>loading'github.com/go-python/gopy/_examples/hi'...gopy>importing'github.com/go-python/gopy/_examples/hi'>>>printhi<module'github.com/go-python/gopy/_examples/hi'from'/some/path/.../hi.so'>>>>printhi.__doc__packagehiexposesafewGofunctionstobewrappedandusedfromPython.
$ cd github.com/go-python/gopy/_examples/hi$ docker run --rm -v `pwd`:/go/src/in -v `pwd`:/out gopy/gopy app bind -output=/out in$ file hi.sohi.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not strippedThe docker image can also be built on local machine:
$ cd $GOPATH/src/github.com/go-python/gopy$ docker build -t go-python/gopy .$ docker run -it --rm go-python/gopyTo know what features are supported on what backends, please refer to theSupport matrix.
Here's an example for how to usesetup.py with gopy to make an installable package:https://github.com/raptor-ml/raptor/blob/master/labsdk/setup.py
Also, see this for cross-platform build:https://github.com/raptor-ml/raptor/blob/master/.github/workflows/labsdk-release.yaml
Many errors will be avoided by specifying the-vm option to gopy, with a full path if needed, or typically just-vm=python3 to use python3 instead of version 2, which is often the default for the plainpython command.
If you get any kind of error about the library module not being able to be imported, or apparently a large number of other random-looking errors, a mismatch between the python version used for compiling vs. what you are using to run is the most likely explanation.
If you get an error like this after importing a generated module:
Fatal Python error:_PyInterpreterState_Get(): no current thread stateit means you are running a different version of python than the one that build the library you are importing -- make sure you've got the paths in your-vm arg aligned with what you are using to import.
If yourimport statement fails to find the module.so file, and it is in the current directory, you may need to ensure that the linkerld will look in the current directory for library files -- add this to your.bashrc file (andsource that file after editing, or enter command locally):
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
you may also need to add your python path, but that is less likely to be an issue.
gopy is part of thego-python organization and licensed underBSD-3.When you want to contribute a patch or some code togopy, please send a pullrequest against thegopy issue trackerAND a pull request againstgo-python/license adding yourself to theAUTHORS andCONTRIBUTORS files.
About
gopy generates a CPython extension module from a go package.
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.