Repository files navigation

ulab is anumpy-like array manipulation library formicropython andCircuitPython.The module is written in C, defines compact containers (ndarrays) for numerical data of one to fourdimensions, and is fast. The library is a software-only standardmicropython user module,i.e., it has no hardware dependencies, and can be compiled for any platform. 8-, and 16-bit signedand unsigned integerdtypes, as well asfloat, and, optionally, complex are supported.Thefloat implementation ofmicropython (32-bitfloat, or 64-bitdouble) is automaticallydetected and handled.

1. Supported functions and methods
   1. ndarray methods
   2. numpy and scipy functions
   3. ulab utilities
   4. user module
2. Usage
3. Finding help
4. Benchmarks
5. Firmware
   1. Customising the firmware
   2. Platforms including ulab
   3. Compiling
      1. UNIX
      2. STM-based boards
      3. ESP32-based boards
      4. RP2-based boards
      5. Compiling for CircuitPython
6. Issues, contributing, and testing
   1. Testing

ulab implementsnumpy'sndarray with the==,!=,<,<=,>,>=,+,-,/,*,**,+=,-=,*=,/=,**= binary operators, and thelen,~,-,+,abs unary operators thatoperate element-wise. Type-awarendarrays can be initialised from anymicropython iterable, lists ofiterables via thearray constructor, or by means of thearange,concatenate,diag,eye,frombuffer,full,linspace,logspace,ones, orzeros functions.

ndarrays can be sliced, and iterated on, and have a number of their own methods, and properties, such asflatten(),itemsize,reshape(),shape,size,strides,tobytes(),tolist(), andtranspose() andT. If the firmware is compiled withcomplex support,theimag, andreal properties are automatically included.

In addition,ulab includesuniversal functions,manynumpy functions, and functions from thenumpy.fft,numpy.linalg,numpy.random,scipy.linalg,scipy.optimize,scipy.signal, andscipy.special modules. A complete list of available routines can be found undermicropython-ulab.

Theutils module contains functions forinterfacing with peripheral devices supporting the buffer protocol. These functions do not have an obviousnumpy equivalent, but share a similar programming interface, and allow direct data input-output betweennumerical arrays and hardware components.

User-defined functions operating on numerical data can easily be added via theuser module. This allows for transparent extensions, without having to change anything in the core. Hints as to how to work withndarrays at the C level can be found in theprogramming manual.

ulab sports anumpy/scipy-compatible interface, which makes porting ofCPython code straightforward. The followingsnippet should run equally well inmicropython, or on a PC.

try:fromulabimportnumpyfromulabimportscipyexceptImportError:importnumpyimportscipy.specialx=numpy.array([1,2,3])scipy.special.erf(x)

Documentation can be found onreadthedocs undermicropython-ulab,as well as atcircuitpython-ulab.A number of practical examples are listed in Jeff Epler's excellentcircuitpython-ulab overview.Thetricks chapter of the user manual discussesmethods by which RAM and speed can be leveraged in particular numerical problems.

Representative numbers on performance can be found underulab samples.

Pre-built, and up-to-date firmware files for select platforms can be downloadedfrommicropython-builder.

If flash space is a concern, unnecessary functions can be excluded from the compiled firmware withpre-processor switches. In addition,ulab also has options for trading execution speed for firmware size.A thorough discussion on how the firmware can be customised can be found in thecorresponding sectionof the user manual.

ulab is also included in the following compiledmicropython variants and derivatives:

1. CircuitPython for SAMD51 and nRF microcontrollershttps://github.com/adafruit/circuitpython
2. MicroPython for K210https://github.com/loboris/MicroPython_K210_LoBo
3. MaixPyhttps://github.com/sipeed/MaixPy
4. OpenMVhttps://github.com/openmv/openmv
5. pimoroni-picohttps://github.com/pimoroni/pimoroni-pico
6. Tulip Creative Computerhttps://github.com/shorepine/tulipcc

If you want to try the latest version ofulab onmicropython or one of its forks, the firmware can be compiledfrom the source by following these steps:

Simply clone theulab repository with

git clone https://github.com/v923z/micropython-ulab.git ulab

and then run

./build.sh [matrix.dims]# Dimensions is 2 by default

This command will clonemicropython, and build theunix port automatically, as well as run the test scripts. If you want an interactiveunix session, you can launch it in

ulab/micropython/ports/unix

First, you have to clone themicropython repository by running

git clone https://github.com/micropython/micropython.git

on the command line. This will create a new repository with the namemicropython. Staying there, clone theulab repository with

git clone https://github.com/v923z/micropython-ulab.git ulab

If you don't have the cross-compiler installed, your might want to do that now, for instance on Linux by executing

sudo apt-get install gcc-arm-none-eabi

If this step was successful, you can try to run themake command in the port's directory as

make BOARD=PYBV11 USER_C_MODULES=../../../ulab all

which will prepare the firmware for pyboard.v.11. Similarly,

make BOARD=PYBD_SF6 USER_C_MODULES=../../../ulab all

will compile for the SF6 member of the PYBD series. If your target isunix, you don't need to specify theBOARD parameter.

Provided that you managed to compile the firmware, you would upload that by running either

dfu-util --alt 0 -D firmware.dfu

or

python pydfu.py -u firmware.dfu

In case you got stuck somewhere in the process, a bit more detailed instructions can be found underhttps://github.com/micropython/micropython/wiki/Getting-Started, andhttps://github.com/micropython/micropython/wiki/Pyboard-Firmware-Update.

ulab can be tested on the ESP32 inwokwi's micropython emulator without having to compile the C code. This utility also offers the possibility to save and share yourmicropython code.

Firmware forEspressif hardware can be built in two different ways, which are discussed in the next two paragraphs. A solution for issues with the firmware size is outlined in thelast paragraph of this section.

Beginning with version 1.15,micropython switched tocmake on the ESP32 port. If your operating system supportsCMake > 3.12, you can either simply download, and run the singlebuild script, or follow the step in this section. Otherwise, you should skip to thenext one, where the old,make-based approach is discussed.

In case you encounter difficulties during the build process, you can consult the (general instructions for the ESP32)[https://github.com/micropython/micropython/tree/master/ports/esp32#micropython-port-to-the-esp32].

First, clone theulab, themicropython, as well as theespressif repositories:

export BUILD_DIR=$(pwd)git clone https://github.com/v923z/micropython-ulab.git ulabgit clone https://github.com/micropython/micropython.gitcd$BUILD_DIR/micropython/git clone -b v4.0.2 --recursive https://github.com/espressif/esp-idf.git

Also later releases ofesp-idf are possible (e.g.v4.2.1).

Then install theESP-IDF tools:

cd esp-idf./install.sh. ./export.sh

Next, build themicropython cross-compiler, and theESP sub-modules:

cd$BUILD_DIR/micropython/mpy-crossmakecd$BUILD_DIR/micropython/ports/esp32make submodules

At this point, all requirements are installed and built. We can now compile the firmware withulab. In$BUILD_DIR/micropython/ports/esp32 create amakefile with the following content:

BOARD = GENERICUSER_C_MODULES =$(BUILD_DIR)/ulab/code/micropython.cmakeinclude Makefile

You specify with theBOARD variable, what you want to compile for, a generic board, orTINYPICO (formicropython version >1.1.5, useUM_TINYPICO), etc. Still in$BUILD_DIR/micropython/ports/esp32, you can now runmake.

If your operating system does not support a recent enough version ofCMake, you have to stay withmicropython version 1.14. The firmware can be compiled either by downloading and running thebuild script, or following the steps below:

First, cloneulab with

git clone https://github.com/v923z/micropython-ulab.git ulab

and then, in the same directory,micropython

git clone https://github.com/micropython/micropython.git

At this point, you should haveulab, andmicropython side by side.

With version 1.14,micropython switched tocmake on theESP32 port, thus breaking compatibility with user modules.ulab can, however, still be compiled with version 1.14. You can check out a particular version by pinning the release tag as

cd ./micropython/git checkout tags/v1.14

Next, update the submodules,

git submodule update --initcd ./mpy-cross&& make# build cross-compiler (required)

and find the ESP commit hash

cd ./micropython/ports/esp32make ESPIDF=# will display supported ESP-IDF commit hashes# output should look like: """# ...# Supported git hash (v3.3): 9e70825d1e1cbf7988cf36981774300066580ea7# Supported git hash (v4.0) (experimental): 4c81978a3e2220674a432a588292a4c860eef27b

Choose an ESPIDF version from one of the options printed by the previous command:

ESPIDF_VER=9e70825d1e1cbf7988cf36981774300066580ea7

In themicropython directory, create a new directory with

mkdir esp32

Yourmicropython directory should now look like

lsACKNOWLEDGEMENTS    CONTRIBUTING.md  esp32     lib      mpy-cross  README.mdCODECONVENTIONS.md  docs             examples  LICENSE  ports      testsCODEOFCONDUCT.md    drivers          extmod    logo     py         tools

In./micropython/esp32, download the software development kit with

git clone https://github.com/espressif/esp-idf.git esp-idfcd ./esp-idfgit checkout$ESPIDF_VERgit submodule update --init --recursive# get idf submodulespip install -r ./requirements.txt# install python reqs

Next, still staying in./micropython/eps32/esd-idf/, install the ESP32 compiler. If using an ESP-IDF version >= 4.x (chosen by$ESPIDF_VER above), this can be done by running. $BUILD_DIR/esp-idf/install.sh. Otherwise, for version 3.x, run the following commands in in.micropython/esp32/esp-idf:

# for 64 bit linuxcurl https://dl.espressif.com/dl/xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz| tar xvz# for 32 bit# curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux32-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz# don't worry about adding to path; we'll specify that later# also, see https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started for more info

Finally, build the firmware:

cd ./micropython/ports/esp32# temporarily add esp32 compiler to pathexport PATH=../../esp32/esp-idf/xtensa-esp32-elf/bin:$PATHexport ESPIDF=../../esp32/esp-idf# req'd by Makefileexport BOARD=GENERIC# options are dirs in ./boardsexport USER_C_MODULES=../../../ulab# include ulab in firmwaremake submodules& make all

If it compiles without error, you can plug in your ESP32 via USB and then flash it with:

make erase&& make deploy

When selectingBOARD=TINYPICO, the firmware is built but fails to deploy, because it is too large for the standard partitions. We can rectify the problem by creating a new partition table. In order to do so, in$BUILD_DIR/micropython/ports/esp32/, copy the following 8 lines to a file namedpartitions_ulab.cvs:

# Notes: the offset of the partition table itself is set in# $ESPIDF/components/partition_table/Kconfig.projbuild and the# offset of the factory/ota_0 partition is set in makeimg.py# Name,   Type, SubType, Offset,  Size, Flagsnvs,      data, nvs,     0x9000,  0x6000,phy_init, data, phy,     0xf000,  0x1000,factory,  app,  factory, 0x10000, 0x200000,vfs,      data, fat,     0x220000, 0x180000,

This expands thefactory partition by 128 kB, and reduces the size ofvfs by the same amount. Having defined the new partition table, we should extendsdkconfig.board by adding the following two lines:

CONFIG_PARTITION_TABLE_CUSTOM=yCONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_ulab.csv"

This file can be found in$BUILD_DIR/micropython/ports/esp32/boards/TINYPICO/. Finally, runmake clean, andmake. The new firmware contains the modified partition table, and should fit on the microcontroller.

RP2 firmware can be compiled either by downloading and running the singlebuild script/build script for Pico W, or executing the commands below.

First, clonemicropython:

git clone https://github.com/micropython/micropython.git

Then, setup the required submodules:

cd micropythongit submodule update --init lib/tinyusbgit submodule update --init lib/pico-sdkcd lib/pico-sdkgit submodule update --init lib/tinyusb

You'll also need to compilempy-cross:

cd ../../mpy-crossmake

That's all you need to do for themicropython repository. Now, let us cloneulab (in a directory outside the micropython repository):

cd ../../git clone https://github.com/v923z/micropython-ulab ulab

With this setup, we can now build the firmware. Back in themicropython repository, use these commands:

cd ports/rp2make USER_C_MODULES=/path/to/ulab/code/micropython.cmake

Ifmicropython andulab were in the same folder on the computer, you can setUSER_C_MODULES=../../../ulab/code/micropython.cmake. The compiled firmware will be placed inmicropython/ports/rp2/build.

Adafruit Industries always include a relatively recent version ofulab in their nightly builds. However, if you really need the bleeding edge, you can easily compile the firmware from the source. Simply clonecircuitpython, and move the commit pointer to the latest version ofulab (ulab will automatically be cloned withcircuitpython):

git clone https://github.com/adafruit/circuitpython.gitcd circuitpyton/extmod/ulab# update ulab heregit checkout mastergit pull

You might have to check, whether theCIRCUITPY_ULAB variable is set to1 for the port that you want to compile for. You find this piece of information in themake fragment:

circuitpython/ports/port_of_your_choice/mpconfigport.mk

After this, you would runmake with the singleBOARD argument, e.g.:

make BOARD=mini_sam_m4

If you find a problem with the code, please, raise anissue! An issue should address a single problem, and should contain a minimal code snippet that demonstrates the difference from the expected behaviour. Reducing a problem to the bare minimum significantly increases the chances of a quick fix.

Feature requests (porting a particular function fromnumpy orscipy) should also be posted atulab issue.

Contributions of any kind are always welcome. If you feel like adding to the code, you can simply issue a pull request. If you do so, please, try to adhere tomicropython'scoding conventions.

However, you can also contribute to the documentation (preferably via thejupyter notebooks, or improve thetests.

If you decide to lend a hand with testing, here are the steps:

1. Write a test script that checks a particular function, or a set of related functions!
2. Drop this script in one of the folders inulab tests!
3. Run the./build.sh script in the root directory ofulab! This will clone the latestmicropython, compile the firmware forunix, execute all scripts in theulab/tests, and compare the results to those in the expected results files, which are also inulab/tests, and have an extension.exp. In case you have a new snippet, i.e., you have no expected results file, or if the results differ from those in the expected file, a new expected file will be generated in the root directory. You should inspect the contents of this file, and if they are satisfactory, then the file can be moved to theulab/tests folder, alongside your snippet.