Repository files navigation

Note: If you are looking for stable releases, checkout master.

Make surecmake is available on your system and run following commands:

$ mkdir build$cd build$ cmake -DPACKAGE_TUTORIALS=ON ..$ make

After the building process finish, you should find the tutorial executables underbuild/tutorials/ directory.

Follow instructions in theREADME.md in each tutorial directories to learn how to useuTensor.

Here are the links to the tutorials:

1. Error Handling with uTensor
2. Custom Operator

uTensor is an extremely light-weight machine learning inference framework built on Tensorflow and optimized for Arm targets. It consists of a runtime library and an offline tool that handles most of the model translation work. This repo holds the core runtime and some example implementations of operators, memory managers/schedulers, and more, and the size of the core runtime is only ~2KB!

A model is constructed and trained in Tensorflow. uTensor takes the model and produces a .cpp and .hpp file. These files contains the generated C++11 code needed for inferencing. Working with uTensor on the embedded side is as easy as copy-and-paste.

Check out the detailed description here

The rearchitecture is fundamentally centered around a few key ideas, and the structure of the code base and build tools naturally followed.Old key points:

• Tensors describe how data is accessed and where from
  • Performance of ops depends on which tensors are used
• Operators are Tensor agnostic
  • High performance ops can fetch blocks of data at once
• Strive for low total power in execution
• Low static and dynamic footprint, be small
  • Low cost per Tensor throughout the entire system, since most generated models have 100+ including intermediates, also impacts dynamic footprint
  • Lightweight class hierarchy
  • Duh

New additional key ideas:

• System safety
  • All tensor metadata and actual data are owned in dedicated regions
    • This can either be user provided, or one we create
  • We can guarantee that runtime will use no more than N bytes of RAM at code gen time or at compile time!
  • Generally should not collide with userspace or system space memory, i.e. dont share heaps
  • Generally implications: a safe runtime means we can safely update models remotely
  • As many compile time errors as possible!
    • Mismatched inputs, outputs, or numbers
    • wrong sizes used
    • Impossible memory accesses
    • etc.
• Clear, Concise, and Debuggable
  • Previous iteration of uTensor relied almost too heavily on codegen, making changes to a model for any reason was near impossible
  • A developer should be able to make changes to the model without relying on code gen
  • A developer should be able to look at a model file and immediately understand what the graph looks like, without a massive amound of jumping around
  • Default tensor interface should behave like a higher level language, but exploit the speed of C++
    • Generally: No more pointer bullshit! C is super error prone, fight me
      • Only specialized operators have access to raw data blocks, and these ops will be wicked fast
  • Extensible, configurable, and optimize-outable error handling
  • GDB debugging IS NOW TRIVIAL

As mentioned before, these key ideas need to be reflected not only in the code, but in the code structure in such a way that it is Maintainable, Hackable, and User-extensible. Pretty much everything in the uTensor runtime can be divided into two components: core, and everything else. The core library contains all the deep low level functionality needed for the runtime to make the above guarantees, as well as the interfaces required for concrete implementation. Furthermore, the overhead of this core engine should be negligible relative to the system operation. Everything not in the core library really should just be thought of a reasonable defaults. For example, tensor implementations, default operators, example memory allocators, or even possible logging systems and error handlers. These modules should be the primary area for future optimization, especially before model deployment.

usingnamespaceuTensor;constuint8_t s_a[4] = {1,2,3,4};constuint8_t s_b[4] = {5,6,7,8};constuint8_t s_c_ref[4] = {19,22,43,50};// These can also be embedded in models// Recommend, not putting these on the heap or stack directly as they can be largelocalCircularArenaAllocator<256> meta_allocator;// All tensor metadata gets stored here automatically, even when new is calledlocalCircularArenaAllocator<256> ram_allocator;// All temporary storage gets allocated herevoidfoo() {// Tell the uTensor context which allocators to useContext::get_default_context()->set_metadata_allocator(&meta_allocator);Context::get_default_context()->set_ram_data_allocator(&ram_allocator);// Tensors are simply handles for accessing data as necessary, they are no larger than a pointer// RomTensor(TensorShape, data_type, data*);  Tensor a =new/*const*/RomTensor({2,2},u8, s_a);  Tensor b =new/*const*/RomTensor({2,2},u8, s_b);  Tensor c_ref =newRomTensor({2,2},u8, s_c_ref);// RamTensors are held internally and can be moved or cleared depending on the memory schedule (optional)  Tensor c =newRamTensor({2,2},u8);// Operators take in a fixed size map of (input_name -> parameter), this gives compile time errors on input mismatching// Also, the name binding + lack of parameter ordering makes ctag jumping and GDB sessions significantly more intuitive  MatrixMultOperator<uint8_t> mult_AB;  mult_AB      .set_inputs({{MatrixMultOperator<uint8_t>::a, a}, {MatrixMultOperator<uint8_t>::b, b}})      .set_outputs({{MatrixMultOperator<uint8_t>::c, c}})      .eval();// Compare results  TensorShape& c_shape = c->get_shape();for (int i =0; i < c_shape[0]; i++) {for (int j =0; j < c_shape[1]; j++) {// Just need to cast the access to the expected typeif(static_cast<uint8_t>(c(i, j)) !=static_cast<uint8_t>(c_ref(i, j)) ) {printf("Oh crap!\n");exit(-1);      }    }  }}

git clone git@github.com:uTensor/uTensor.gitcd uTensor/git checkout proposal/rearchgit submodule initgit submodule updatemkdir buildcd build/cmake -DPACKAGE_TESTS=ON -DCMAKE_BUILD_TYPE=Debug ..makemake test

The uTensor core library is configured as a mbed library out of the box, so we just need to import it into our project and build as normal.

mbed new my_projectcd my_projectmbed import https://github.com/uTensor/uTensor.git# Create main file# Run uTensor-cli workflow and copy model directory herembed compile # as normal

TODONote: CMake Support for Arm is currently experimentalhttps://stackoverflow.com/questions/46916611/cross-compiling-googletest-for-arm64

Default build

mkdir build && cd buildcmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=../extern/CMSIS_5/CMSIS/DSP/gcc.cmake  ..

With CMSIS optimized kernels

mkdir build && cd buildcmake -DARM_PROJECT=1 -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=../extern/CMSIS_5/CMSIS/DSP/gcc.cmake  ..

• Why Edge Computing
• Why the Future of Machine Learning is Tiny
• TensorFlow
• Mbed
• Node-Viewer
• How to Quantize Neural Networks with TensorFlow
• mxnet Handwritten Digit Recognition