Native code interoperability¶
Scala Native provides an interop layer that makes it easy to interactwith foreign native code. This includes C and other languages that canexpose APIs via C ABI (e.g. C++, D, Rust etc.)
All of the interop APIs discussed here are defined inscala.scalanative.unsafe package. For brevity, we’re going to referto that namespace as justunsafe.
Extern objects¶
Extern objects are simple wrapper objects that demarcate scopes wheremethods are treated as their native C ABI-friendly counterparts. Theyare roughly analogous to header files with top-level functiondeclarations in C.
For example, to call C’smalloc one might declare it as following:
importscala.scalanative.unsafe._@externobjectlibc{defmalloc(size:CSize):Ptr[Byte]=extern}
extern on the right hand side of the method definition signifies thatthe body of the method is defined elsewhere in a native library that isavailable on the library path (seeLinking with native libraries). The signature of theexternal function must match the signature of the original C function(seeFinding the right signature).
Finding the right signature¶
To find a correct signature for a given C function one must provide anequivalent Scala type for each of the arguments:
| C Type | Scala Type |
|---|---|
void | Unit |
bool | unsafe.CBool |
char | unsafe.CChar |
signed char | unsafe.CSignedChar |
unsigned char | unsafe.CUnsignedChar[^1] |
short | unsafe.CShort |
unsigned short | unsafe.CUnsignedShort[^2] |
int | unsafe.CInt |
long int | unsafe.CLongInt |
unsigned int | unsafe.CUnsignedInt[^3] |
unsigned long int | unsafe.CUnsignedLongInt[^4] |
long | unsafe.CLong |
unsigned long | unsafe.CUnsignedLong[^5] |
long long | unsafe.CLongLong |
unsigned long long | unsafe.CUnsignedLongLong[^6] |
size_t | unsafe.CSize |
ssize_t | unsafe.CSSize |
ptrdiff_t | unsafe.CPtrDiff[^7] |
wchar_t | unsafe.CWideChar |
char16_t | unsafe.CChar16 |
char32_t | unsafe.CChar32 |
float | unsafe.CFloat |
double | unsafe.CDouble |
void* | unsafe.CVoidPtr[^8] |
int* | unsafe.Ptr[unsafe.CInt][^9] |
char* | unsafe.CString[^10][^11] |
int (*)(int) | unsafe.CFuncPtr1[unsafe.CInt, unsafe.CInt][^12][^13] |
struct { int x, y; }* | unsafe.Ptr[unsafe.CStruct2[unsafe.CInt, unsafe.CInt]][^14][^15] |
struct { int x, y; } | Not supported |
Linking with native libraries¶
C compilers typically require to pass an additional-lmylib flag todynamically link with a library. In Scala Native, one can annotatelibraries to link with using the@link annotation.
importscala.scalanative.unsafe._@link("mylib")@externobjectmylib{deff():Unit=extern}
Whenever any of the members ofmylib object are reachable, the ScalaNative linker will automatically link with the corresponding nativelibrary.
As in C, library names are specified without thelib prefix. Forexample, the librarylibuv correspondsto@link("uv") in Scala Native.
It is possible to rename functions using the@name annotation. Its useis recommended to enforce the Scala naming conventions in bindings:
importscala.scalanative.unsafe._@link("uv")@externobjectuv{@name("uv_uptime")defuptime(result:Ptr[CDouble]):Int=extern}
If a library has multiple components, you could split the bindings intoseparate objects as it is permitted to use the same@link annotationmore than once.
Variadic functions¶
Scala Native supports native interoperability with C’s variadicargument list type (i.e.va_list), and partially for... varargs.For examplevprintf andprintf defined in C as:
intvprintf(constchar*format,va_listarg);intprintf(constchar*format,...);
can be declared in Scala as:
importscala.scalanative.unsafe._@externobjectmystdio{defvprintf(format:CString,args:CVarArgList):CInt=externdefprintf(format:CString,args:Any*):CInt=extern}
The limitation of... interop requires that itsarguments needs to passed directly to variadic arguments function orarguments need to be inlined. This is required to obtain enoughinformation on how arguments show be passed in regards to C ABI. Passinga sequence to extern method variadic arguments is not allowed and wouldresult in compilation failure.
Forva_list interop, one can wrap a function in a nicer API like:
importscala.scalanative.unsafe._defmyprintf(format:CString,args:CVarArg*):CInt=Zone{mystdio.vprintf(format,toCVarArgList(args.toSeq))}
SeeMemory management for a guide of usingunsafe.Zone And then call it just like a regular Scala function:
myprintf(c"2 + 3 = %d, 4 + 5 = %d",2+3,4+5)printf(c"2 + 3 = %d, 4 + 5 = %d",2+3,4+5)
Exported methods¶
When linking Scala Native as library, you can mark functions that shouldvisible in created library with@exported(name:String) annotation. Incase if you omit or use null as the argument for name extern functionname match the name of the method. Currently, only static object methodscan be exported. To export accessors of field or variable in staticobject use@exportAccessors(getterName:String,setterName:String).If you omit the explicit names in the annotation constructor, ScalaNative would create exported methods withset_ andget_ prefixes andname of field.
intScalaNativeInit(void); function is special exportedfunction that needs to be called before invoking any code defined inScala Native. It returns0 on successful initializationand a non-zero value otherwise.
For dynamic libraries a constructorwould be generated to invokeScalaNativeInit functionautomatically upon loading library or startup of the program.
If for some reason you need to disable automatic initialization of Scala Nativeupon loading dynamic library and invoke it manually in user code setSCALANATIVE_NO_DYLIB_CTOR environment variable. You canalso disable generation of library constructors by defining-DSCALANATIVE_NO_DYLIB_CTOR inNativeConfig::compileOptions of your build.
importscala.scalanative.unsafe._objectmyLib{@exportAccessors("mylib_current_count","mylib_set_counter")varcounter:Int=0@exportAccessors("error_message")valErrorMessage:CString=c"Something bad just happend!"@exporteddefaddLongs(l:Long,r:Long):Long=l+r@exported("mylib_addInts")defaddInts(l:Int,r:Int):Int=l+r}
// libmylib.hintScalaNativeInit(void);longaddLongs(long,long);intmylib_addInts(int,int);intmylib_current_count();voidmylib_set_counter(int);// test.c#include"libmylib.h"#include<assert.h>#include<stdio.h>intmain(intargc,char**argv){// This function needs to be called before invoking any methods defined in Scala Native.// Might be called automatically unless SCALANATIVE_NO_DYLIB_CTOR env variable is set.assert(ScalaNativeInit()==0);addLongs(0L,4L);mylib_addInts(4,0);printf("Current count %d\n",mylib_current_count());mylib_set_counter(42);// ...}
Pointer types¶
Scala Native provides a built-in equivalent of C’s pointers viaunsafe.Ptr[T] data type. Under the hood pointers are implemented usingunmanaged machine pointers.
Operations on pointers are closely related to their C counterparts andare compiled into equivalent machine code:
| Operation | C syntax | Scala Syntax |
|---|---|---|
| Load value | *ptr | !ptr |
| Store value | *ptr = value | !ptr = value |
| Pointer to index | ptr + i,&ptr[i] | ptr + i |
| Elements between | ptr1 - ptr2 | ptr1 - ptr2 |
| Load at index | ptr[i] | ptr(i) |
| Store at index | ptr[i] = value | ptr(i) = value |
| Pointer to field | &ptr->name | ptr.atN |
| Load a field | ptr->name | ptr._N |
| Store a field | ptr->name = value | ptr._N = value |
WhereN is the index of the fieldname in the struct. SeeMemorylayout types for details.
Function pointers¶
It is possible to use external functions that take function pointers.For example given the following signature in C:
voidtest(void(*f)(char*));
One can declare it as follows in Scala Native:
deftest(f:unsafe.CFuncPtr1[CString,Unit]):Unit=unsafe.extern
CFuncPtrN types are final classes containing pointer tounderlying C function pointer. They automatically handle boxing callarguments and unboxing result. You can create them from C pointer usingCFuncPtr helper methods:
deffnDef(str:CString):CInt=???valanyPtr:CVoidPtr=CFuncPtr.toPtr{CFuncPtr1.fromScalaFunction(fnDef)}typeStringLengthFn=CFuncPtr1[CString,CInt]valfunc:StringLengthFn=CFuncPtr.fromPtr[StringLengthFn](anyPtr)func(c"hello")
It’s also possible to createCFuncPtrN from ScalaFunctionN. You can do this by using implicit methodconversion method from the corresponding companion object.
importscalanative.unsafe.CFuncPtr0defmyFunc():Unit=println("hi there!")valmyFuncPtr:CFuncPtr0[Unit]=CFuncPtr0.fromScalaFunction(myFunc)valmyImplFn:CFuncPtr0[Unit]=myFunc_valmyLambdaFuncPtr:CFuncPtr0[Unit]=()=>println("hello!")
On Scala 2.12 or newer, the Scala language automatically converts fromclosures to SAM types:
valmyfuncptr:unsafe.CFuncPtr0[Unit]=()=>println("hi there!")
Memory management¶
Unlike standard Scala objects that are managed automatically by theunderlying runtime system, one has to be extra careful when working withunmanaged memory.
Zone allocation. (since 0.3)
Zones (also known as memory regions/contexts) are a technique forsemi-automatic memory management. Using them one can bindallocations to a temporary scope in the program and the zoneallocator will automatically clean them up for you as soon asexecution goes out of it:
importscala.scalanative.unsafe._// For Scala 3Zone{valbuffer=alloc[Byte](n)}// For Scala 2, works, but is not idiomatic on Scala 3Zone.acquire{implicitz=>valbuffer=alloc[Byte](n)}
allocrequests memory sufficient to containnvaluesof a given type. If number of elements is not specified, it defaultsto a single element. Memory is zeroed out by default.Zone allocation is the preferred way to allocate temporary unmanagedmemory. It’s idiomatic to use implicit zone parameters to abstractover code that has to zone allocate.
One typical example of this are C strings that are created fromScala strings using
unsafe.toCString. The conversion takesimplicit zone parameter and allocates the result in that zone.When using zone allocated memory one has to be careful not tocapture this memory beyond the lifetime of the zone. Dereferencingzone-allocated memory after the end of the zone is undefinedbehavior.
Stack allocation.
Scala Native provides a built-in way to perform stack allocationsof using
unsafe.stackallocfunction:valbuffer=unsafe.stackalloc[Byte](256)
This code will allocate 256 bytes that are going to be availableuntil the enclosing method returns. Number of elements to beallocated is optional and defaults to 1 otherwise. Memoryiszeroed out by default.
When using stack allocated memory one has to be careful not tocapture this memory beyond the lifetime of the method.Dereferencing stack allocated memory after the method’s executionhas completed is undefined behavior.
Manual heap allocation.
Scala Native’s library contains a bindings for a subset of thestandard libc functionality. This includes the trio of
malloc,reallocandfreefunctions that are defined inlibc.stdlibextern object.Calling those will let you allocate memory using system’sstandard dynamic memory allocator. Every single manual allocationmust also be freed manually as soon as it’s not needed anylonger.
Apart from the standard system allocator one might also bind toplethora of 3-rd party allocators such asjemalloc to serve the same purpose.
Undefined behavior¶
Similarly to their C counter-parts, behavior of operations that accessmemory is subject to undefined behaviour for following conditions:
Dereferencing null.
Out-of-bounds memory access.
Use-after-free.
Use-after-return.
Double-free, invalid free.
Memory layout types¶
Memory layout types are auxiliary types that let one specify memorylayout of unmanaged memory. They are meant to be used purely incombination with native pointers and do not have a correspondingfirst-class values backing them.
unsafe.Ptr[unsafe.CStructN[T1,...,TN]]Pointer to a C struct with up to 22 fields. Type parameters are thetypes of corresponding fields. One may access fields of the structusing
_Nhelper methods on a pointer value:valptr=unsafe.stackalloc[unsafe.CStruct2[Int,Int]]()ptr._1=10ptr._2=20println(s"first${ptr._1}, second${ptr._2}")
Here
_Nis an accessor for the field number N.unsafe.Ptr[unsafe.CArray[T,N]]Pointer to a C array with statically-known length
N. Length isencoded as a type-level natural number. Natural numbers are typesthat are composed of base naturalsNat._0,...Nat._9and anadditionalNat.DigitNconstructors, whereNrefers to number ofdigits in the given number. So for example number1024is going tobe encoded as following:import scalanative.unsafe._, Nat._type _1024 = Digit4[_1, _0, _2, _4]
Once you have a natural for the length, it can be used as an arraylength:
val arrptr = unsafe.stackalloc[CArray[Byte, _1024]]()
You can find an address of n-th array element via
arrptr.at(n).
Byte strings¶
Scala Native supports byte strings viac"..." string interpolator thatgets compiled down to pointers to statically-allocated zero-terminatedstrings (similarly to C):
importscalanative.unsafe._importscalanative.libc._// CString is an alias for Ptr[CChar]valmsg:CString=c"Hello, world!"stdio.printf(msg)
It does not allow any octal values or escape characters not supported byScala compiler, like\a or\?, but also Unicode escapes. It ispossible to use C-style hex values up to value 0xFF, e.g.c"Hello\x61\x62\x63"
Additionally, we also expose two helper functionsunsafe.fromCStringandunsafe.toCString to convert between C-styleCString(sequence of Bytes, usually interpreted as UTF-8 or ASCII) andJava-styleString (sequence of 2-byte Chars usuallyinterpreted as UTF-16).
It’s worth to remember thatunsafe.toCString andc"..." interpreter cannot be used interchangeably as they handle literals differently.Helper methodsunsafe.fromCString andunsafe.toCString are charset aware.They will always assumeString is UTF-16, and take aCharset parameter to know what encoding to assume for the byte string (CString) - if not present it is UTF-8.
If passed a null as an argument, they will return a null of the appropriatetype instead of throwing a NullPointerException.
Platform-specific types¶
Scala Native defines the typeSize and its unsigned counterpart,USize.A size corresponds toInt on 32-bit architectures and toLong on 64-bitones.
Size and alignment of types¶
In order to statically determine the size of a type, you can use thesizeoffunction which is Scala Native’s counterpart of the eponymous C operator. Itreturns the size in bytes:
println(unsafe.sizeof[Byte])// 1println(unsafe.sizeof[CBool])// 1println(unsafe.sizeof[CShort])// 2println(unsafe.sizeof[CInt])// 4println(unsafe.sizeof[CLong])// 8
It can also be used to obtain the size of a structure:
typeTwoBytes=unsafe.CStruct2[Byte,Byte]println(unsafe.sizeof[TwoBytes])// 2
Additionally, you can also usealignmentof to find the alignment of a given type:
println(unsafe.alignmentof[Int])// 4println(unsafe.alignmentof[unsafe.CStruct2[Byte,Long]])// 8
Unsigned integer types¶
Scala Native provides support for four unsigned integer types:
unsigned.UByteunsigned.UShortunsigned.UIntunsigned.ULongunsigned.USize
They share the same primitive operations as signed integer types.Primitive operation between two integer values are supported onlyif they have the same signedness (they must both signed or both unsigned.)
Conversions between signed and unsigned integers must be done explicitlyusingbyteValue.toUByte,shortValue.toUShort,intValue.toUInt,longValue.toULong,sizeValue.toUSizeand converselyunsignedByteValue.toByte,unsignedShortValue.toShort,unsignedIntValue.toInt,unsignedLongValue.toLong,unsignedSizeValue.toSize.
Universal equality is supported between signed and unsigned integers, for example-1.toUByte==255 or65535==-1.toUShort would yieldtrue,However, similar to signed integers on JVM, class equality between different (boxed) integer types is not supported.Usage of-1.toUByte.equals(255) would returnfalse, as we’re comparing different boxed types (scala.scalanative.unsigned.UByte withjava.lang.Integer)
Continue tonative.
