- Notifications
You must be signed in to change notification settings - Fork0
Working through "Modern Systems Programming with Scala Native" by Richard Whaling, in Scala 3
License
spamegg1/modern-systems-scala-native
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Update (July 2024)
The book is getting really hard to understand.I'm on Chapter 7 and I don't really know what's going on.Promises, timer handles,libcurl...The book doesn't really explain what we're doing and why.It's probably written for much more experienced peoplewho strongly internalized low-level stuff.Anyway, please keep on reading. Thanks for dropping by!
Updating the code inModern Systems Programming with Scala Native to
- Scala 3.5.0+,
- Scala Native version 0.5.0+,
scala-cliversion 1.4.0+, and- not using the Docker container provided by thebook's website.
So I...
- changed the syntax to Scala 3 syntax:
- removed all the optional braces, added the fewer braces syntax,
- replaced
if (...) {...} else {...}withif ... then ... else ...everywhere, using Python-style indentation, - removed the
returnkeyword, replacedNonLocalReturnsusage with the newutil.boundaryandboundary.break, - changed all the
snake_casenames tocamelCase, - got rid of unnecessary
mainobject wrappings and used@mainannotations instead, - changed Bash scripts to Scala-cli scripts,
- and so on.
We are usingScala-cli,soSBT(or Mill, or any other build tool) is not needed.
For Scala Native, you'll need the requirements such as Clang / LLVM stuffas listed onScala Native page.
You can compile and run@main methods in VS Code with Metals by clicking the run button above them:
There are 35+@main methods in the project. To compile a specific one to a binary, you can use inside the root directory, for example:
scala-cli package. --main-class ch08.simplePipe.runThis will place the binary executable in the project root directory:
Wrote /home/spam/Projects/modern-systems-scala-native/ch08.simplePipe.run, run it with ./ch08.simplePipe.run
Here the class is the import path to the method:ch08 andsimplePipe are package names, andrun is the name of the@main method:
packagech08packagesimplePipe// ...@maindefrun:Unit=???// so this is ch08.simplePipe.run
If in doubt, you can use the--interactive mode, which lets you pick the@main method you want:
$ scala-cli package. --interactiveFound several main classes. Which would you like to run?[0] ch01.helloWorld.run[1] ch06.asyncTimer.run[2] ch09.jsonSimple.run[3] ch05.httpServer.run[4] ch08.fileOutputPipe.run[5] ch01.helloNative.run[6] ch06.asyncHttp.run[7] ch09.lmdbSimple.run[8] ch08.fileInputPipe.run[9] ch01.testNullTermination.run[10] ch01.cStringExperiment1.run[11] ch01.sscanfIntExample.run[12] ch01.testingBadStuff.run[13] ch08.filePipeOut.run[14] ch02.aggregateAndCount.run[15] ch01.goodSscanfStringParse.run[16] ch01.badSscanfStringParse.run[17] ch04.badExec.run[18] ch07.simpleAsync.run[19] ch06.asyncTcp.run[20] ch03.httpClient.run[21] ch08.simplePipe.run[22] ch01.maxNgramFast.run[23] ch07.curlAsync.run[24] ch02.sortByCount.run[25] ch08.filePipe.run[26] ch03.tcpClient.run[27] ch01.maxNgramNaive.run[28] ch04.nativePipeTwo.run[29] ch01.moreTesting.run[30] ch01.cStringExperiment2.run[31] ch04.nativeFork.run[32] ch04.nativePipe.run[33] ch10.libUvService.run[34] ch01.bug.run[35] ch07.timerAsync.run21[info] Linking (multithreadingEnabled=true, disableif not used) (2353 ms)[info] Discovered 1119 classes and 7040 methods after classloading[info] Checking intermediate code (quick) (76 ms)[info] Discovered 1050 classes and 5504 methods after optimization[info] Optimizing (debug mode) (2199 ms)[info] Produced 9 LLVM IR files[info] Generating intermediate code (1689 ms)[info] Compiling to native code (3083 ms)[info] Linking with [pthread, dl, uv][info] Linking native code (immix gc, none lto) (239 ms)[info] Postprocessing (0 ms)[info] Total (9395 ms)Wrote /home/spam/Projects/modern-systems-scala-native/ch08.simplePipe.run, run it with ./ch08.simplePipe.run
The book uses@link and= extern constructs of Scala Native to link with libraries such aslibuv,libcurl andliblmdb. For example:
@link("lmdb")@externobjectLmdbImpl:defmdb_env_create(env:Ptr[Env]):Int= externdefmdb_env_open(env:Env,path:CString,flags:Int,mode:Int):Int= extern
On Ubuntu I had to install these (I thinklibcurl might have been pre-installed already?):
sudo apt install clang libuv1-dev libcurl4-gnutls-dev liblmdb-dev libhttp-parser-dev
The author did all of this work. But if we wanted to do this on our own,it would be difficult to get right the type signatures of the functions.Scala Native main contributor's advice is to directly takethe header file of such a library,and usesn-bindgen to generate the bindings:
I haven't tried that myself, but that's the way to go.
Thehttp-parse library of Chapter 10 is no longer maintained.It was ported tollhttp.It is possible to install this on Ubuntu with
sudo apt install node-llhttp
But I don't know how to link it with Scala Native.It's written in Typescript, which generates C output.The output then has to be compiled, and then linked to SN.
I modified theinstall_gatling.sh script from the book, now it's a Scala-cliscriptscripts/installGatling.sc with Gatling bundle version 3.10.5+.
From the root directory, run
./scripts/installGatling.sc
This will download into a foldergatling in the root directory,and copy the simulation file from chapter 5 into the relevant subdirectory.
You need to compile and run the HTTP server from chapter 5. (Also on chapter 7.)Read the compilation message for the name of the binary executable:
scala-cli package. --main-class ch05.httpServer.run...Wrote /home/spam/Projects/modern-systems-scala-native/project, run it with ./projectThen run that to start the server. This starts the server listening on port 8080.
I wrote another scriptscripts/runGatling.sc that sets up the needed environment variablesthen handles the interactive simulation for you by providing necessary inputs.This will compile the simulation file undergatling/user-files/simulations/.These files have to be written in Scala 2.13 unfortunately!Gatling cannot handle Scala 3. So... run the simulation with:
./scripts/runGatling.sc
Here's what the Terminal output looks like:
$ ./scripts/runGatling.scFinished setting up environment variablesfor Gatling simulation.Now running the Gatling binary:GATLING_HOME isset to /home/spam/Projects/modern-systems-scala-native/gatlingDo you want to run the simulation locally, on Gatling Enterprise, or just package it?Type the number corresponding to your choice and press enter[0]<Quit>[1] Run the Simulation locally[2] Package and upload the Simulation to Gatling Enterprise Cloud, and run it there[3] Package the Simulationfor Gatling Enterprise[4] Showhelp andexit>>>>> Choosing option [1] to run locally!Gatling 3.11.1 is available! (you're using 3.10.5)ch05.loadSimulation.GenericSimulation is the only simulation, executing it.Select run description (optional)>>>>> Providing optional name: testSimSimulation ch05.loadSimulation.GenericSimulation started...================================================================================2024-04-28 18:21:04 GMT 2s elapsed---- Requests ------------------------------------------------------------------> Global (OK=5000 KO=0 )> Web Server (OK=5000 KO=0 )---- Test scenario -------------------------------------------------------------[##########################################################################]100% waiting: 0 / active: 0 / done: 100================================================================================Simulation ch05.loadSimulation.GenericSimulation completed in 2 secondsParsing log file(s)...Parsing log file(s) done in 0s.Generating reports...================================================================================---- Global Information --------------------------------------------------------> request count 5000 (OK=5000 KO=0 )> min response time 5 (OK=5 KO=- )> max response time 116 (OK=116 KO=- )> mean response time 38 (OK=38 KO=- )> std deviation 15 (OK=15 KO=- )> response time 50th percentile 35 (OK=35 KO=- )> response time 75th percentile 50 (OK=50 KO=- )> response time 95th percentile 64 (OK=64 KO=- )> response time 99th percentile 72 (OK=72 KO=- )> mean requests/sec 2500 (OK=2500 KO=- )---- Response Time Distribution ------------------------------------------------> t < 800 ms 5000 (100%)> 800 ms <= t < 1200 ms 0 ( 0%)> t >= 1200 ms 0 ( 0%)> failed 0 ( 0%)================================================================================Reports generated, please open the following file: file:///home/spam/Projects/modern-systems-scala-native/gatling/results/genericsimulation-20240428182101491/index.html
The graphical results are ingatling/results/.../index.html.With 1000 users and 50000 requests, I got 1% failure rate(connection timeouts), and 300ms average response time.Quite amazing!
If I use the async server usinglibuv and the event loop in chapter 7,then again with 1000 users and 50000 requests,I get 100% success with 231ms mean response time! Great!
I noticed many things have changed.
There are lines of code in the zip file provided onthe book's website. Some of these are also printed in the book!
For example, in Chapter 4'snativeFork there is
for (j<- (0 to count)) {}
which does nothing. There is also
valpid= unistd.getpid()
which is never used. There are also illegal things like:
valp=SyncPipe(0)valp=FilePipe(c"./data.txt")
There are lots of other examples. There are also many unused / unnecessary imports in the files. Whenever I ran into these, I removed them.
There is also a lot of code duplication, I suppose, to make each individual file "runnable" by itself. I removed redundant code by adding package declarations, then importing the duplicated code from other files instead.
For example, Chapter 4'sbadExec.scala duplicates a lot of code fromnativeFork.scala. I solved it by separating duplicate code into a file, and adding package declarations:
// this is common.scalapackagech04// ...
// this is nativeFork.scalapackagech04packagenativeFork// ...// then use code from common.scala here
// this is badExec.scalapackagech04packagebadExec// ...// then use code from common.scala here
There is a lot of this duplication in later chapters. I fixed them.
The book usesInts for a lot of calculations such as string length, how much memory should be allocated, etc. But the current version of Scala Native is usingCSize for these now. So theInts have to be converted.CSize / USize are actuallyULong, so we need.toCSize, or.toUSize, or.toULong conversion. For this, we need to import:
importscalanative.unsigned.UnsignedRichLong
This also works:
importscalanative.unsigned.UnsignedRichInt
Moreover, we are nowable to use direct comparison betweenCSize /USize types andInt. For example:
// here strlen returns CSize, normally we would have to do 5.toULongif string.strlen(myCString)!=5then???
There are many function calls in the book that only take type arguments and no value arguments, such asstackalloc[Int] etc. This is because there is a default argumentn with value1 if none is provided, and in Scala 2 we can drop empty parentheses:stackalloc[Int] instead ofstackalloc[Int]().
In Scala 3, we need to provide the empty parentheses for the default parameter of1, or just provide1 as an argument:
stackalloc[Int]// does not work in Scala 3stackalloc[Int]()// this defaults to n = 1stackalloc[Int](1)// same as previous
The book uses things like
valserver_sockaddr= server_address.cast[Ptr[sockaddr]]
.cast is no longer available; we use.asInstanceOf[...] instead.
Function pointer classes now have different syntax. The book overrides classes likeCFuncPtr2 by providing a customapply method like so:
valby_count=newCFuncPtr2[Ptr[Byte],Ptr[Byte],Int] {defapply(p1:Ptr[Byte], p2:Ptr[Byte]):Int= {valngram_ptr_1= p1.asInstanceOf[Ptr[NGramData]]valngram_ptr_2= p2.asInstanceOf[Ptr[NGramData]]valcount_1= ngram_ptr_1._2valcount_2= ngram_ptr_2._2return count_2- count_1 }}
We can no longer do this, as these classes are declaredfinal. We must use the companion object'sfromScalaFunction[...] method instead (which is nicer, since we don't have to remember that we have to implementdef apply):
valbyCount=CFuncPtr2.fromScalaFunction[Ptr[Byte],Ptr[Byte],Int]: (p1:Ptr[Byte],p2:Ptr[Byte])=>valngramPtr1= p1.asInstanceOf[Ptr[NGramData]]valngramPtr2= p2.asInstanceOf[Ptr[NGramData]] ngramPtr2._2- ngramPtr1._2
The book and the code have some inconsistencies.There are sometimes two different names for the same thing,and the types are also different: For example:
// these are supposed to be the same thing.typeTimer=Ptr[Ptr[Byte]]// book, ch06typeTimerHandle=Ptr[Byte]// book, later in the same chaptertypeTimerHandle=Ptr[Byte]// code, in ch06typeTimerHandle=Ptr[Ptr[Byte]]// code, in other chapters
The book clearly says, in a "warning box":
There are many more issues. For example, given:
typeTCPHandle=Ptr[Ptr[Byte]]// book and codetypeClientState=CStruct3[Ptr[Byte],CSize,CSize]
but then:
valcloseCB=CFuncPtr1.fromScalaFunction[TCPHandle,Unit]: (client:TCPHandle)=>// ...valclientStatePtr= (!client).asInstanceOf[Ptr[ClientState]]
Sinceclient isTCPHandle = Ptr[Ptr[Byte]],!client isPtr[Byte].So we are casting aPtr[Byte] into aPtr[CStruct3[Ptr[Byte], CSize, CSize]]!Does this implyByte = CStruct3[Ptr[Byte], CSize, CSize]?No, it does not work that way I think... 😕
There are many more instances of this. For example, given
typeTCPHandle=Ptr[Ptr[Byte]]typeShutdownReq=Ptr[Ptr[Byte]]
we have:
defshutdown(client:TCPHandle):Unit=valshutdownReq= malloc(uv_req_size(UV_SHUTDOWN_REQ_T)).asInstanceOf[ShutdownReq]!shutdownReq= client.asInstanceOf[Ptr[Byte]]
Again, here!shutdownReq is aPtr[Byte], butclient is aPtr[Ptr[Byte]].So we are trying to squeeze aPtr[Ptr[Byte]] into aPtr[Byte]!We do this by pretending that the nested pointer does not exist withasInstanceOf[].OK fine, we can trick the compiler this way, but can we later actually use the innernested pointer ofclient correctly?Because later these are passed to actuallibuv functions...
😕 😕 😕
Big brain moment: basically, pretty muchanything can be cast toPtr[Byte]...Since "everything is a byte", the "beginning of a block of anything" is aPtr[Byte]!
🧠 🧠 🧠 🎉 🎊 🥳
Not sure how to handle this, it will be guesswork.If compilation fails during linking phase then I'll know the types are wrong.But if linking does not fail, then I'll have to figure it out from the execution.
The book uses the usual C idiom ofallocating memory that is 1 more than the length of a string, copying it, then manually null-terminating the new copy:
valstring_ptr= toCString(arg)// prepare pointer for mallocvalstring_len= string.strlen(string_ptr)// calculate length of string to be copiedvaldest_str= stdlib.malloc(string_len+1).asInstanceOf[Ptr[Byte]]// alloc 1 morestring.strncpy(dest_str, string_ptr, arg.size+1)// copydest_str(string_len)=0// manually null-terminate the new copy
If you do this you'll get errors: first is theCSize errors:
arg.size+1
when you are trying to add 1, which isInt, tostring_len, which isCSize, for which you have to use.toUSize.
The second isnone of the overloaded alternatives for method update of Ptr[Byte]... which complains when we are trying to manually null-terminate the new copy of the string:
dest_str(string_len)=0
It has to beByte instead.
Fixing all these problems and rewriting in Scala 3 style, we get:
valstringPtr= toCString(arg)// prepare pointer for mallocvalstrLen= string.strlen(stringPtr)// calculate length of string to be copiedvaldestStr= stdlib.malloc(strLen+1.toUSize)// alloc 1 morestring.strncpy(destStr, stringPtr, strLen)// copy JUST the string, not \0destStr(strLen)=0.toByte// manually null-terminate the new copy
or we can simply copy the string, including the null-terminator:
valstringPtr= toCString(arg)// prepare pointer for mallocvalstrLen= string.strlen(stringPtr)// calculate length of string to be copiedvaldestStr= stdlib.malloc(strLen+1.toUSize)// alloc 1 morestring.strncpy(destStr, stringPtr, strLen+1.toUSize)// copy, including \0
If we for some reason don't truststrncpy and want extra super-duper safety, we can do both:
valstringPtr= toCString(arg)// prepare pointer for mallocvalstrLen= string.strlen(stringPtr)// calculate length of string to be copiedvaldestStr= stdlib.malloc(strLen+1.toUSize)// alloc 1 morestring.strncpy(destStr, stringPtr, strLen+1)// copy, including \0destStr(strLen)=0.toByte// null-terminate the new copy, JUST IN CASE!
Now it's null terminated twice: once with the copying, then again manually.
The book uses the old-school C-style "argv" approach to command-line arguments from Scala 2:
objectMain {defmain(args:Array[String]):Unit= {??? }}
This does not work with Scala 3@main annotations, as it will complain aboutno given instance of type scala.util.CommandLineParser.fromString[Array[String]]... Things have changed in Scala 3 when it comes to main methods, command line arguments and code-running. They have been greatly simplified, the main method no longer has to be named "main", and now there is greater capability to use any user-defined type for the command-line arguments, but the compiler has to be "taught" how to do it.
We could do that by providing the given instance... but instead we fall back on the "arbitrary number of parameters of the same type" approach (and rename the method while we're at it):
@maindefnativePipeTwo(args:String*):Unit=???
In Scala 3.4.1+, Native 0.5.0+, thebad_sscanf_string_parse example given in the book does not cause a segfault like it does in the book. Or rather, we have to use avery long string to get a segfault, like > 100 characters. If we use the author's version (Scala 2.11, Native 0.4.0, and some old SBT version) then it works; we get a segfault immediately with as few as 8 characters every time. It won't segfault even withstackalloc[CString](1).
So I'm gonna drop down into C to see some reliable, reproducible segfault examples.
Well... that produced the same result, only for large string inputs (around 30 characters but not reliably).
./segfaultddddddddddddddddddddddddddscan results: dddddddddddddddddddddddddddddddddddddddddddddddddddscan results: dddddddddddddddddddddddddddddddddddddddddddddddddddddddmalloc(): corrupted top sizeAborted (core dumped)About
Working through "Modern Systems Programming with Scala Native" by Richard Whaling, in Scala 3
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Uh oh!
There was an error while loading.Please reload this page.