Repository files navigation

Chipmunk is a framework for testing persistent-memory (PM) file systems for crash consistency bugs. Using Chipmunk, we found 23 new bugs in 5 PM file systems. This repository includes the Chipmunk test harness, the Syzkaller fuzzer, and the ACE systematic workload generator.

Please cite the following paper if you use Chipmunk:

@inproceedings {chipmunk-eurosys23,    author = {Hayley LeBlanc and Shankara Pailoor and Om Saran and Isil Dillig and James Bornholt and Vijay Chidambaram}    title = "{Chipmunk: Investigating Crash-Consistency in Persistent-Memory File Systems}",    booktitle = {EuroSys '23: Eighteenth European Conference on Computer Systems},    year = {2023},    month = may, }

• Ubuntu 20.04
• At least 8GB of RAM

Chipmunk doesnot require real PM hardware.

• chipmunk/ includes all testing infrastructure.
  • chipmunk/executor/ace/ includes our modified version of the ACE workload generator. See the setup instructions for information on how to run ACE.
  • chipmunk/executor/harness/ includes most of Chipmunk's test harness code.
    • chipmunk/executor/harness/test_harness.cpp compiles to the executable that runs ACE tests.
  • chipmunk/executor/tests/ contains ACE-generated workloads.
  • chipmunk/executor/executor_cc.cc compiles to the executable that runs Syzkaller-generated tests.
  • chipmunk/loggers/ includes system-specific logger modules.
• scripts/ includes setup scripts.
• vmshare/ includes files that are shared with the VM running the tests.
  • vmshare/linux-5.1 includes the source code for the Linux kernel, including file system code.
  • vmshare/crashConsistencyProgs is used to store Syzkaller-generated workloads.
  • vmshare/logs is used to store logs and bug reports from failing tests.

1. Runscripts/dependencies.sh to install the dependencies for Chipmunk, Syzkaller, and ACE.
2. Install Go:https://go.dev/doc/install
3. Runscripts/build_kernel.sh <cores> to automatically configure and compile the kernel.
4. cd tochipmunk/ and runmake to build the testing infrastructure. It make take several minutes to build.

Troubleshooting

• If you encounter this error:error: '-mindirect-branch' and '-fcf-protection' are not compatible, install gcc-8 and runsudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 1.
• Compiling Chipmunk involves building some kernel modules against the kernel you will run them on. The kernel must be compiled before Chipmunk can be compiled.

1. Runscripts/create-image.sh script. This creates an image called stretch.img in the top-level directory and a key pair in$HOME/.ssh/.
2. Boot the VM usingscripts/boot-vm.sh. The VM can be accessed directly in this terminal, but to avoid visual glitches, we recommend sshing into the VM usingssh root@localhost -p 2222 -i ~/.ssh/stretch.id_rsa.
3. The scripts should automatically share thevmshare/ directory with the VM. To confirm that this and the PM emulation worked, try loading the NOVA file system module withinsmod tmpdir/linux-5.1/fs/nova/nova.ko and mount it at/mnt/pmem withmount -t NOVA -o init /dev/pmem0 /mnt/pmem. This should succeed, and runningdf should show that/dev/pmem0 is mounted at/mnt/pmem. The VM can now be shut down.

1. Inchipmunk/, create a copy ofconfig.template namedconfig and edit it so that:
   • "workdir" is set to the absolute path tochipmunk/syzkaller/workdir
   • "kernel_obj" is set to the absolute path tochipmunk/vmshare/linux-5.1
   • "image" is set to the absolute path tostretch.img
   • "sshkey" is set to the absolute path to thestretch.id_rsa file created by thecreate-image.sh script. It should have been placed in$HOME/.ssh/ by this script.
   • "syzkaller" is set to the absolute path tochipmunk/syzkaller
   • "tester_dir" is set to the absolute path tochipmunk/
   • "filesystem" and "logger" reflect the file system you would like to test
   • in the VM options, "kernel" is set to the absolute path to the bzImage for the kernel and "share_dir" is set to the absolute path tochipmunk/vmshare. You can also update the number of VMs for Syzkaller to spawn and the amount of memory and number of CPUs to allocate for each VM here.
2. Runsudo ./bin/syz-manager -config config. To run in debug mode with extra output, add-debug. This generates a significant amount of output, so you should redirect the output to files.
3. As Chipmunk runs, it will create a set of directories with names likevmshare-# (one for each VM) and store output from that VM's fuzzing instance there. It also stores some data inchipmunk/syzkaller/workdir.
4. While the fuzzer runs, a dashboard listing bug reports and statistics about the run is accessible via your web browser at the IP address of the host.

1. Runcd chipmunk; cp -r bin/* ../vmshare/syzkallerBinaries to copy the binaries for the Chipmunk files into the shared directory.
2. Boot the VM and copy the contents of/root/tmpdir/syzkallerBinaries/ into another directory/root/syzkallerBinaries - this is necessary because some of the memory management that code from Syzkaller does will not work if it is run from within the shared directory.
3. cd to/root/syzkallerBinaries/linux_amd64 and run./syz-execprog -crashConsistency -mountCov <program name>. The program name should just be the hexadecimal name of a file produced by the fuzzer and placed in thevmshare-#/crashConsistencyProgs and syz-execprog will automatically look in/root/tmpdir/crashConsistencyProgs for it. You may need to move the program from an instance-specific vmshare directory to the main one.
   • By default, execprog tests NOVA. To test a different file system, add the arguments-fs=<fs name> -fs_path=<absolute path to FS .ko file on the VM> -logger=<absolute path to logger .ko file on the VM>. If the FS under test is built into the kernel, pass the empty string "" to-fs_path.
   • syz-execprog mocks some of the fuzzing infrastructure that the executor expects, but otherwise runs the exact same code as the fuzzer when testing the file system.

1. cd tochipmunk/executor/ace. Runpython3 ace.py -l <seq length> -n false -d false -t <fs type> to generate tests.
   • The sequence length option can be 1, 2, or 3.
   • The test type options supported by Chipmunk arepm andcrashmonkey. The default option iscrashmonkey; these tests are placed inchipmunk/executor/tests/dax_seq# and should be used only with EXT4-DAX or XFS-DAX. To test NOVA, PMFS, WineFS, or other PM-specific file systems, use thepm option. These tests are placed inchipmunk/executor/tests/seq#.
2. Move back to the top-level directory and compile tests by runningmake <test dir>_tests where<test dir> is the location of the test you'd like to build.
   • For example, compiling seq2 tests generated usingpm mode would bemake seq2_tests. Seq1pm tests are compiled along with Chipmunk and don't need to be compiled with a specific command.
3. Runcp -r bin/* chipmunk/vmshare/syzkallerBinaries to copy the testing infrastructure and compiled tests to a directory that is shared with the VM.
4. Boot the VM andcd totmpdir/syzkallerBinaries.
5. To run an individual ACE test, use the command./ace-executor_cc -v -f <fs type> tests/seq1/j-lang1.so, replacingseq1/j-lang1.so with the relative path to the test you want to run. If the FS type is not provided, the tester defaults to NOVA.

Instructions to reproduce bugs found by Chipmunk can be found inbugs.md.

This version of Chipmunk does not currently support testing SplitFS; we are working on merging this support in.