Repository files navigation

(C) 2016-2018 byDamir Cavar

Last edited: 2018-08-06,Damir Cavar

This code example shows how aFoma-based FST can be used to process multi-word expressions that are given in a dictionary and compiled into a Finite State Transducer.

There is a default window size specified in the code. It can be altered using command line arguments.

The maximum multi-word window size can actually be compiled into the Finite State Transducer and read out using the C-wrapper. This way one can avoid unnecessary lookups. The advantage of this method, assuming that one has a comprehensive list of multi-word expressions to compile into the transducer, is that it is very fast and that it shows internal structural and morphosyntactic properties of multi-word expressions.

The example implementation should be straight forward to understand. It expects an input in form of a file (or a stream) that contains a tokenized sentence per line. See the includedtest.txt file for an example.

To compile this example, you need to have the entire Foma collection of binaries, includes and libraries set up on your system. You will also need some C++11 compiler and various other libraries for it, for example theBoost libraries.

The project is aCMake project. Make sure that you have alsoCMake installed and set up on your system.

To create the running binary for the code inFomaMWT, in the folder run:

cmake CMakeList.txt

This will generate theMakefile and other files in the same folder. Run:

make

and it should compile correctly, if all the paths and folders are OK, and if the libraries were found.

If you want to test the speed of the processor, run the following command:

time ./mwtagger test.txt > res.txt