Repository files navigation

This package contains a bridging library for multiple FST libraries and toolkitsand set of tools for processing of finite-state automata especially forlinguistic systems. HFST library and tools are licensed under GNU GPL licenceversion 3, you may read the full licence in the file named COPYING. Theauthors specified in AUTHORS file may be contacted about licensing issues.

For additional information, see the project pagehttps://hfst.github.io.

Installation method depends on operating system and the version you want toinstall. For stable versions there exists packages for some of the betteroperating system and package manager combinations. For bleeding edge newestversions, development and non-supported operating systems and versions youwill have to performinstallation from the sources.

Debian packages for HFST are distributed viaApertium project. This foldercontains debian packages for HFST API library, command line tools and Python bindings.The debian packages are experimental; the requirements of debian or ubuntuinstallations are same as main packages. SFST is excluded from the packagesas it has portability issues with hash_maps and hash_sets.For installation instructions, seedownloads.

Binaries for Windows are distributed viaApertium project. This folder containsready-compiled HFST library and command line tools.For installation instructions, seedownloads.Python bindings for Windows are currently available as wheels for 32-bitPython versions 2.7, 3.4, 3.5 and 3.6 on ourPyPI page.Performinginstallation from the sources is also possible on Windows with MinGWand Cygwin, if Python bindings are not needed.

Binaries for OS X are distributed viaApertium project. This folder containsready-compiled HFST library and command line tools.For installation instructions, seedownloads.Python bindings for OS X are currently available as wheels forPython versions 2.7, 3.4, 3.5 and 3.6 on ourPyPI page.Performinginstallation from the sources is also possible on Mac.

HFST software is available inscience overlay. For portage, use:

layman -a science

To add the repository. Then:

emerge -av hfst

to install. Parts of the spell-checking tools are inFinnish overlay. If youuse Paludis, just try tocave resolve sci-misc/hfst to get the currentinstructions.

For rest of the systems, HFST needs to be installed from the source, thetraditional GNU/linux way. To begin your installation, you need to start bygathering the dependencies, as adviced in the following chapter.

Please note that the dependencies for the library can be set during the compiletime. Settings that determine dependencies relate to features the resultinglibrary will have; e.g. if youdisable openfst you cannot use weightedfinite-state automata. If youenable foma, you will be able to read fomaformat automata as HFST files and use foma's algorithms to process automata.The command./configure --help lists all features that can be controlledwith configure switches and whether they are enabled or disabled by default.

• OpenFST version 1.7.9 or older (newer does not work) (default)
  • to disable OpenFST support, configure switch--without-openfst maybe used (however, this seriously limits the use of HFST)
  • if you have problems linking, make sure openfst is compiled with c++17compilers
• To use theSFST backend (default):
  • the SFST library version 1.4.6g is bundled with HFST and includedby default when building HFST
  • SFST requires readline and ncurses
  • The SFSTfrontend a.k.a. the SFST-PL parser a.k.a.hfst-sfstpl2fstdoesnot require the SFST library to be installed; the library is onlyused for library-stuff like reading SFST format automata.
  • to disable SFST backend, configure switch--without-sfst must be used
• foma version 0.10.0 or newer (default)
  • to disable foma backend, configure switch--without-foma may be used
  • thehfst-xfst frontend does not require foma binary or foma library
• ICU version 50 or newer
• To compile corpus processing toolhfst-proc, you need to use the

configure switch--enable-proc (or--enable-all-tools)

• To use the Python interface:
  • Compiling the bindings requires swig
  • NOTE: the Python API is not under autotools, you must compile it yourself;for more information, see filepython/README

Note that if you did install dependent libraries, such as libxml or ICUto your home directory instead of using your system's package manager(or supported default location):

• If you only have a local version of a library, you can use it withappropriate LDFLAGS and CXXFLAGS, eg../configure LDFLAGS=-L/path/to/local/lib--prefix=/path/to/local/installation andmake CXXFLAGS=-I/path/to/local/headers

If you are building a development version youloaded from the version controlsystem, you must have new brand of GNU development tools installed:

• autoconf >=2.62
• automake >=1.11.1
• libtool >=2.2
• >=gettext-0.17
• GNU tool-chain is also needed with distributed packages if the user wishes to

modify Makefile.am or configure.ac files.

• Mac OS X users are advised to use MacPorts; Mac OS X 10.6 with

XCode 2.3.2 at least is not sufficient

• A package loaded from hfst web site doesnot have these requirements

The source codesloaded from the version controls system will also requireparse generator system:

• GNU flex 2.5.35 or compatible and
• GNU bison 1.31 (2.4 suggested) or compatible
• flex 2.5.4-2.5.33 will choke on perfectly valid rules used in hfst
• bison older than 1.31 do not support name-prefix needed for havingmultiple parsers in one library
• A package loaded from hfst web site doesnot have these requirements
• source code loaded from version control system requires them only tobootstrap; if you use systems with archaic versions of flex or bison andcannot install updates, you might be able to get the needed files fromsomewhere

When running HFST software or using HFST libraries from HFST-enabled software:

• If the executable is dynamically linked (almost always), the operating systemmust be able to find hfst libraries
• If you install the libraries in non-standard paths, you need to ensurethat operating system is aware of this; In linux this may happen by settingLD_LIBRARY_PATH, on MacDYLD_LIBRARY_PATH
• thehfst-xfst frontend needs GNU compatible getopt, or basic getoptswithout GNU-incompatible getopt installed
• thehfst-sfstpl2fst frontend doesnot require SFST libraries orbinaries, you donot need to enable SFST libraries (via the switch--with-sfst) in order to compile SFST-PL scripts to HFST automata.

INSTALL describes the GNU build system in detail, but for most users the usual:

./configuremake(as root) make install

should result in a local installation and:

(as root) make uninstall

in its uninstallation.

If you aren't going to be linking to the library afterbuilding it and don't need to be able to send debugging information, you cansave a considerable amount of space and memory by doing:

make install-strip

instead of make install. This strips all the symbols from the binaries,reducing sizes by a factor of 5-10.

If you would rather install in e.g. your home directory(or aren't the system administrator), you can tell ./configure:

./configure --prefix=$HOME

The HFST library may link to numerous FST handling backends with varyinglicences. If you are going to redistribute the HFST library you compiled, makesure there are no clashes in the licences of the linked libraries.

If you are checking out the development versions from GIT you must first createand install the necessary autotools files from the host system:

autoreconf -i

It is common practice to keepgenerated files out of version control.

For further installation instruction refer to fileINSTALL, which containsthe standard installation instructions for GNU autoconf based software.

HFST includes an experimental, very limited C API. The API is built into it'sown library, aptly named libhfst_c. On Linux systems, pkg-config shouldlocate the necessary parts, like the libhfst_c.so shared library file, andthe libhfst_c.h header file.

Versioning of the C API is unrelated to the versions of hfst itself, usethe version of hfst_c instead. For now, it is pre-1.0, that is, it's notstable. Changes must be expected.

Note that there are separate bindings to Python from HFST, so there is no needto go through the C API for using HFST from Python.

In this section we list the errors that pop up commonly on our support channel or inour issue tracker,

During the compilation errors of form:

/usr/include/c++/4.3/cstdlib:124: error: '::malloc' has not been declared

or:

your configure failed to find malloc, check README for further instructions

are mosts often caused by broken library installation. The simplest solutionin Linux-based platforms is ldconfig:

ldconfig -v

This is actually told and performed by the autotools libtool libraryinstallation, but it's easy to miss. It looks like this:

Libraries have been installed in:   /usr/local/libIf you ever happen to want to link against installed librariesin a given directory, LIBDIR, you must either use libtool, andspecify the full pathname of the library, or use the '-LLIBDIR'flag during linking and do at least one of the following:   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable     during execution   - add LIBDIR to the 'LD_RUN_PATH' environment variable     during linking   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag   - have your system administrator add LIBDIR to '/etc/ld.so.conf'See any operating system documentation about shared libraries formore information, such as the ld(1) and ld.so(8) manual pages.

If you installed a library on non-standard path, or installed it to thedefault/usr/local/lib, but your variant of Linux doesn't support librariesthere, you may need to set it up and/or ldconfig the directory explicitly:

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/libldconfig -v -n /usr/local/lib

If all else fails, try installing the library to wherever your blend of Linuxinstalls all its libraries, such as /usr/lib.

See also:<http://nerdland.net/unstumping-the-internet/malloc-has-not-been-declared/> forthe gory details.

After installing HFST and running programs, the installed programs shouldon most systems be able to find and use the shared libraries that just gotinstalled alongside the programs that were installed on the same go, but thisis not always the case. Typically on first installation of the HFST libraryor after a major version update of HFST library, the system may not know aboutit. To fix this, you must runldconfig on GNU systems. To ensure properlinking, useldconfig -v to get a print out of known libraries, thelisting should include libhfst.so indicating the current version.

If this is the first time you install a library on your system by hand, it mayhappen on some systems that the library configuration does not include thatdirectory at all. Notably on ubuntu it seems that /usr/local/lib is not one oflibrary directories, and that is the default target for manually installedlibraries. Either fix this by doing./configure --prefix=/usr or checkyour distributions manuals on how to set up new library directories. Sameapplies for libraries installed to e.g. home directory.

See also the previous error description.

During the compilation phase, an error message including things like:

scanner1.ll:22:27: fatal error: htwolcpre1.h: No such file or directorycompilation terminated.Makefile:1029: recipe for target `scanner1.o' failed

indicates that the header files generated by flex/yacc have the extension .hhinstead of .h. This error has been encountered at least on some versions ofCygwin. Currently, the best solution is to manually rename eachflex/yacc-generated header file of the form foo.hh as foo.h. You probablyhave to do this iteratively after each error message of this type.

During the make check phase. This indicates that a test that tests forexpected failure fails expectedly, which is also indicated by a green wordPASS orXFAIL on the next line. This is expected behaviour and not anindication of a bug. If there is a bug effecting that or other tests in thetest suite, it will be indicated by a line starting with wordFAIL orXPASSin red colour.

Same applies for other messages printed during make check phase. The cases wheresomething actually fails will be clearly stated in the end of the test suite bya message such as:

======================================2 of 36 tests failedPlease report to hfst-bugs@helsinki.fi======================================

These errors can be reported either to the stated mail address or theHFST bugtracker on Github.

During use of bash-based scripts, an error message including things like:

cat: -l: No such file or directorycat: version,quiet,format:,output:,latin1::,utf8::: No such file or directorycat: -n: No such file or directorycat: hfst-lexc: No such file or directorycat: --: No such file or directory

indicate that script is trying to use Mac OS X's getopt as if it was GNU getopt.However default getopt in Mac OS X does not work at all like GNU getopt.Easiest solution is to install working getopt, e.g. by using MacPorts:

sudo port install getopt

The newer versions of bash scripts detect Mac OS X's getopt and fallback tousing getopts. Note that getopts does not support long options and filenamesmust be last parameters on commandline with it, so its use is stronglydiscouraged.

During program execution(Mac OS X only), errors of form:

terminate called throwing an exceptionAbort trap: 6

Can be caused by, just about any exceptional situation that does not havespecific handler. On Linux it will read:

terminate called after throwing an instance of 'ImplementationTypeNotAvailableException'

And then you'll know that this specific exception is about backend that was disabled during configure phase. Or it might read:

terminate called after throwing an instance of 'UndefinedSymbolPairsFound'

And you'd know it's something with the alphabet. But OS X won't tell us this. Soit is an unexpected error situation. Usual suspects are still: missing libraryin configure, empty file, reading error, writing error...

During program execution:

syntax erroron line 1:Aborted.This is an hfst interface bug:Exception: HfstException in file: htwolcpre3-parser.yy on line: XXX

This can be caused by buffer size limit in hfst-twolc, a rule file larger than 10 megabytes (after pre-processing) will not be parsed properly.

TheHfst wiki site contains furtherdetails of the HFST system.

Bugs can be reported via email toHFST team bug mail address, or preferably toHFST's bug tracking systemat GithubWhen reporting, please include at least following:

• version of software used, if command-line tool (hfst-toolname --version)
• version of hfst-library, if possible
• steps to reproduce, attach or all related files if possible
• information about platform used (e.g. uname -a)