Repository files navigation

Spiral is a Python module that provides several different functions for splitting identifiers found in source code files. The nameSpiral is a loose acronym based on "SPlitters for IdentifieRs: A Library".

Authors:Michael Hucka
Repository:https://github.com/casics/spiral
License: Unless otherwise noted, this content is licensed under theGPLv3 license.

April 2018: Version 1.1.0 fixesa bug that prevented importing Spiral, and another bug that causesetup.py to fail to install dependencies automatically. Additional enhancements include improved command-line help and internal code refactoring.

• Introduction
• Please cite the paper
• Installation instructions
• Basic operation
• Performance of Ronin
• Other splitters in Spiral
• Limitations
• More information
• Getting help and support
• Contributing — info for developers
• Acknowledgments

Spiral is a Python 3 package that implements numerous identifier splitting algorithms.Identifier splitting (also known asidentifier name tokenization) is the task of breaking apart program identifier strings such asgetInt orreadUTF8stream into component tokens: [get,int] and [read,utf8,stream]. The need for splitting identifiers arises in a variety of contexts, including natural language processing (NLP) methods applied to source code analysis and program comprehension. Spiral provides some basic naive splitting algorithms, such as a straightforward camel-case splitter, as well as more elaborate heuristic splitters, such as a new algorithm calledRonin.

Article citations arecritical for academic developers. If you use Spiral and you publish papers about your software, we ask that youplease cite the Spiral paper:

Hucka, M. (2018). Spiral: splitters for identifiers in source code files.Journal of Open Source Software, 3(24), 653,https://doi.org/10.21105/joss.00653

Please also indicate the specific version of Spiral you use, to improve other people's ability to reproduce your results. You can use the Zenodo DOIs we provide for this purpose:

• Spiral release 1.0.1 ⇒10.5281/zenodo.1211835

For basic usage, the following is probably the simplest and most direct way to install Spiral on your computer:

sudo pip3 install git+https://github.com/casics/spiral.git

Alternatively, you can clone this GitHub repository and then runsetup.py:

git clone https://github.com/casics/spiral.gitcd spiralsudo python3 -m pip install.

The above should be all you need to run Spiral out of the box. If you plan on experimenting with alternative parameter values or alternative dictionaries, you will additionally need to install the following:

• Platypus-Opt, an optimization library (if you want to optimize parameters).Important:This Platypus is not the same as theother package called "Platypus" in PyPI. Make sure to getPlatypus-Opt or install from thePlatypus repo.

• Data modules fromNLTK, particularlynltk_words andntlk_wordnet from thenltk.corpus module and thenltk.stem module. These cannot be installed automatically bysetup.py—please refer to theNLTK installation instructions for instructions on how to do it.

Spiral is extremely easy to use. To use a Spiral splitter in a Python program, simply import a splitter and then call it.

fromspiral.simple_splittersimportpure_camelcase_splitprint(pure_camelcase_split('TestString'))

Some splitters take optional parameters, and the more complex splitters have aninit() function that you can optionally call to set additional parameters or load data. Currently, only Ronin and Samurai have initialization functions, and callinginit() is optional—if you do not call it, Ronin and Samurai will call theirinit() functions automatically.

Here are some examples of using the Ronin splitter algorithm. The following input,

fromspiralimportroninforsin ['mStartCData','nonnegativedecimaltype','getUtf8Octets','GPSmodule','savefileas','nbrOfbugs']:print(ronin.split(s))

produces the following output:

['m','Start','C','Data']['nonnegative','decimal','type']['get','Utf8','Octets']['GPS','module']['save','file','as']['nbr','Of','bugs']

Spiral also includes a command-line program namedspiral in thebin subdirectory; it will split strings provided on the command line or in a file. (Note: Ronin and Samurai load internal data files when they start up. In normal use, called from an application program, the resulting startup delay will only happen once. In the command-line program, the data is reloaded at every invocation, causing a startup delay every time. The delay is not typical for normal Spiral usage.)

By the way, if your goal is to split identifiers obtained during source code mining and you need to filter out gibberish strings of characters before attempting to split them, you may findNostril (the Nonsense String Evaluator) useful.

Splitting identifiers is deceptively difficult. It is a research-grade problem for which no perfect solution exists. Even in cases where the input consists of identifiers that strictly follow conventions such as camel case, ambiguities can arise. For example, correctly splittingJ2SEProjectTypeProfiler intoJ2SE,Project,Type,Profiler requires the reader to recognizeJ2SE as a unit. The task of splitting identifiers is made more difficult when there are no case transitions or other obvious boundaries in an identifier. And then there is the small problem that humans are often simply inconsistent!

Ronin is an advanced splitter that uses a variety of heuristic rules, English dictionaries, and tables of token frequencies obtained from mining source code repositories. Ronin includes a default table of term frequencies derived from an analysis of over 46,000 randomly selected software projects in GitHub that contained at least one Python source code file. The tokens were extracted using software from Spiral's parent project,CASICS (specifically, theextractor package), and the frequency table constructed using a procedure encoded in the small programcreate_frequency_fileincluded with Spiral. Ronin also has a number of parameters that need to be tuned; this can be done using the optimization programoptimize-roninin the dev/optimization subdirectory. The default parameter values were derived by optimizing performance against two data sets available from other research groups:

1. TheLoyola University of Delaware Identifier Splitting Oracle (Ludiso)
2. The INTT data set, extracted from thezip archive of INTT

Spiral includes copies of these data sets in thetests/data subdirectory. The parameters derived primarily by running against the INTT database of 18,772 identifiers and their splits. The following table summarizes the results:

Many of the "failures" against these sets of identifiers are actually not failures, but cases where Ronin gets a more correct answer or where there is a legitimate difference in interpretation. Here are some examples:

Names likefread could be considered appropriate to leave alone, but thef infread actually stands for "file", and thus IMHO it makes sense to split it—splitting identifiers into meaningful tokens is the purpose of Ronin, after all. Conversely, tokens such asUtf8 should be left as units because they are meaningful. Differences involving some other terms such ascolor space are due to Ronin splitting terms that are typically considered separate words but are treated as one word in Ludiso, or vice versa. (The main entry in Wikipedia for "color space" is two words, while for "filename" it is one word.) This sometimes also arises because Ronin recognizes prefixes and sometimes does not split words because they would not be in normal written English, such asnonblocking instead ofnonblocking. Still other differences between Ronin's output and the expected splits given in Ludiso and INTT are due to inconsistencies in the Ludiso and INTT data sets. For example, sometimes a token within a larger identifier is split in one case but not in another. Finally, some other differences are cases where the Ludiso split seems to favor program-specific names. For example,QWheelEvent is split as [ QWheel,Event ] whereas Ronin will split it as [ Q,Wheel,Event ].

So, Ronin's performance may be better than the pure numbers imply. However, without checking every Ludiso case and manually reinterpreting the splits, we can't say for sure. All that aside, Ronin definitely gets many cases wrong.

Here is a list of all the splitters implemented in Spiral at this time:

The following table illustrates the differences between the simpler splitters.

Ronin and Samurai are more advanced than any of the simple splitters above. Ronin in particular does everythingheuristic_splitter does, but handles far more difficult cases.

Ronin is the most advanced splitter in the Spiral package, but it is not perfect by any means. Certain limitations are known:

• Ronin is tuned for splitting program identifiers, which isnot the same as splitting strings of concatenated words. With its default parameter values, it is not good at splitting strings likedriveourtrucks which are not composed of tokens commonly encountered in source code contexts. (By the way, reducing the value of parameterlow_freq_cutoff inronin.init() to something like10 makes Ronin more likely to split concatenated word strings likedriveourtrucks orsocietynamebank. However, this comes at the expense of worsening scores on the INTT and Ludiso data sets.)

• Ronin was trained on source code repositories containing Python code. The default frequency table may not be optimal for splitting things that come from non-Python source files, although the test results on Ludiso and INTT show that it is pretty good on non-Python content.

• Spiral includes a set of predefined special terms in the fileconstants.py so thatheuristic_split and the Ronin splitter can recognize certain character sequences that should be treated as units. Examples includeutf8,ipv4,checkbox,J2SE and others. The use of specialized dictionaries like this is an approach taken in other splitters (including Simon Butler'sINTT) and is the only known way to prevent incorrect splits of special terms, but it is also an obvious limitation: the list is surely incomplete (and can neverbe complete because new terms are invented all the time), and may even cause incorrect splits in some contexts.

• The current algorithm implemented in Ronin arose organically after a lot of trial-and-error experimentation, and as a consequence, is rather gnarly and difficult to explain. The implementation could be made more efficient and the algorithm clarified by a nice about of refactoring.

The name "Ronin" is a play on the use of the name "Samurai" by Enslen et al. (2009) for their identifier splitting algorithm. The core loop of Ronin is based on Samurai, but it would be inappropriate to call this implementation Samurai too. In an effort to imply the lineage of this modified algorithm, I chose "Ronin" (a name referring to a drifter samurai without a master, during the Japanese feudal period).

I implemented Samurai based on the description of the algorithm published in the following paper, then modified it repeatedly in an attempt to improve performance. Ronin is the result. A goal for Ronin was to produce a splitter that had good performance even without using a local frequency table.

The Ludiso data set is described in the following paper:

The INTT splitter and data set are described in the following paper:

If you find an issue, please submit it inthe GitHub issue tracker for this repository.

A lot remains to be done on CASICS in many areas. We would be happy to receive your help and participation if you are interested. Please feel free to contact the developers either via GitHub or the mailing listcasics-team@googlegroups.com.

Everyone is asked to read and respect thecode of conduct when participating in this project.

This material is based upon work supported by theNational Science Foundation under Grant Number 1533792 (Principal Investigator: Michael Hucka). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The installation of Spiral includes a dictionary containing words from theNTLK project, specifically thewords andwordnet dictionaries. Theword dictionary is public domain, but thewordnet dictionary is copyright 2006 by Princeton University and made available underlicense terms that permit free redistribution. For more information, please see Princeton University"About WordNet" (Princeton University, 2010).

The vector artwork of the segmented spiral illusion at the top of this page was obtained fromwww.123freevectors.com.