Finding duplicated code with CPD

Overview

Duplicate code can be hard to find, especially in a large project.But PMD’sCopy/Paste Detector (CPD) can find it for you!

CPD works with Java, JSP, C/C++, C#, Go, Kotlin, Ruby, Swift andmany more languages.It can be used viacommand-line, or via anAnt task.It can also be run with Maven by using thecpd-check goal on theMaven PMD Plugin.

Your own language is missing?See how to add ithere.

Why should you care about duplicates?

It’s certainly important to know where to get CPD, and how to call it, but it’s worth stepping back for a moment andasking yourself why you should care about this, being the occurrence of duplicate code blocks.

Assuming duplicated blocks of code are supposed to do the same thing, any refactoring, even simple, must be duplicatedtoo – which is unrewarding grunt work, and puts pressure on the developer to find every place in which to performthe refactoring. Automated tools like CPD can help with that to some extent.

However, failure to keep the code in sync may mean automated tools will no longer recognise these blocks as duplicates.This means the task of finding duplicates to keep them in sync when doing subsequent refactorings can no longer beentrusted to an automated tool – adding more burden on the maintainer. Segments of code initially supposed to do thesame thing may grow apart undetected upon further refactoring.

Now, if the code may never change in the future, then this is not a problem.

Otherwise, the most viable solution is to not duplicate. If the duplicates are already there, then they should berefactored out. We thus advise developers to use CPD tohelp remove duplicates, not to help keep duplicates in sync.

Refactoring duplicates

Once you have located some duplicates, several refactoring strategies may apply depending on the scope and extent ofthe duplication. Here’s a quick summary:

• If the duplication is local to a method or single class:
  • Extract a local variable if the duplicated logic is not prohibitively long
  • Extract the duplicated logic into a private method
• If the duplication occurs in siblings within a class hierarchy:
  • Extract a method and pull it up in the class hierarchy, along with common fields
  • Use theTemplate Method design pattern
• If the duplication occurs consistently in unrelated hierarchies:
  • Introduce a common ancestor to those class hierarchies

Novice as much as advanced readers may want toread on on Refactoring Gurufor more in-depth strategies, use cases and explanations.

Finding more duplicates

For some languages, additional options are supported. E.g. Java supports--ignore-identifiers. This has theeffect, that all identifiers are replaced with the same placeholder value before the comparing. This helps toidentify structurally identical code that only differs in naming (different class names, different method names,different parameter names).

There are other similar options:--ignore-annotations,--ignore-literals,--ignore-literal-sequences,--ignore-sequences,--ignore-usings.

Note that these options aredisabled by default (e.g. identifiers arenot replaced with the same placeholdervalue). By default, CPD finds identical duplicates. Using these options, the found duplicates are not anymoreexactly identical.

CLI Usage

CLI options reference

Examples

Minimum required options: Just give it the minimum duplicate size and the source directory:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java

You can also specify the language:

~ $pmd cpd --minimum-tokens 100 --dir src/main/cpp --language cpp

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\cpp --language cpp

You may wish to check sources that are stored in different directories:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java --dir src/test/java

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java --dir src\test\java

There is no limit to the number of--dir, you may add.

You may wish to ignore identifiers so that more duplications are found, that only differ in naming:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java --ignore-identifiers

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java --ignore-identifiers

And if you’re checking a C source tree with duplicate files in different architecture directoriesyou can skip those using--skip-duplicate-files:

~ $pmd cpd --minimum-tokens 100 --dir src/main/cpp --language cpp --skip-duplicate-files

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\cpp --language cpp --skip-duplicate-files

You can also specify the encoding to use when parsing files:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java --encoding utf-16le

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java --encoding utf-16le

You can also specify a report format - here we’re using the XML report:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java --format xml

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java --format xml

The default format is a text report, but there areother supported formats

Note that CPD’s memory usage increases linearly with the size of the analyzed source code; you may need to give Java more memory to run it, like this:

~ $export PMD_JAVA_OPTS=-Xmx512m~ $pmd cpd --minimum-tokens 100 --dir src/main/java

C:\>set PMD_JAVA_OPTS=-Xmx512mC:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java

If you specify a source directory but don’t want to scan the sub-directories, you can use the non-recursive option:

~ $pmd cpd --minimum-tokens 100 --dir src/main/java --non-recursive

C:\>pmd.bat cpd --minimum-tokens 100 --dir src\main\java --non-recursive

Exit status

Please note that if CPD detects duplicated source code, it will exit with status 4 (since 5.0) or 5 (since 7.3.0).This behavior has been introduced to ease CPD integration into scripts or hooks, such as SVN hooks.

Logging

PMD internally usesslf4j and ships with slf4j-simple as the logging implementation.Logging messages are printed to System.err.

The configuration for slf4j-simple is in the fileconf/simplelogger.properties. There you can enablelogging of specific classes if needed. The--debug command line option configures the default log levelto be “debug”.

Supported Languages

SeeCPD Capable Languages for the full list of supported languages.

Available report formats

• text : Default format
• xml (and xslt)
• csv
• csv_with_linecount_per_file
• vs
• markdown

For details, seeCPD Report Formats.

Ant task

Andy Glover wrote an Ant task for CPD; here’s how to use it:

<pathid="pmd.classpath"><filesetdir="/home/joe/pmd-bin-7.21.0/lib"><includename="*.jar"/></fileset></path><taskdefname="cpd"classname="net.sourceforge.pmd.ant.CPDTask"classpathref="pmd.classpath"/><targetname="cpd"><cpdminimumTokenCount="100"outputFile="/home/tom/cpd.txt"><filesetdir="/home/tom/tmp/ant"><includename="**/*.java"/></fileset></cpd></target>

Attribute reference

Also, you can get verbose output from this task by running ant with the-v flag; i.e.:

ant -v -f mybuildfile.xml cpd

Also, you can get an HTML report from CPD by using the XSLT script in pmd/etc/xslt/cpdhtml.xslt. Just runthe CPD task as usual and right after it invoke the Ant XSLT script like this:

<xsltin="cpd.xml"style="etc/xslt/cpdhtml.xslt"out="cpd.html"/>

Seesection “xslt” in CPD Report Formats for more examples.

GUI

CPD also comes with a simple GUI. You can start it through the unified CLI interface provided in thebin folder:

~ $pmd cpd-gui

C:\>pmd.bat cpd-gui

Here’s a screenshot of CPD after running on the JDK 8 java.lang package:

Suppression

Arbitrary blocks of code can be ignored through comments onJava,C/C++,Dart,Go,Groovy,Javascript,Kotlin,Lua,Matlab,Objective-C,PL/SQL,Python,Scala,Swift,C# andApex by including the keywordsCPD-OFF andCPD-ON.

publicObjectsomeParameterizedFactoryMethod(intx)throwsException{// some unignored code// tell cpd to start ignoring code - CPD-OFF// mission critical code, manually loop unrollgoDoSomethingAwesome(x+x/2);goDoSomethingAwesome(x+x/2);goDoSomethingAwesome(x+x/2);goDoSomethingAwesome(x+x/2);goDoSomethingAwesome(x+x/2);goDoSomethingAwesome(x+x/2);// resume CPD analysis - CPD-ON// further code will *not* be ignored}

Additionally,Java allows to toggle suppression by adding the annotations@SuppressWarnings("CPD-START") and@SuppressWarnings("CPD-END")all code within will be ignored by CPD.

This approach however, is limited to the locations were@SuppressWarnings is accepted.It is legacy and the new comment based approach should be favored.

//enable suppression@SuppressWarnings("CPD-START")publicObjectsomeParameterizedFactoryMethod(intx)throwsException{// any code here will be ignored for the duplication detection}//disable suppression@SuppressWarnings("CPD-END")publicvoidnextMethod(){}

Other languages currently have no support to suppress CPD reports. In the future,the comment based approach will be extended to those of them that can support it.

Credits

CPD has been through three major incarnations:

• First we wrote it using a variant of Michael Wise’s Greedy String Tiling algorithm (our variant is describedhere).

• Then it was completely rewritten by Brian Ewins using theBurrows-Wheeler transform.

• Finally, it was rewritten by Steve Hawkins to use theKarp-Rabin string matching algorithm.