Repository files navigation

Version 1.0.4

28 July 2017

This document provides procedures that ARL Government personnel MUST followwhen releasing software source code and software-related material to thepublic, and for accepting software-related contributions from the generalpublic. Themaster branch of this repository is the only official document;all other branches are for discussion and development only, and may or may notbecome part of a future official policy.

• Definitions
• Goals and Rationale
  • Why Releasing Software is Important
    • Issues Related to Releasing Source Code
      • OPSEC and Security
      • Intellectual Property Issues
      • Liability and Fitness for Purpose
    • The CC0 License and the ARL Contributor License Agreement (ARL CLA)
    • Release Rights
• Release Instructions
  • Major Reviews
    • Informal Approval
    • Code Cleanup and Release Preparation
    • File an ARL Form 1
    • Obtain Invention Evaluation Committee (IEC) Approval
    • Intellectual Property Review
    • Distribution Methods
    • Final Release and Principal Developer Responsibilities
  • Minor Reviews
  • Incorporating External Contributions
  • A Note on Impact and Metrics
    • Evidence of Impact
    • Software Maturity and Software Engineering
• CC0 1.0 Universal (CC0 1.0) Public Domain Dedication
• Contributor License Agreement (CLA)
• Legal Analysis - Software Protection & Release Mechanisms
• Glossary
• Footnotes

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD","SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to beinterpreted as described in RFC 2119. See RFC 2119 "Key words for use in RFCsto Indicate Requirement Levels", Request for Comments: 2119; InternetEngineering Task Force, March 1997 (https://tools.ietf.org/html/rfc2119) forthe complete definitions.

The goal of this memorandum is to both clarify how software developed by ARLpersonnel may be released to the public and encourage ARL personnel to do so,while remaining firmly within the legal and regulatory requirements of theUnited States and the Army. These goals are in some ways conflicting, whichis why the process described in this document may seem bureaucratic andonerous. This chapter explains why publishing software is important and someof the legal and regulatory constraints on doing so. Reading it is OPTIONAL,but doing so may give the reader some insight into the reasons for theprocess.

Software has become an integral part of modern research. It is used ineverything from simulation and modeling, to data gathering and analysis. Inmany cases, the only way to assimilate the details of research published invarious technical papers is by analyzing the software used in the research. Asa result, research that has its source code published has a significantlygreater impact on the public than research that does not. Published sourcecode accomplishes the following:

• Allows external researchers to analyze and verify that the software iscorrect, which helps prove the claims in any accompanying papers are valid.
• Attracts public attention.
• Reduces the barrier to entry for others to join in on the research. As aresult, collaborations are formed as others want to learn more and build onan organization's work.
• If the software becomes dominant, then the organization that owns it becomesthe de-facto leader in the field, and all others follow their lead.

Conversely, when software is not published, the following are true:

• Claims in papers based on the software cannot be verified, potentiallyleading to doubts in the claims.
• Barriers to collaboration are raised significantly:
  • Published research may be ignored, because external researchers do notfully understand it.
  • New researchers must "reinvent the wheel" by writing their own software.
• The ARL may lose its leadership position to others solely due to othershaving published their code first.

The ARL leadership has recognized these facts and, to ensure that ARL remainsa leader rather than a follower, has created an organizational site on GitHub,a social networking website focused on sharing code, to publish ARL softwareto the world as a part of Open Campus. However, ARL is not a private entityand MUST obey numerous laws and regulations when releasing any material to thepublic. These protocols are intended to protect sensitive material frominadvertent release, protect the Intellectual Property (IP) rights of ARL, andprevent ARL from accidentally trespassing on the rights of others.

The ARL faces a number of issues when defining a software release process.Legal and regulatory issues are fully outlined inLegal Analysis - SoftwareProtection & Release Mechanisms, whileother issues are summarized here.

As an organization within the Department of Defense (DOD), ARL MUST be able toproperly evaluate information that is proposed for publication to ensure nosensitive information is accidentally released. This includes software. Sinceit is possible to accidentally violate various laws with even a seeminglytrivial change¹, it is critical that all changes bereviewed by a competent Operational Security (OPSEC) officer before release.This step is intended to protect ARL personnel from the repercussions of sucha release by reducing the chances of it occurring in the first place.

Moreover, just as aggregating unclassified information may raise itsclassification level, combining a set of changes into a whole may also raiseits classification level. This is why any review MUST consider the softwareas a whole, not just the portion that changed.

For most publications, once an ARL Form 1 is completed, the publication isapproved for public release. However, the ARL Form 1 process was designedprimarily for OPSEC and quality control purposes, and does not address IPconcerns. This can cause significant legal challenges if not correctlyaddressed, both when ARL accidentally releases valuable IP without protectingit and when ARL accidentally trespasses on the rights of others. There arethree main types of IP rights that ARL MUST be considered for softwarerelease: trademark, copyright, and patent rights.

Trademarks are intended to prevent consumer confusion over who is the sourceof goods and services. Thus, the name given to software can become a valuablepiece of IP, particularly as the user community starts to associate thesoftware with ARL. Trademark misuse can take a number of forms. Improper orunauthorized use can imply to a consumer that a trademark owner is providingor endorsing a product when they are not. Provided no such implication ismade when using trademarks owned by others, ARL should be relatively safe.More on this can be found inLegal Analysis - Software Protection & ReleaseMechanisms. To help protect ARL IP, allARL-developed software SHOULD use "ARL" in its name.

Copyright is another form of IP that can give owners a method of control overtheir works. Works generated by non-Governmental persons or Governmentemployees acting outside of their official duties automatically have copyrightattached to them. This means that any outside contribution to a Governmentsoftware project will not be owned by the Government unless the copyright isassigned to the Government. Because of this, unless a contract exists betweenthe Government and the outside entity, the outside entity can claim at leastpart ownership of the project and make additional demands of the Government,which may lead to a project being shut down.

Thus far, the Government has had the following options to protect itselfagainst these problems:

• Refuse to share software with entities not covered by a Cooperative Researchand Development Agreement (CRADA), Cooperative Agreement (CA), or anothersimilar vehicle.
• Refuse all outside contributions.
• Arrange for a contractor to author and own the work, and then assigncopyright to the Government.

The process in this document outlines a new option, and as such, requirescareful consideration of copyright implications.

Patents are another method of protecting IP rights. If software is releasedby ARL without adequate review, ARL's patent rights may be impacted. Forinstance, if contractors or other collaborators have co-invented a"software-related invention"², they have the rightto pursue a patent first. Contractors may be legally permitted to patent thesoftware, preventing others from using it. In this case, ARL SHOULD NOTrelease the software to the public under this process. If a project willeventually be released under this process, developers SHOULD consult with ARLLegal to determine the best course of action to take with regard to anycontractors or outside contributors. If this is not done, then at some laterdate, developers may find that there are patents that effectively prevent ARLfrom releasing the software to the public³.

In addition, unless this process is followed, ARL will not be waiving itspatent rights; instead, the right to pursue patent protection falls back tothe inventors (i.e., ARL employees who have created a "software-relatedinvention"). This can lead to legal complications that ARL would ratheravoid.

A major difference between software and a journal paper or other presentationis that software is effectively a machine. As such, there are concerns aboutfitness for purpose, as well as the liability incurred, should the softwarenot meet its stated capabilities. Releasing software without an adequatelicense or contract may leave both the authors of the software and ARL open toliability issues that could be avoided.

With few exceptions, U.S. Government works do not enjoy copyright protection.Because of this, licenses that rely on copyright for their protectionmechanism may be declared invalid under U.S. Law⁴.For this reason, for works that do not have copyright attached, ARL relies ona combination of the Creative Commons Zero (CC0) license(https://creativecommons.org/publicdomain/zero/1.0/legalcode and reproduced atCC0 1.0 Universal (CC0 1.0) Public DomainDedication) and the contributor licenseagreement shown inContributor License Agreement(CLA). All external contributors MUSTexecute and return a CLA to the project owners before their contributions canbe accepted to ensure that all IP issues are settled.

The most complicated aspect of a software release is usually determining ifthere are adequate rights held by the Government to perform a release to thepublic. If the software does not yet exist but there is public releaseintention, it is advantageous to establish adequate rights in advance,particularly for works developed with contractor involvement.

The first question to consider is whether the software or related materialshould be released to the public. While there are considerable collaborativebenefits that have the potential to reduce acquisition costs for theGovernment, there are also potential new costs to consider and risks tomitigate. For example, ARL software in active use that is converted into anopen-source project can reasonably expect to receive externally developedsoftware improvements, but there will be additional information assurance (IA)requirements to review before integrating those contributions. To releasesoftware, a supervisor MUST decide that there is an overall expected benefitto the Government to perform the release.

The second question to consider is whether or not the software can be releasedunder the ARL Form 1 process with a distribution statement of Distribution A,"Approved for public release; distribution unlimited." If this is notpossible, then the software cannot be released under the process in thisdocument. Contact ARL Legal and ARL Security with the specifics of theproject to see if there are changes that can be made that will allow it tohave a Distribution A, "Approved for public release; distribution unlimited"statement. The following is a non-exhaustive list illustrating why somesoftware and related material face complications in releasing it to the publicdue to law, regulation, or policy. Software that falls into these categorieswill require further scrutiny. Consult with ARL Legal to determine if it ispossible to release the software.

The third question involves the rights of others. Copyright attaches to workswhen they are created⁵; if ARL does not own thecopyright to a work, it MUST obtain the rights to release the material beforeit may do so. If a software project does not yet exist, then there are nocopyright, authorship or licensing issues preventing release. However, ifwhat is being prepared for release already exists, a thorough review toascertain origination, provenance, and licensing MUST be conducted to ensurethat the rights of others are not trespassed.

If the work was developed solely by Government employees as part of theirofficial duties, then there are no copyright concerns and the Government hasthe right to release the work. If there are any contractors or other externalparties involved, they may have rights that prevent the Government fromreleasing and/or redistributing their contributions without their permission.If there are any third-party libraries, applications, or data that areincorporated into the work, there MUST be appropriate license and/or rights todistribute them.

When works are developed by a mixture of Government employees and Governmentcontractors, determining who has the right to release the software to thepublic as open source depends on what contract clauses are in effect. Consultwith ARL Legal to determine what clauses are in effect and what options thereare to release the software.

These instructions MUST be followed when ARL personnel release software to, oraccept contributions from, the general public. If a project does not yet haveany software associated with it (such as when a project is being initiallyformulated) and the project is intended to be released to the general public,then this process MUST still be followed.

The major review process MUST be followed if any of the following are true:

• This is the first release of the project.
• The project's scope has changed sufficiently that any of the principaldevelopers (PDs), their OPSEC officers, or anyone in their chains of commandbelieve a new one ought to be filed.
• It has been more than 1 year since the last time a major review has beendone and the project is still active (material is being published to thepublic).
• The PDs feel they have accomplished something of note and wish to get creditfor it in their performance metrics.

Before developer(s) release software, they MUST obtain informal approval fromtheir supervisor(s). If their supervisors do not approve release of thesoftware, then the software MUST NOT be released. Do not continue with thisprocess. When deciding if a project can be released, review the requirementsof RELEASE RIGHTS. The requirements in that chapter MUST be met before anysoftware or related materials are released.

Fast-moving projects often accumulate useless, nonfunctional, or otherwiseundesirable code and other material that needs to be cleaned up. Beforemoving forward with the formal portions of the process, the project MUST becleaned up to ensure it meets certain minimum standards. Remember that aformal review can only be done on what is actually being released, so cleaningup the code after the formal review is not an option. If a project is beingformulated, but does not yet have any code associated with it, this sectionSHOULD be used as a guide for how to write the software.

This section applies to everything that is being released, including any oldercommits in any repositories. By design, repositories preserve history, whichcan include material that should not be published. It is the responsibilityof the project's developers to ensure that both the current code and anyhistory in any repositories that are proposed for release have been properlyscrubbed before the material is reviewed for release.

Software that is released to the public is similar to a publication and SHOULDbe treated like one. The author(s) MUST ensure that there is no embarrassing,disparaging, or otherwise unprofessional language in what is released.Language that would not be used in a professional journal MUST not be used insoftware. Direct any questions about this to the ARL Public Affairs Office(PAO).

Where possible, it is wise to follow best practices in software engineering.Because of the wide variety of programming languages in use, project goals,etc., ARL wants to avoid forcing a single process on any developers or group.For this reason, ARL has chosen a minimal set of requirements and providessome best practice suggestions. Individual implementation of the voluntaryportions is recommended as they may have an effect on impact and metrics. SeeA Note on Impact and Metrics for details onmetrics.

Unless a project is required to follow other guidelines, the guidelinesdescribed in the latest Semantic Versioning guidelines (http://semver.org/)SHOULD be followed when setting the version number for any release. A fileMAY be created in the top-level directory called "VERSION." If the "VERSION"file is created, then it MUST be a plain-text file in either ASCII or UTF-8encodings. The file MUST contain the version number of the release and MUSTNOT contain anything else. By following this guideline, automated systems aremore likely to be able to determine if the project has been updated and howsignificant those updates are simply by parsing the "VERSION" file.

The license to be used depends on whether or not the project has copyrightattached to it. Works generated solely by Government employees in the courseof their duties do not have any copyright attached to them. Works producedwith non-Government persons or organizations may have copyright attached. Ifthere is uncertainty about the copyright status of a project, ARL Legal SHOULDbe consulted to determine the legal state of the project and determine whichlicense to use.

If the project does not have copyright, the Creative Commons Zero 1.0Universal (CC0 1.0) Public Domain Dedication MUST be used. Follow theinstructions in that section for how to use it.

If the project has copyright, any license that ARL Legal approves of MAY beused. It is RECOMMENDED that the Apache 2.0 license(http://www.apache.org/licenses/LICENSE-2.0) be used.

Unless the guidelines the project is following require otherwise, the longform of any license used SHOULD be in a file called LICENSE in the top-levelof the project directory. For any LICENSE file, the file MUST be a plain-textfile in either ASCII or UTF-8 encoding. The README file (described below)MUST state the name of the file that contains the completelicense⁶. If the project has a webpage, the licensebeing used MUST be stated somewhere on the webpage, with a link pointing towhere the file containing the license can be found.

All contributions to the project MUST be done under the license and thecontributions MUST be irrevocable. Questions about this can be directed toARL Legal for clarification.

A "README" file MUST be created at the top level of the directory with atleast the following in it:

• The intended purpose of the software.
• A note pointing to the license or contract covering the software.
• If there is no "VERSION" file, the version number of the release SHOULD beincluded.
• At least some basic documentation on how to build and use the software.

The "README" file MUST be a plain-text file in either ASCII or UTF-8 encoding.

While only the "README" file is mandatory, other documentation is highlydesirable. This can include comments in the source code, high-level designdocumentation, etc. There are many methods and tools to support this type ofdocumentation. Where reasonable, please document both high- and low-leveldesign details, as well as how the software should be used.

Example code is also a form of documentation. If the project is a library, itcan be useful to provide small, complete, and well-documented programs thatillustrate how to use the software. If possible, refer to publications andprojects that used or referred to this code; they can become additionalexamples of how to use the code, as well as testimonials for why the codeshould be used.

Unit tests are strongly RECOMMENDED. They can not only increase confidencethat the code was written correctly, but they can also convince a user thatthe code is behaving as expected on the system on which it is installed. Thiswill increase the likelihood that others will be willing to use the code,making it wise to include unit tests in the project. In addition, unit testscan serve as examples of how to use the code; this can be invaluable when auser is trying to understand the documentation.

For legal reasons, all language talking about the project MUST be prefixedwith "ARL". For example, if a project is named WhizBang, then all literaturein the package SHOULD refer to it as "ARL WhizBang" or the "ARL WhizBangproject." "ARL" is a federally registered trademark and using it in thismanner adds some degree of trademark protection to a project.

An ARL Form 1 MUST be filed. In the process described in this document, theprimary purpose of the ARL Form 1 is for OPSEC review, but it also serves thesecondary purposes of describing releases for publicity and metrics. Tosupport this, a short abstract describing the software MUST be written. Theabstract SHOULD be at most one page in length and provide the followinginformation:

• The name of the project.
• A description of the project. This includes what it does and what itsintended purpose is. This SHOULD be as complete as is reasonably possible.Portions of the "README" file MAY be copied here.

This information will be used both for publicity purposes, and by a supervisorand others to decide if it is time for a project to be subjected to anothermajor review.

If this is not the first major review of the software, then the abstract MUSTalso include the following information:

• The change or changes that caused this review to become necessary.
• What the impact of the software has been since the last major review forthis project. SeeA Note on Impact and Metrics for examplesof how impact is measured and for what to include.

Note that while ARL wants to credit an author for the impact the software hasmade, it will not "double count" what authors have done by including theimpact from earlier major reviews. Only the impact made since the last timethe major review process was completed SHOULD be included.

This abstract, along with everything planned on being released (software,source code, documentation, etc.), MUST be fully reviewed by a level 1 OPSECofficer. If this is not an initial review, the OPSEC officer MAY choose toonly review what has changed since the last review, but both the author andthe OPSEC officer are responsible for the release as a whole. Thus, even ifthe changes are cleared for public release, if the release as a whole cannotbe cleared for release, then the changes are not cleared for release either.To be cleared for release, the project as a whole MUST receive an "Approvedfor public release; distribution unlimited" statement. Finally, no one ispermitted to OPSEC-approve material that he or she created. Review RELEASERIGHTS for what needs to be considered.

The ARL may have IP interests in the software. Before it can be released, theIEC MUST determine that it is in the best interest of the Government and ARLto waive any IP rights that ARL might have and release it to the public. Todo so, the PD MUST inform the current chair of the IEC (or the chair'sdelegate) of the intention to release the software by sending the chair adigitally signed email that contains the following:

• The abstract that was submitted as a part of the ARL Form 1 process above.The related software is not sent to the IEC chair (or the chair's delegate)unless he or she requests it.
• A list of all software-related inventions that the PD and his or hersupervisor believe are contained in the work⁷.
• A statement that, in the opinion of the PD and his or her supervisor, allIP, including the listed inventions, should be irrevocably placed in thepublic domain.

If the chair (or the chair's delegate) agrees that ARL should waive any patentor other IP interests ARL may have and that the software should be put intothe public domain, then the chair (or the chair's delegate) will reply backwith a digitally signed email with a statement similar to the following:

<<Software name>> is to be dedicated to the public domain for promoting its commercial and non-commercial use. It is not intended to become the subject of any patent application or license. All intellectual property rights ARL may be able to assert or establish are hereby waived.

If the PD has received approval from IEC chair (or the chair's delegate), thenthe PD MAY continue with the rest of this process.

If the IEC chair (or the chair's delegate) believes that ARL has IP intereststhat ARL wants protected, then the PD and the chair (or the chair's delegate)MUST discuss the issues to determine how to move forward. This discussion MAYbe performed by email, telephone, or any other convenient and legal means.Records of the final determination MUST be kept by the PD. If it isdetermined to be in the best interests of ARL and the Government to seekpatent protection, then the rest of this process does not apply. Do notcontinue with the rest of this process.

To determine the appropriate person to contact within the IEC, consult ARLLegal.

This step MAY be done in parallel with the steps described below.

Although ARL MAY choose to waive ARL's rights to any IP established insoftware, if an author has incorporated contributions from others, thosecontributors may have rights to those contributions that restrict ARL'sability to release the software. Thus, before the software can be released,the following questions MUST be addressed:

• Has every part of the proposed release been generated by Governmentemployees in the course of their duties?
• If not, is there permission from every other rights holder to release all ofthe other parts under the project's license?

The PD MUST consult with ARL Legal to perform an IP review (this review MAY bedone by email or any other convenient and legal means). If there are externalcontributions that were not contributed under the project's license, then thePD MUST determine the license and copyright information for each contribution.Provide these to ARL Legal for review and final determination if the licensesare compatible with the license or contract under which the software is beingreleased. If ARL Legal determines that there are impediments to releasing thesoftware, whatever permissions are necessary MUST be obtained before thesoftware is released. If it is not possible to obtain the necessarypermissions, then the software MUST NOT be released. Do not continue with therest of this process. If ARL Legal agrees that there are no IP impediments toreleasing the software, then ARL Legal MUST send a digitally signed email tothe PD stating so.

Note that copyright protection attaches to all literary works, includingsoftware, when they are created⁸. This includessoftware copied off of blogs, sites likehttp://stackoverflow.com/, and anyother sources. If such code or documentation has been copied into a projectwithout permission and permission from the rights holder cannot be obtained,it may not be possible to release the software.

There are many different ways of distributing software. The ARL GitHub siteis the RECOMMENDED method, but is not the only method. The PDs MAY choose todistribute by FTP, email, another website, or other means, in addition to orin place of GitHub. For any and all distribution methods chosen, the PDs areresponsible for creating their own accounts. If the distribution method usesemail addresses as part of the sign up process, then the PDs MUST use theirGovernment email addresses. All login accounts MUST be reported to the PDs'supervisors. Passwords MUST be kept secret. If a site uses cryptographicauthentication such as public/private key pairs, PDs MAY choose to use thisfacility in addition to, or instead of, passwords. Private keys MUST betreated like passwords and kept secret.

Where possible, it is strongly RECOMMENDED that projects be distributed viathe ARL GitHub site. This will make it significantly easier to gauge theimpact of the project by the Technology Transfer and Outreach Office (T2O2)and by a supervisor, which will impact performance metrics.

If the software is approved for final release and there is not yet a projectfor it on GitHub, then the T2O2 will generate a project on the ARL GitHubsite. If there is already a project on the ARL GitHub site for the software,then the T2O2 will note the release for metrics purposes, but will take noother action. If an author chooses not to use the ARL GitHub site, he or sheMAY request that the T2O2 not create a project. However, as mentioned inDistribution Methods, it is strongly RECOMMENDED that GitHub be used todistribute software.

The PDs SHOULD put their software on their project's site. The T2O2 willpublicize this site on behalf of the PDs. The software MAY also be distributedby any and all other methods the PDs see fit. The PD (or PDs) MAY promotetheir projects on their own, but SHOULD do so in consultation with the T2O2,to both ensure that all promotions are professional in nature and minimize anyduplication of effort.

If the PDs choose to use GitHub as a distribution method for their software,then the PDs are responsible for generating a digital object identifier (DOI)for the release. The instructions to do so are athttps://guides.github.com/activities/citable-code/.

Minor releases include all releases that the PD, the PD's OPSEC officer, andthe PD's supervisor do not believe require a major review. These include bugfixes and minor updates. A minor release only requires review by a level-1OPSEC officer before being published. These reviews are called "minorreviews" and are subject to the following:

• The OPSEC officer MAY also be the technical reviewer for the release.
• Only if the project as a whole, including the minor changes being proposedfor release, receives an "Approved for public release; distributionunlimited" statement can the changes be released.

The process for a minor release is as follows:

• The person who created the material sends it to the OPSEC reviewer.
• The OPSEC reviewer reviews the material and sends a digitally signed emailback to the creator either stating that it may not be released or that itmay be released.
• If the material may be released, then the creator pushes the material out tothe public.

Note that there are numerous methods of sending the material. If the materialis stored under git, and both the creator and the OPSEC officer have access tothe same private repository⁹, then the creator MAYchoose to push to the private repository and ask the OPSEC officer to pull itand review the changes. Alternatively, git bundles MAY be used, or one couldeven email text files. Regardless of the method chosen, there are tworequirements that MUST be met:

• The chosen method MUST uniquely identify the set of changes underdiscussion. Examples include the complete git commit hash under discussion,emails with the complete change set, or any cryptographically signedmethod. This ensures non-repudiation or confusion.
• The OPSEC reviewer MUST track what he or she has approved. This MAY be doneby saving the digitally signed emails that were sent.

These steps are REQUIRED for audit purposes. Without them, ARL cannot proveto the Department of the Army that it is properly reviewing material before itis released.

If a release appears to be a major release in the opinions of any of the PDs,the OPSEC officer, or any of their superiors, then a new ARL Form 1 MUST befiled and the full release process described above re-executed. A majorrelease is any release that significantly changes the scope of the project ormay violate any of the checks described in this document.

Once the software has passed the process outlined above and has been publiclydistributed, any contributions to the project MUST be subject to its license.

External contributions do not need to undergo OPSEC review as they are assumedto be public at the time of contribution. They SHOULD be reviewed for qualitypurposes before being accepted into a project to ensure that they areprofessional in nature and perform as expected. All external contributorsMUST have a CLA on file before their contributions are accepted into theproject. A CLA only needs to be executed once by each legal entity. Projectowners MUST turn over executed CLAs to the T2O2 for record keeping. Projectowners MUST explain in the README file that external contributors MUST executea CLA before their contributions will be accepted.

Publishing software as a regular business practice is new territory for ARL,and how to measure a project's impact on the general public is still a matterfor debate. While GitHub gathers numerous statistics on projects, from thenumber of downloads, to the number of followers, etc., these are, at best,suggestions of what a project's true impact is. For the sake of metrics, aproject's PDs MUST create a short report (at most one page) describing whatthey think the impact is and providing evidence to back up this claim. Thegreater impact a project has had, the better it is for metrics and performanceevaluation.

To be absolutely clear, impact is based on major reviews (releases that followthe procedure outlined in Major Reviews). Minor reviews, as described in MinorReviews, might lead toward a major review, but will not count toward metrics.If an author wants releases to count toward metrics, the procedures outlinedin Major Reviews MUST be followed.

It is up to the PD to decide what types of evidence to use when describing theimpact of a project. As noted in File an ARL Form 1, ARL will not "doublecount" the impact; ARL is primarily interested in the impact since the lastmajor review. That said, some forms of evidence are difficult to express as adifference from a prior major review. In that case, a PD MAY wish to expressboth the current absolute numbers and the changes since the project was lastfiled.

Below are some examples of evidence PDs MAY wish to consider giving:

• The number of citations of different releases. It is possible to citereleases on GitHub just as papers might be cited. Seehttps://guides.github.com/activities/citable-code/ for more information onhow to do this.
• If a project is a library or something else that is intended to beincorporated into other projects, list those projects and describe theirimportance.
• The number of followers, including the level of interest as measured byissues opened and persons contacting the PDs
• The number of forks
• The number of contributions from outside (non-ARL) sources
• Letters of acknowledgment, thanks, or other forms of recognition
• Software maturity

The PDs should not feel that they are limited to these pieces of evidence, norshould they feel the need to include all of these examples. As statedearlier, ARL is still learning which metrics to use when determining theimpact of a project. The PDs should feel free to use any metric they wish, butshould note that supervisors and others will make the final decision on howheavily to weight the chosen metrics. For example, the number of citationsfor a software project is more important than the number of downloads it has.If a project is a library, then the number and importance of projects usingthe library will have greater weight than the number of people that haveforked the project¹⁰.

In general, supervisors will want to know how a project has made a differencein the world and how important that difference has been since its last majorreview. The better evidence provided, the easier this becomes.

The ARL would like to be a world leader in computer science research. To be aworld leader means that ARL must have an impact and to have an impact meansthat ARL's software must be used. Poorly written software that isincomprehensible or difficult to compile or use will not be used, and willhave little, if any, impact. Thus, if the primary improvement to a projectinvolves bringing it in line with generally accepted best practices insoftware engineering to facilitate its transition to others, then this is alsoof value and SHOULD be credited. Examples of this include, but are notlimited to, the following:

• Providing a well-designed, thoroughly documented application programminginterface (API) or other interface.
• Adding tests (unit tests, regression tests, integration tests, etc.) thatincrease confidence that your software is performing correctly.
• Increasing code coverage of tests.
• Adding examples and high-level documentation. This includes ensuring thatall examples and documentation are up to date with the current software.
• Simplifying building and installation of software.

These are just some examples of mature software. If there is other evidenceof the maturity or quality of software, a PD should feel free to use it whendescribing the impact of the software. Supervisors MUST consider improvementsin software engineering when considering the impact of software.

The text of the CC0 license can be foundhere.A local copy of the license is in the fileLICENSE.txt.

The ARL Contributor License Agreement (ARL Form 266) can be foundhere¹¹. Each externalcontributor must execute and return a copy for each project that he or sheintends to contribute to. Once ARL receives the executed form, it will remainin force permanently. Thus, external contributors need only execute the formonce for each project that they plan on contributing to.

An in-depth analysis of the issues surrounding the release of software can befoundhere.

A glossary of terms can be foundhere.

¹ As an example, U.S. law restricts thestrength of any encryption software that is being released to the world ingeneral by restricting the length of the keys being used. Since this could bea hard-coded number in a piece of source code, a change in a single line mayconvert it from legal to illegal.

² Software may be copyrighted but notpatented. "Software-related inventions" may be patented. At the time of thiswriting, there was no clear definition of the difference between "software"and "software-related invention". Consult ARL Legal for a more completeexplanation.

³ Contracts usually protect the FederalGovernment from the related patent issues. That said, if you have concerns,consult ARL Legal and your contracting officer to determine what rights thereare.

⁴ At the time of this writing, this hasnot been litigated in Federal Court. Consult the ARL Chief Counsel's Officeto determine what the current laws are.

⁵ Except for works created by U.S. FederalGovernment employees in the course of their duties. These are automaticallyin the public domain.

⁶ Some licenses have guidelines thatdiffer from the ones described here. For example, the GPL states that thefilename must be COPYING. This is why your README MUST specify which filecontains the license.

⁷ If you have questions about what are"software-related inventions", consult with ARL Legal.

⁸ Excluding works generated by Governmentemployees in the course of their duties.

⁹ A private repository is only accessiblefrom within ARL; it MUST NOT be publically accessible!

¹⁰ As an example, if a library is onlyused within the Linux kernel, it will have been used by very few projects, butwill have an extraordinary impact.

¹¹ This form may not preview correctly inyour browser. If you have trouble opening the file, or if the file has aphrase similar to "Please wait... If this message is not eventually replacedby...", then try downloading the form and opening it in the latest version ofAdobe® Acrobat Reader. ARL is aware of the issue, and is working to correctthe problem.