US20210294731A1 - System and method for executing manual tests integrating automation - Google Patents

Abstract

A test manager receives notification by a version control hosted service of one or more files, containing plain text instructions, being stored in a repository to be used for guiding a human tester to perform operations and observe behavior for a system under test. The manual test includes callable automated software, executed by a serverless computing platform, to set the system under test in a predefined state required for the manual test. The test manager is configured to display a user interface having information indicating the manual tests to be executed, including required automated software to be executed, and to provide guiding instruction for the execution of the tests. The test manager communicates with the serverless computing platform, issuing instructions regarding the required automation. The user interface is also configured to receive status information from both the human tester and the serverless computing platform.

Description

BACKGROUND
• It should be noted that this Background is not intended to be an aid in determining the scope of the claimed subject matter nor be viewed as limiting the claimed subject matter to implementations that solve any or all of the disadvantages or problems presented above. The discussion of any technology, documents, or references in this Background section should not be interpreted as an admission that the material described is prior art to any of the subject matter claimed herein.
• Software teams use a variety of tools during development and a large percentage of teams rely on tools that are provided, both commercial and internally implemented, such as version control and serverless computing.
• Exemplary commercially available version control hosting services include GitHub, GitLab, and Bitbucket. These examples are based on the open-source Git distributed version control tool. Git is a version control system used for managing source code, documents, and other artifacts providing a complete history and full version tracking abilities. Software development teams use version control hosting services to store their source code, along with changing history and variations of the source code called branches, in repositories.
• Exemplary commercially available serverless computing platforms focused on application execution include AWS Lambda, Cloudflare Workers, and Azure Functions. Application focused serverless computing platforms provide infrastructure for application execution; part of an existing application and/or an entire application.
• Exemplary commercially available serverless computing platforms focused on Continuous Integration and delivery (CI/CD) include GitHub, GitLab, Bitbucket, CircleCI, AWS CodePipeline, and CloudBees. Continuous integration and delivery (CI/CD) serverless computing platforms provide infrastructure for software teams to automate stages of application development.
• Exemplary internally implemented serverless computing platforms include organizations that allocate servers and the management of their resources such as runtimes, security, etc., enabling an application to offload parts of its code execution environment and/or automate stages of application development.
• For the purpose of this document, “serverless computing platform” shall be used for representing both continuous integration and development (CI/CD) computing and application execution computing, both commercially available and internally implemented.
• Software development teams leverage hosted version control services to store and maintain source code. Teams also use serverless computing platforms for automating the building of the application, executing a series of tests, and optionally deploying the application onto a server, desktop, embedded device, or another type of executing device. Teams also use serverless computing platforms to execute parts of and/or an entire application.
SUMMARY
• It is understood that various configurations of the subject technology will become apparent to those skilled in the art from the disclosure, wherein various configurations of the subject technology are shown and described by way of illustration. As will be realized, the subject technology is capable of other and different configurations and its several details are capable of modification in various other respects, all without departing from the scope of the subject technology. Accordingly, the summary drawings and detailed descriptions are to be regarded as illustrative in nature and not as restrictive.
• One aspect comprises a system for executing manual tests, using automation to set the state of the system under test, as part of the required steps for the execution of the manual tests by the human tester. The system comprises a processor; and a test manager, executing on the processor. The test manager is configured to be notified in response to the creation of a test project, and configured to identify, one or more files containing plain text instructions that are stored and maintained by a version control hosting service. The test instructions are to be executed as a manual test by a human tester on the system under test. The manual test includes callable automated software, executed by a serverless computing platform, to set the system under test in a predefined state required for the manual test. The test manager is configured to display a user interface having information indicating the manual tests to be executed, including actuatable triggers for the human tester to instruct the test manager to communicate with the serverless computing platform to execute the automated software, and to provide guiding instruction for the execution of the tests. The user interface is also configured to receive status information from the human tester and the serverless computing platform executing the automated software.
• Another aspect comprises a method for writing and executing manual tests using automation to set the state of the system under test, as part of the required steps for execution of manual tests by the human tester. The method comprises receiving notification in response to the creation of a test project, and configured to identify one or more files containing plain text instructions that are stored and maintained by a version control hosting service. The test instructions are to be executed as a manual test by a human tester on the system under test. The manual test includes callable automated software, executed by a serverless computing platform, to set the system under test in a predefined state required for the manual test. The method comprises displaying a user interface having information indicating the manual test to be executed, including actuatable triggers for the human tester to communicate with the serverless computing platform to execute the automated software, and to provide guiding instruction for the execution of the tests. The method receives status information via the user interface from the human tester and is configured to receive status information from the serverless computing executing the automated software.
• Yet another aspect comprises a program storage device storing instructions that when executed perform the method comprising receiving notification in response to the creation of a test project, and configured to identify one or more files containing plain text instructions that are stored and maintained by a version control hosting service. The test instructions are to be executed as manual tests by a human tester on the system under test. The manual test includes callable automated software, executed by a serverless computing platform, to set the system under test in a predefined state required for the manual test. The program storage device also includes instructions for displaying a user interface having information indicating the manual tests to be executed, including actuatable triggers for the human tester to communicate with the serverless computing platform to execute the automated software, and to provide guiding instruction for the execution of the tests. The program storage device also includes instructions for receiving status information via the user interface from the human tester and from the serverless computing platform executing the automated software.
BRIEF DESCRIPTION OF THE DRAWINGS
• Various embodiments are discussed in detail in conjunction with the FIGURES described below, with an emphasis on highlighting the advantageous features. These embodiments are for illustrative purposes only and any scale that may be illustrated therein does not limit the scope of the technology disclosed.
• FIG. 1 is a block diagram illustrating a system for retrieving and providing information to a version control hosting service and a serverless computing platform.
DETAILED DESCRIPTION
• It is understood that various configurations of the subject technology will become apparent to those skilled in the art from the disclosure. The following description and examples illustrate some exemplary implementations, embodiments, and arrangements of the disclosed invention in detail. Those of skill in the art will recognize that there are numerous variations and modifications of this invention that are encompassed by its scope. Accordingly, the description of a certain example embodiment should not be deemed to limit the scope of the present invention.
• Various aspects of the novel systems, apparatuses, and methods are described more fully hereinafter with reference to the accompanying drawings. The teachings' disclosure may, however, be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Based on the teachings herein one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspect of the novel systems, apparatuses, and methods disclosed herein, whether implemented independently of or combined with any other aspect of the disclosure. For example, a system or an apparatus may be implemented, or a method may be practiced using any one or more of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such a system, apparatus or method which is practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth herein. It should be understood that any aspect disclosed herein may be set forth in one or more elements of a claim. Although some benefits and advantages of the preferred aspects are mentioned, the scope of the disclosure is not intended to be limited to particular benefits, uses, or objectives. The detailed description and drawings are merely illustrative of the disclosure rather than limiting, the scope of the disclosure being defined by the appended claims and equivalents thereof.
• With respect to the use of plural vs. singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity.
• It will be understood by those within the art that terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are intended as “open” terms unless specifically indicated otherwise (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.).
• It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and/or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).
• In those instances where a convention analogous to “at least one of A, B, and C” is used, such a construction would include systems that have A alone, B alone, C alone, A and B together without C, A and C together without B, B and C together without A, as well as A, B, and C together. It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include A without B, B without A, as well as A and B together.” Various modifications to the implementations described in this disclosure can be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the claims, the principles and the novel features disclosed herein. The word “exemplary” is used exclusively herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations. Certain features that are described in this specification in the context of separate implementations also can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also can be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination.
• The methods disclosed herein comprise one or more steps or actions for achieving the described method. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.
• FIG. 1 is a block diagram illustrating atest manager100 retrieving and providing information concerning a versioncontrol hosting service102 and aserverless computing platform104. The versioncontrol hosting service102 can be used to store plain text instructions used for guiding a human tester to perform operations and observe behavior for a system under test. Theserverless computing platform104 can be used to set the system under test in a predefined state, required for the manual test.
• Thetest manager100, in one embodiment, communicates with the versioncontrol hosting service102 andserverless computing platform104interfaces105,107 via anetwork109. Versioncontrol hosting services102 can be based on the open source Git distributed version control system, which is used, among many other features, for tracking changes in source code during software development. Examples of version control hosting services based on Git that are commercially available products, such as but not limited to are GitHub, Bitbucket, and GitLab.Serverless computing platforms104 provide tooling and infrastructure for automating steps required in the software development process and/or the execution of all or parts of the application. Examples ofserverless computing platform104 focused on application execution that are commercially available products, such as but not limited to are AWS Lambda, Cloudflare Workers, and Azure Functions, Examples of serverless computing platforms focused on continuous integration and deployment (CI/CD) that are commercially available products, such as but not limited to are GitHub, Bitbucket, GitLab, CircleCI, AWS CodePipeline, and CloudBees. An example of internally implemented serverless computing platforms are organizations allocating servers and managing their resources such as runtimes, security, etc., enabling an application to offload parts of its code execution environment. Thenetwork109 can include the Internet, a local area network, or a wide area network, or a virtual network being emulated on a point-to-point system.
• As used herein, atest manager100 is software that guides a human tester to run at least one manual test, from plain text instructions stored in a versioncontrol hosting service102 for the system under test. The manual test includes callable automated software, executed by aserverless computing platform104, to set the system under test in a predefined state required by the manual test. In one aspect, a process of placing the system under test in a predefined state is called “fixturing.” For the purposes of this document, the automated software is a “fixture” used to place the system under test in a state required for the manual test. The automated software can provide status of its intended operation, including, but not limited to, pending, aborted, queued, completed, and other types of operational information, even detailing the status of test cases executed during the execution of the automated software. The automated software can include, but not limited to, setting the operating state of the system, configuring the data content of the system such as database fields and/or persisted files, and other settings affecting the system under test operating behavior.
• Thetest manager100 may be configured to run on a processor of a computing device, such as a desktop application, or as a server application on the Internet that is accessed by a tester via a web browser. The processor may be any conventional general-purpose single- or multi-chip microprocessor such as but not limited to a x86, x86-64, ARM64, an 8051 processor, a MIPS® processor, a Power PC® processor, or an Alpha® processor. In addition, the microprocessor may be any conventional special purpose microprocessor such as a digital signal processor or a graphics processor. The microprocessor typically has conventional address lines, conventional data lines, and one or more conventional control lines. A human tester accesses one of more user interfaces of thetest manager100 via thenetwork109 and a network browser of an electronic device connected to thenetwork109.
• The versioncontrol hosting service102 and theserverless computing platform104 export application programming interfaces (“API”)105 allowing 3rd parties to communicate and provide additional functionality to users of the hosting service. An exemplary API is described in the GitHub Developer Guide, https://developer.github.com/v3/, of February 2020, which is hereby incorporated by reference in its entirety. As used herein, the API, e.g., GET, PUT, PATCH, POST, and DELETE commands may be used to extract and post information to and from the respective service. The API may also include a callback technique commonly calledwebhooks107 as a mechanism by which information is provided to thetest manager100. Thetest manager100, acting as a client, uses the API, to access a user's set of repositories and/or an organization's set of repositories in the versioncontrol hosting service102. A repository is defined as a container of content such as source code. History tracking and revision control are built into the versioncontrol hosting service102. Thetest manager100, acting as a client, may also issue a request via the API to theserverless computing platform104 to execute software. An exemplary AWS Lambda function call, to execute hosted automation software, can be invoked using a HTTP request, is described in the Amazon AWS Lambda Developer Guide, as of February 2020, which is hereby incorporated by reference in its entirety, see https://docs.aws.amazon.com/lambda/latest/dg/welcome.html. The exemplary API called the “dispatch” enables a 3rd party to trigger execution of automation, is described in the GitHub Developer Guide used for continuous integration and deployment (CI/CD), as of January 2020, see https://developer.github.com/v3/repos/#create-a-repository-dispatch-event.
• Thus, there are generally two groups of individuals accessing the version control hosting service: human testers who access certain information from the versioncontrol hosting service102 via thetest manager100 and software developers and/or managers who interface directly with an interface of a web portal of the versioncontrol hosting service102. A software developer and/or manager of the versioncontrol hosting service102 may access the service via thenetwork109 and use a network browser of an electronic device of the developer. Unless proscribed by an organization, a direct user of the version control hosting service is not precluded from accessing the version control hostingservice information102 via thetest manager100 and vice versa. As used herein, a “user of the version control hosting service” refers to an individual who accesses the version control hostingservice information102 directly via a web portal of the versioncontrol hosting service102.
• The content of theserverless computing platform104 is generally accessed by software developers and/or managers who interface directly with an interface of a web portal and/or command line interface (CLI) of theserverless computing platform104. A software developer and/or manager may access theserverless computing platform104 via thenetwork109 and using a network browser/CLI of an electronic device of the developer. Unless proscribed by an organization, a direct user of the serverless computing platform is not precluded from accessing theserverless computing platform104 via thetest manager100 and vice versa. As used herein, a “user of the serverless computing platform” refers to an individual who accesses theserverless computing information104 directly via a web portal/CLI of theserverless computing platform104.
• The hostingservices102,104 and thetest manager100 can be provided by a single commercial product or by multiple commercial products. For example, in one embodiment, the versioncontrol hosting service102 and theserverless computing platform104 are integrated into a single application and thetest manager100 configured as a separate application communicates with the services of each of them via an API provided by the single integrated application. In another embodiment, each of the versioncontrol hosting service102, theserverless computing platform104, and thetest manager100 are integrated into a single application and the respective services are invoked by the services of the other via an internal API or otherwise directly invoked via a private link, API, port, or communication channel. In another embodiment, the versioncontrol hosting service102 andtest manager100 are integrated into a single application package and communicate with theserverless computing platform104 via an interface such as described above. In yet another embodiment, theserverless computing104 andtest manager100 are integrated into a single application package and communicate with the versioncontrol hosting service102 via an interface such as described above.
• Test files are written instructions using a readable language in plain text, providing a set of instructions for testing software manually and/or calling automated software to be executed by aserverless computing platform104. Test files containing textual instructions are handled exactly the same as files containing textual based programming language instructions (e.g. source code) when being stored and maintained in a repository. The test files can be organized and version controlled within folders/sub-folders and separate branches of the repository. A branch is a controlled variation of an existing baseline within a repository. As will be described further below, thetest manager100 may be used to extract the test files from the repository and convey information about the test files with the test files containing required automated software “calls” to be executed by aserverless computing platform104. In one embodiment, the test files are plain text files using a lightweight markup language such as, but not limited to, markdown or textile, defining one or more software automation calls and/or operations to be performed on the system under test by a human tester. In another embodiment, the test files are plain text files using a business driven development language format, but not limited to, such as Gherkin. In one embodiment the test files can define the required callable automated software by referencing an automation descriptor within the test files. Automation descriptors can be represented by textual description within the test file, by an external file, and/or within the user interface, and are used to define the callable automated software identifier, payload, and other attributes required when triggering the software automation to be executed by theserverless computing platform104. An automation descriptor, and theservice computing platform104 API, are used for calling the software automation. Automation descriptors, when represented by files can be organized and version controlled within folders/sub-folders and separate branches of the repository. In one embodiment, automation descriptors follow prescribed syntax as defined by the serverless computing platform. The system under test could be maintained with the same repository, or elsewhere.
• The flow includes defining manual steps for the human tester, defining automated software to be called directly or indirectly by the human tester, configuring automation descriptors used to invoke callable software automation to be executed by theserverless computing platform104, file storage using the versioncontrol hosting service102, and automatic discovery of test files maintained and updated within the versioncontrol hosting service102 by thetest manager100. Depending on the embodiment, the hosting services can be provided by a single commercial product or by separate commercial products. Also, additional steps may be added, others removed, and the ordering of the steps rearranged.
• Starting at step a user defines a test file consisting of manual test steps and automated software calls, to be executed by theserverless computing platform104.
• Set forth below is a certain exemplary test file that may be used to define test steps including both manual test steps and automated software calls:

• # MY TEST
  This test covers very important stuff.
  ## FIRST CHECK THIS
  Verifies that stuff comes up as expected
  - <run::turnOn-gadget>
  - Now check if gadget is online
  - Determine if gadget dashboard displaying “OK”
  - <run::turnOff-gadget>
  - Now check if gadget is offline

• The character “#” indicates the start of a test definition followed by the name of the test. The character set “##” indicates a specific case to be verified. The keyword “run::” indicates specific software automation to be executed by aserverless computing platform104. The following bullets are manual test steps required by the human tester to verify. The characters and keywords used for instructions are examples only and other text based indicators may be used. Also, additional steps, both manual and automated, may be added, others removed, and the ordering of the steps rearranged. The number of tests and/or cases, represented by “#” and “##”, may be added, removed, or rearranged.
• Continuing the user has the option to use thetest manager100 to initiate automated software calls on the creation, termination, and/or during the execution of the manual test, to be executed by theserverless computing platform104.
• An exemplary user interface screen display of thetest manager100 configuration used to associate callable automated software with a manual test. Using thetest manager100, a user can define what set of tests require automated software as part of the test, and associate the required software automation with the test. The callable automated software can be selected to be executed before (setup), after (teardown), and/or during the human tester execution of one or more manual tests and/or manual test steps.
• Continuing at step the user configures an automation descriptor to be used by thetest manager100 when invoking aserverless computing platform104 to execute automated software. Set forth below is a certain exemplary automation descriptor for executing software using GitHub, a serverless computing platform focused on continuous integration and delivery (CI/CD). Thetest manager100 uses the automation descriptor when constructing the API using the dispatch event, as described in the repository dispatch event, https://developer.github.com/v3/repos/#create-a-repository-dispatch-event. The repository dispatch event can be used to schedule the automated software contained in a workflow to be executed by theserverless computing platform104.

• {
  “service”: “github”,
  “repo”: “myorg/myrepo”
  “name”: “turnon-gadget.yml”,
  “payload”: “{
  “start”: “later”,
  “end”: “soon”
  }”,
  }

• The keyword “service” indicates theserverless computing platform104 being used to execute the automated software. The “repo” keyword indicates where the workflow to execute is stored within GitHub. The “name” represents a specific workflow to execute. The “payload” is used to represent the set of available parameters used for controlling the behavior of the callable automated software. The automation descriptor may be represented by plain text, a file and optionally maintained in the repository. The automation descriptor may also be configured and maintained by a user interface of thetest manager100. The keywords used for describing the callable automated software are examples only and other indicators may be used. Also, additional descriptor fields may be added, others removed, and the ordering of fields rearranged.
• Set forth below is a certain exemplary GitHub workflow used to contain callable software automation, as represented with a YAML (Human-readable data-serialization language) file.

• name: turnon-gadget
  on: repository_dispatch
  jobs:
  job1:
  runs-on: ubuntu-latest
  steps:
  - name: Install JavaScript dependencies
  run: |
  npm install ci
  - name: Run script to turn on gadget
  run: |
  node scripts/gadgeton.js

• The keyword “name” at the top indicates the symbolic tag for the automation steps to be executed by the GitHubserverless computing platform104. The “on” keyword indicates the event type used to trigger the execution. The automation can be organized in separate jobs, each running simultaneously. The following instructions perform installation steps, in this example used for a JavaScript associated file. The “gadgeton.js” script is used to automate steps for turning on the gadget being verified. The keywords used are for GitHub only and other serverless computing platforms specify their required syntax. The example is described in GitHub Help documentation, https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions, as of January 2020, which is hereby incorporated by reference in its entirety.
• Set forth below is another exemplary automation descriptor that can be used for AWS Lambda, a serverless computing platform focused on application execution. An AWS Lambda function, described in the AWS documentation guide https://docs.aws.amazon.com/lambda/latest/dg/resource-model.html, as of February 2020, which is hereby incorporated by reference in its entirety. Thetest manager100 uses the automation descriptor and the AWS API, when invoking the automated software, as described here, https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html.

• {
  “service”: “aws”,
  “region”: “west-2”
  “name”: “turnon-gadget-xyzB12x”,
  “payload”: “{
  “start”: “later”,
  “end”: “soon“
  }”,
  }

• The keyword “service” indicates theserverless computing platform104 being used to execute the automated software. The “region” keyword indicates where the function is stored within AWS server global locations worldwide. The “name” represents a specific named URL required to invoke the function. The “payload” is used to represent the set of available parameters used for controlling the behavior of the callable automated software. The automation descriptor may be represented by plain text, a file and optionally maintained in the repository126. The automation descriptor may also be configured and maintained by a user interface of thetest manager100. The keywords used for naming the callable automated software are examples only and other indicators may be used. Also, additional descriptor fields may be added, others removed, and the ordering of fields rearranged.
• A user of the version control hosting service commits files in a repository. The test files are plain text files using a lightweight markup language such as, but not limited to, markdown or textile, defining one or more callable automated software operations and human tester operations to be performed on the system under test. The test files themselves can reference, but not limited to, external links and images. Images can be stored in the repository or elsewhere. The automation descriptors themselves can reference, but not limited to, other files that can be stored in the repository or elsewhere.
• The versioncontrol hosting service102 notifies thetest manager100 of the existence of changes to the files within the repository. Files can be added, removed, or modified. In one embodiment, thetest manager100 previously registered, via an application programming interface (“API”), with the versioncontrol hosting service102 to receive a notification via awebhook107 when a new commit of changes occurs within the repository.
• A test execution process of manual testing integrated with callable automated software setting the state of the system under test. The flow includes retrieving test files from the versioncontrol hosting service102 and presenting on thetest manager100 guiding instructions for a human tester. Thetest manager100 visually indicates steps that require human verification and steps requiring the calling of automated software execution. When automated software is triggered by the human tester, thetest manager100 generates a call to theserverless computing platform104 to execute the automated software. Thetest manager100 presents the status of the automation and allows manual steps status to be entered. Depending on the embodiment, the version control hosting services and serverless computing platform can be provided by a single commercial product or by separate commercial products. Also, additional steps may be added, others removed, and the ordering of the steps rearranged.
• Thetest manager100, in one embodiment, utilizing the API of the versioncontrol hosting service102, extracts a copy of the test files120 in the repository122. Thetest manager100 displays a user interface that includes information guiding a tester to perform operations and observe behavior based on the extracted test files from the versioncontrol hosting service102. The user interface can include: a window displaying the name of the tests, a graphical control to trigger the execution of automated software, a form allowing the tester to specify whether the test passed or failed, and any notes the tester may have regarding the execution of the test. The user interface can also be adapted to show the status of currently executing automated software to be viewed by the tester, control when manual test steps can be executed, allowing sequencing of dependent manual steps.
• Next, thetest manager100 instructs, in one embodiment, via the application programming interface, theserverless computing platform104 to execute software automation as defined by the automation descriptor file126, contained in the Repository122, user interface, or elsewhere.
• In one embodiment, the status of the automated software is presented to the human testers via the user interface. The user interface displays the automation software execution status such as “queued”, “in progress”, “success”, or “failed.”
• An exemplary screen display of thetest manager100 that may be used to guide a human tester to execute a specific test. In one embodiment, the displayed name of each of the tests is selectable via a link and upon selection displays further information regarding each test case of the respective test. Using thetest manager100, the human tester can trigger the execution of each of the test cases, including the callable automated software, indicate whether the test case passed or failed, optionally provide observational information, and assess the status of automated software, regarding the execution of the test.
• In one embodiment, when a manual test requires automated software to be executed by anserverless computing platform104, parameters can be declared within thetest manager100 configuration and/or within the test file. Set forth below is a certain exemplary test file that may be used to define test steps including both manual and automated, wherein the callable automated software accepts parameters:

• # MY TEST
  This test covers very important stuff.
  ## FIRST CHECK THIS
  Verifies that stuff comes up as expected
  - <run::turndn-qadget>
  {
  “this”: “hey”,
  “what”: true
  }
  - Now check if gadget is online
  - Determine if gadget dashboard displaying “OK”

• The above character sequence “{ . . . }” includes fields which define name value pairs. The first parameter name is “this”, with the associated value of “hey.” The second parameter name is “what” and has a value of “true.” There are no limits to the number of parameters. The characters and keywords used for instructions are explarmary only and other text based indicators may be used.
• Set forth below is another certain exemplary test file that may be used to define test steps including both manual and automated, whereas the callable automated software accepts parameters whose values are defined in a separate file:

• # MY TEST
  This test covers very important stuff.
  ## FIRST CHECK THIS
  Verifies that stuff comes up as expected
  - <run::turnon-gadget:/.github/workflows/input1.json>
  - Now check if gadget is online
  - Determine if gadget dashboard displaying “OK”

• The character sequence “:/path/to/file” is used to reference a file containing parameters, versus declaring them explicitly in the test file. There are no limits to the number of parameters contained in the test file. The syntax used to reference the file is used for instructions only and other techniques may be used.
• A test execution process of callable automated software accepting parameters. The flow includes a test retrieved from the versioncontrol hosting service102 with defined callable automated software that include declared parameters within the test file and/or referencing an externally defined file also maintained in file storage using the versioncontrol hosting service102, or elsewhere. Another method available is extraction of parameters from the test manager's user interface configuration, A user can define specific parameters to be associated with the callable automated software. The test manager extracts the parameters and provides the information as parameters to be used for controlling the behavior of the automated software being executed by theserverless computing platform104.
• Thetest manager100, in one embodiment, utilizing the API of the versioncontrol hosting service102, extracts a copy of the test files in the repository.
• Next, thetest manager100, identifies and copies the defined parameters declared or referenced via external file within the text file and/or configured within the user interface of thetest manager100.
• Continuing, thetest manager100, in one embodiment, via the application programming interface, theserverless computing platform104 to execute software automation as defined by the automation descriptor file, contained in the Repository, user interface, or elsewhere. The test manager provides additional information representing the input parameters. An exemplary GitHub workflow, used as callable software automation, as described in https://developer.github.com/v3/repos/#create-a-repository-dispatch-event, using the “client_payload” within the dispatch event to provide parameters to control the automated software executed by theserverless computing platform104.
• In one embodiment, when a manual test requires automated software to be executed by aserverless computing platform104, parameters declared within thetest manager100 configuration and/or within the test file, the input values can be provided by the human tester. The user entered values during the execution of the manual test are sent as parameters to the callable automated software, controlling its behavior, being executed by theserverless computing platform104.
• A test execution process of callable automated software with user entered parameters. The flow includes a test retrieved from the versioncontrol hosting service102 with defined software automation that include declared parameters within the test file and/or the test manager's user interface configuration. A user can during the execution of the manual test enter in input parameters to be associated with the callable software automation. Thetest manager100 extracts the parameters and provides the information as input parameters to be used for controlling the behavior of the automated software being executed by theserverless computing platform104.
• Thetest manager100, in one embodiment, utilizing the API of the versioncontrol hosting service102, extracts a copy of the test files in the repository.
• Next, thetest manager100, identifies the defined parameters declared or referenced via external file within the text file and/or configured within the user interface and presents a user interface to the human tester to receive input values.
• Continuing, thetest manager100, copies the user provided input and uses them as input parameters to control the callable automated software being executed by theserverless computing platform104.
• In one embodiment, when a manual test requires callable automation software to be executed by an serverless computing platform, parameters declared within thetest manager100 configuration and/or within the test file, can use dynamic data values. Dynamic data values can be resolved during test execution runtime by thetest manager100, Set forth below is a certain exemplary test that may be used to define test steps including both manual steps and callable automated software, whereas the input leverages dynamic data values, that requires thetest manager100 to resolve the values dynamically during runtime and use the values as parameters for the callable automated software to be executed by theserverless computing platform104,

• # MY TEST
  This test covers very important stuff.
  ## FIRST CHECK THIS
  Verifies that stuff comes up as expected
  - <run::turnOn-gadget>
  {
  “this”: “hey”,
  “who”: ${{tester.name}}
  }
  output:
  status:
  count:
  - Now check if gadget is online
  - <run::turnUp-gadget>
  {
  “newcount”: ${{turnOn-gadget.output.count}},
  “who”: ${{tester.name}}
  }
  - Determine if gadget dashboard displaying “OK”

• The character sequence ${{tester.name}} references a defined dynamic data value that is supported by thetest manager100 and is to be associated with the parameter named “who.” This example has thetest manager100 determine the user's name executing the test and provides the name as the value of the “who” field. Another example is using the output of a called automated software as input to another callable automated software. The character sequence ${{turnOn-gadget.output.count}} references a defined dynamic data value that is resolved by the test manager and is used as input to a subsequent callable automated software. There are no limits to the number of dynamic data values supported by the test manager. The characters and keywords used for instructions are examples only and other techniques may be used.
• A test execution process of callable automation software with parameters using dynamic data values. The flow includes a test file retrieved from the versioncontrol hosting service102 with defined callable automated software that includes declared parameters within the text file using dynamic data values as parameters. Another method available is extraction of the parameters from the test manager's user interface configuration, where parameters using dynamic data values can also be defined, associated with the callable automated software. Thetest manager100 extracts the parameters, determines the values during runtime, and provides the resolved values as input parameters to be used for controlling the behavior of the automated software being executed by theserverless computing platform104.
• Thetest manager100, in one embodiment, utilizing the API of the versioncontrol hosting service102, extracts a copy of the test files in the repository.
• Next, thetest manager100, identifies and copies the defined parameters declared within the text file and/or within the user interface configuration. For parameters referencing dynamic data values, they are resolved during execution and associated with the input field names.
• Continuing, thetest manager100, copies the resolved dynamic data values and provides them as input parameters to control the automated software being executed by theserverless computing platform104.
• Instructions or code utilized by or for thetesting application100 may be written in any programming language such as but not limited to C, C++, BASIC, Pascal, or Java. The functions of thetesting application100 may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Claims (17)

What is claimed is:

1. A system for executing tests for a system under test, comprising:

a processor; and

a test manager, executing on the processor, the test manager configured to be notified in response to the creation of a test project, and configured to identify one or more files containing plain text instructions that are stored and maintained by a version control hosting service, the test instructions to be executed as a manual test by a human tester on the system under test, the test manager also configured to receive automation descriptors used to communicate with a serverless computing platform to execute automated software to place the system under test in a predefined state,

the test manager configured to display a user interface having information indicating the manual test to be executed and providing guiding instructions for the test execution of the system under test, the test manager having a user interface including one or more actuatable triggers for the human tester to instruct the test manager to communicate with the serverless computing platform to execute automated software to place the system under test in the predefined state, the user interface also configured to receive status information from the human tester and the serverless computing platform.

2. The system ofclaim 1, wherein the test manager receives instructions to sequence a presentation of the one or more actuatable triggers, the received instructions being from the manual test instructions and/or persisted instructions received via the user interface, the received instructions to sequence the actuatable triggers at the beginning, during, or after the human tester executes the manual test on the system under test.

3. The system ofclaim 2, wherein the received actuatable triggers identify one or more parameters that are used for directing the execution of automated software, the test manager using the parameters to control the behavior of the automated software executed by the serverless computing platform.

4. The system ofclaim 3, the test manager configured to display a user interface to receive values from the human tester as input for the parameters used for controlling the behavior of the automated software.

5. The system ofclaim 3, wherein the test manager is further configured to resolve dynamic data values during the runtime execution of the manual test as input for the parameters used for controlling the behavior of the automated software.

6. The system ofclaim 1, wherein the test manager communicates with the version control hosted service and the serverless computing platform via an external application programming interface that is exported by each of the respective services.

7. The system ofclaim 1, wherein services of the test manager, the version control hosting service, and the serverless computing platform are integrated together within a single application.

8. A method for executing tests for system under test, comprising:

receiving notification in response to the creation of a test project;

identifying one or more files containing plain text instructions that are stored and maintained by a version control hosting service, the test instructions to be executed as a manual test by a human tester on the system under test;

receiving at least one automation descriptor used to communicate with a serverless computing platform to execute automated software to place the system under test in a predefined state;

displaying a user interface having information indicating the manual test to be executed and providing guiding instructions for the execution of the tests;

communicating with the serverless computing platform to execute the automated software to place the system under test in the predefined state; and

receiving status information from the human tester and the serverless computing platform.

9. The method ofclaim 8, further comprising receiving instructions to sequence a presentation of the one or more actuatable triggers, the received instructions being from the text instructions and/or persisted instructions received via a user interface, the received instructions to sequence the actuatable triggers at the beginning, during, or after the human tester executes the manual test on the system under test.

10. The method ofclaim 9, wherein the actuatable triggers identify one or more parameters that are used for directing the execution of automated software, using the parameters to control the behavior of the automated software executed by the serverless computing platform.

11. The method ofclaim 10, wherein received user input identified as values for the parameters used for controlling the behavior of automated software.

12. The method ofclaim 8, wherein the text instructions are received via an external application programming interface that is exported by the version control hosting service.

13. A program storage device storing instructions that when executed performs the method comprising:

receiving notification in response to the creation of a test project;

identifying one or more files containing plain text instructions that are stored and maintained by the version control hosting service, the test instructions to be executed as a manual test by a human tester on system under test;

receiving automation descriptors used to communicate with a serverless computing platform to execute automated software to place the system under test in a predefined state;

displaying a user interface having information indicating the manual test and providing guiding instructions for the execution of the test;

Communication with a serverless computing platform to execute the automated software to place the system under test in the predefined state; and

receiving status information from the human tester and the serverless computing platform.

14. The program storage device ofclaim 13, further comprising instructions to execute automated software as part of the manual test.

17. The program storage device ofclaim 13, wherein receive automation descriptors used to communicate with the serverless computing platform hosting the automated software.

18. The program storage device ofclaim 13, wherein received instructions identify parameters that are used for the execution of automated software, the test manager using the parameter to control the execution of the automated software by the serverless computing platform.

19. The program storage device ofclaim 13, further comprising instructions for communicating with the version control hosting service via an external application programming interface.

Images