Executing & Reviewing Test Scripts (Validation Management)

Paperless test execution is a fundamental part of the validation and qualification lifecycle for computerized systems, equipment, and other types of entities. Executors challenge user requirements and functional requirements using pre-approved test scripts. Using the Validation Management Test Execution Interface, Executors can easily perform procedures defined by test authors, understand expected results, reference requirements being challenged, describe their observed results and compare against expected results, and upload attachments to provide objective evidence. Discrepancies encountered during execution can be created directly from the test step where the issue was encountered. Discrepancies are also known as exceptions, nonconformances, deviations, or defects, depending on the organization.

Using the Test Review Interface, Independent Reviewers, Approvers, and Quality Unit Approvers can easily use a review-by-exception approach and ensure that all steps were completed in a compliant manner. Review-by-exception helps reviewers, approvers, and Quality Unit Approvers to focus their efforts on higher risk areas to streamline the review process.

About the Test Execution Interface

When you perform the Execute Test action on a test which is in the In Execution lifecycle state, Vault takes you to the Test Script Execution Interface.

Test Script Header

This section includes the Name, Title, and Run Number (if any) of the Test Script being executed. This section also includes a link to view the associated protocol document in a mini-browser window when the Test Script is not a stand-alone test script and is part of a Test Protocol.

The navigation panel on the left side of the interface has several useful functions. At the top of this panel is a progress bar, which shows your percentage of assigned steps completed. This percentage is specific to each Executor, so if two or more Executors are working in the same Test Script, they may see different values.

Clicking on a step or section in this panel navigates your view directly to it in the content panel. You can collapse and expand sections to efficiently navigate through large numbers of steps.

The step navigation panel displays an icon (Incomplete Step) when you have viewed a step or section, but have not yet completed it. The panel displays a checkmark once you have completed the step or section.

You can limit your view to only your assigned steps by selecting the Show my steps only checkbox in the navigation panel. Steps that are not assigned to you will show as gray in the navigation panel.

Content Panel

The content panel in the lower right portion of the interface contains the Setup Steps and Execution Steps which you will execute. The content panel displays up to 25 steps per page.

Action Bar

The top right of the interface includes a next step locator icon which takes you to the next incomplete step in the Test Script. The pause or terminate icons may also be available depending on your Vault’s configuration.

You must complete all your assigned steps before you can access the Review button.

Executing Test Steps

You can update and complete any step that is assigned to you, in any order. To execute a test step:

  1. Scroll to the step in the content panel or click the step in the navigation panel on the left.
  2. Perform the test as instructed in the step.
  3. If required, add an attachment.
  4. If you are performing an execution step, select a Pass, Fail, or N/A verdict as needed.
  5. If necessary, add comments, values, or discrepancies.
  6. If necessary, pause or terminate the test execution. This option may not be available depending on your Vault’s configuration.
  7. Click Complete to finish the step. If you selected N/A as the verdict in step 4, Vault prompts for confirmation in a pop-up dialog. Click OK to confirm the verdict selection and complete the step.

Adding Attachments

Various actions during test execution allow you the option to add attachments. When an Attachments box appears, you can upload attachments by dragging and dropping files into the box, pressing CTRL + V or CMD + V to paste an image from the system clipboard, or by clicking Upload files and selecting one or more files. You can add up to five (5) attachments per execution step. Execution steps may not require attachments if they’ve been assigned a verdict of N/A.

Attachment File Names

The resulting name of an attachment file differs depending on the source:

  • When you copy a source image from screenshot capture software like Snipping Tool or Snagit and pasted using the system shortcut, the resulting attachment file name is based on the system-managed name for the Test Script record, combined with the system managed name for the Test Step record, and followed by _clip[#].[File Extension], for example, TS-000012_STN-000234_clip3.png.
  • Vault preserves the original file name when you copy a file from a file location such as the desktop, Finder, or Windows Explorer, then paste using the system shortcut.

Attachment Known Issues

  • When using the Upload files option to add attachments, use the system dialog to either drag and drop files or select and open them. Attempting both actions with the same system dialog may result in missed files.
  • You must click into the step or edit dialog before pasting from the system clipboard. Pasting may not work as expected with the Mozilla Firefox browser if multiple files are copied to the clipboard.
  • Copying and pasting files from a Windows desktop environment into the Attachments box using CTRL + V may not function as expected in the Mozilla Firefox browser.

Adding Comments

Click + Comment to add a comment in a pop-up dialog, then click Save. Once saved, a comment icon appears in the Test Step. You can click the comment icon to edit your comment. You can make edits as long as you have not yet completed the step.

Adding Discrepancies

If a discrepancy is required for a step, click + Discrepancy to create a Discrepancy record. In the pop-up dialog, add the Title, Description, and Date, upload attachments if necessary, then click Save. Once saved, a discrepancy warning icon appears on the test step card. You can click the discrepancy warning icon to view the Discrepancy records associated with the step in a dialog. From the dialog, you can navigate to record data and view attachments.

Pausing Test Execution

The option to pause test execution may be available depending on your Vault’s configuration. To pause test execution, click the pause icon in the action bar of the Test Script Execution Interface. This action is also available from the Actions menu on the record’s details page. In the dialog, provide a required Justification to Pause Test, then click Pause. You may be prompted to provide a verdict and electronic signature.

The Test Script record enters a paused state where it cannot be executed. Click View Justification to view the justification for why the test was paused. In this dialog, you have the option to Resume or Terminate test execution. Choosing either option prompts for a justification.

Click View Execution Change History in the Actions menu to view a record of when the test execution was paused, resumed, or terminated.

Terminating Test Execution

The option to terminate test execution may be available depending on your Vault’s configuration. To terminate test execution, click the terminate icon in the action bar of the Test Script Execution Interface. This action is also available from the Actions menu on the record’s details page. In the dialog, provide a required Justification to Terminate Test, then click Terminate. You may be prompted to provide a verdict and electronic signature.

The Test Script record enters a terminated state where its execution cannot be resumed. Click View Justification to view the justification for why the test was terminated.

Click View Execution Change History in the Actions menu to view a record of when the test execution was paused, resumed, or terminated.

Reassigning a Test Step

The option to reassign a test step may be available when a Test Script is In Execution depending on your Vault’s configuration. To assign a different Executor, select Reassign Step from the Actions menu of a test step.

In the dialog, assign a New Value for the Executor. Depending on your Vault’s configuration, Vault may also prompt for a Change Reason and Justification. Click Save to confirm the reassignment. Vault sends a task to the new Executor.

Executor Review

The Review button in the action bar becomes available once all Test Steps are completed. This section is a summary of the Test Script, including any discrepancies, failed steps, and changes made. Click on the numbers at the top of the Test Execution Interface to filter and review discrepancies, failed steps, and revisions pertaining to your assigned steps. This allows you to review-by-exception, clicking through each filter to review and decide if any changes are necessary. Click Back to return to the Test Execution Interface.

After finishing your review, click Complete in the task bar and provide an eSignature.

Making Changes in Completed Test Steps

Once a step is complete, a timestamp is applied to the step and it receives a green checkmark in the left navigation panel. Any additional changes are tracked as though you are re-executing the step, and Executors are required to provide an eSignature for every change to a completed step. Changes can be made to completed steps for Actual Results, Comments, Attachments, Verdict, and Executor (initiated by the Reassign action). To make changes to the actual results for an execution step:

  1. Click on the pencil icon next to the Actual Results field.
  2. Add a New Value.
  3. Add a Change Reason. Select from the following options: Calculation Error, Entry Error, Late Entry, Other, or Spelling Error.
  4. Add a Justification for the change to the completed step.
  5. Click Save. Provide your eSignature if prompted.

After successfully providing the eSignature for the edits, Vault updates the completion date/time to reflect the time the change was made by the logged in Executor.

Steps with changed results are marked with a delta icon . Clicking this icon displays all Test Step Changes for the step.

The Reassign Step, Edit Comment, Edit Verdict, and Add Discrepancy actions become available after a step is complete. These actions require additional information similar to changing the actual results for a step and require eSignatures for the change to take effect.

If you change the verdict of a completed test step from N/A to Pass or Fail, there may be missing required fields or attachments. In this scenario, Vault automatically moves the test step to the In Progress lifecycle state, clears the completion date/time, and removes the green checkmark from the left navigation panel for that particular step. You must populate all required fields and add at least one attachment before you can access the Complete button. If there are no missing required fields or attachments, Vault executes the test step change and successfully modifies the verdict.

About the Test Script Review Interface

Once the Lead Executor has provided an overall Test Script verdict, the Test Script then moves into a series of review phases before final Quality Unit approval. Vault provides a special Test Review Interface to facilitate this process. This process is performed by Independent Reviewers, Approvers, and finally, Quality Unit Approvers depending on your business process.

When you perform the Review Test action on a Test Script record in the In Independent Review, In Approval, or In QA Approval state, Vault takes you to the Test Script Review Interface.

Test Header

This section includes the Name, Title, and Run Number, if any, of the Test Script being executed. This section also includes a link to view the associated protocol document in a mini-browser window if the test script is part of a test protocol and not a stand-alone test script.

The navigation panel on the left side of the interface has several useful functions.

Clicking on a step or section in this panel navigates your view directly to it. You can collapse and expand sections to efficiently navigate through large numbers of steps.

Main Section

The main section in the lower right portion of the interface contains the filters and fully executed Test Steps. Click on the numbers at the top of the main section to filter and review discrepancies, failed steps, and revisions. This allows you to perform a review by exception, clicking through each filter to review and decide if any changes are necessary and whether you will approve or reject the test script. The main section displays up to 25 steps per page.

Click Complete in the top right of the interface to complete your assigned review task and provide your verdict and eSignature.

Blank Field Value Behavior

In test execution, the following behaviors apply to field values which are blank for any reason:

Field Functions As Applicable Step Types
Procedure Setting Procedure Enabled Setup Step, Execution Step
Procedure Fit to Width FALSE Setup Step, Execution Step
Actual Results Setting Actual Results Disabled Setup Step
Actual Results Setting Actual Results Required Execution Step
Actual Results Fit to Width FALSE Setup Step, Execution Step
Expected Results Setting Expected Results Enabled Execution Step
Expected Results Fit to Width FALSE Execution Step
Attachments Setting Attachments Disabled Setup Step, Execution Step

To use the Test Execution Interface or the Test Review Interface, users require these minimum object and field-level permissions, and a security profile which grants the following object access:

  • Validation Entity: Read
  • Validation Requirement Entity Version: Read
  • Validation Activity: Read
  • Validation Deliverable (Testing Deliverable object type): Read
  • Validation Requirement: Read
  • Requirements Traceability Matrix: Read
  • Test Protocol: Read
  • Test Script: Read
  • Test Script Execution Change:
    • Test Execution Interface: Read, Create, Edit
    • Test Review Interface: Read
  • Test Script Execution Change Signature: Read
  • Test Step: Read, Edit
  • Test Step Change: Read, Create, Edit
  • Test Step Additional Prompt:
    • Test Execution Interface: Read, Edit
    • Test Review Interface: Read
  • Unit of Measure: Read
  • Discrepancy:
    • Test Execution Interface: Read, Create, Edit, Delete
    • Test Review Interface: Read
  • Related Discrepancy:
    • Test Execution Interface: Read, Create, Edit, Delete
    • Test Review Interface: Read