Authoring Test Scripts (Validation Management)

Vault Validation Management provides a collaborative and intuitive user interface to help validation professionals author, review, and pre-approve test scripts before they can become available for execution by Executors. The interface allows Test Authors and Test Co-Authors to simultaneously add, copy, and delete setup steps and execution steps to ensure that applicable requirements are challenged effectively. Test Authors and Test Co-Authors can link Requirements to test steps which helps with the automatic creation of the requirements traceability matrix.

About the Test Authoring Interface

When you perform the Author Test action on a test that is in the In Authoring lifecycle state, Vault takes you to the Test Authoring Interface.

Test Header

This section includes the Test Script ID if available or the Name with the Title of the test script being authored. If the test name and title are too long to display, you can hover over the header to view the full name and title.

Step Navigation Panel

The navigation panel on the left side of the interface allows you to select either the Step View or Requirements Burndown View.

Step View

This view displays sections, setup steps, and execution steps of the test script. Clicking on a step or section in this panel navigates your view directly to it in the content panel. You can collapse and expand sections to efficiently navigate through large numbers of test steps.

Requirements Burndown View

This view displays requirements on the associated Validation Entity and allows you to drag and drop requirements onto execution steps that challenge them, linking them to the test step. Once a requirement is linked (challenged) by an execution step, the requirements traceability matrix is automatically updated to reflect this as well.

The Validation Requirement cards in this burndown view display their Risk Level, ID, and Description. Requirements in this view may be automatically filtered to a particular lifecycle state configured by an Admin. As requirements are challenged by an execution step, the requirement card is removed from the burndown list. Eventually, once all requirements have been challenged, the burndown list should not show any more requirements that need to be challenged.

Click the search icon to search for specific Validation Requirements. You must escape characters using \ when executing a search with the following text operators: AND, OR, or ,. For example, \AND.

Click the filter icon to narrow the list of requirements down by Requirement metadata like Risk Level, Category, Source, Type, or Lifecycle State. Click Apply to set the filter options, or Reset to remove them. You can also filter by Requirements’ challenge status:

  • Not Challenged: A Requirement record exists on the Validation Entity which has not yet been linked to a Test Step. This is the default filter for the Requirements View.
  • Challenged Inside this Script: A Requirement record exists on the Validation Entity and it is linked to a Test Step in the current Test Script record.
  • Challenged Outside this Script: A Requirement record exists on the Validation Entity and it is linked to a Test Step in a different Test Script record.

Content Panel

The content panel on the right side of the interface is where the primary authoring of the Test Steps takes place. After selecting the Setup Step section or Execution Step section, click + Step to add a new step within the selected section. Each section displays up to 25 steps per page.

Within each step section, you can use the actions at the top right of the step card to Copy, Delete, or Save changes to the step.

Status & Action Bar

The top right of the interface includes the save status and general actions. The save status indicates when you have unsaved changes in a test step in the content panel. The Reorder Steps action becomes available once you have at least two steps saved in a section, and allows you to reorder steps in a condensed view in the content panel. The Complete button allows Test Authors to complete their assigned authoring workflow task. The Reload button allows Test Authors to reload the Test Script. The Complete button becomes available once you have at least one step saved.

Copying Test Scripts

An Admin may configure an action to deep copy an existing Test Script record in a particular lifecycle state. This action may be named according to the lifecycle state the new Test Script will be transitioned to. For example, the action labeled Re-Run Test Script may copy a test script and transition it to the Pre-Approved lifecycle state. In addition, an Admin may configure an action to deep copy an existing Test Script record to a different Validation Deliverable.

Upon initiating a deep copy of a Test Script from its Actions menu, a dialog appears to select a Validation Deliverable or create a new one before clicking Continue. If the chosen Validation Deliverable does not share the same Validation Entity as the original Test Script, Vault does not copy requirements and warns the user before proceeding with the operation. Vault sends a notification if the operation is successful or fails.

When a user copies a Test Script, Vault copies all of its test steps and requirements to the new record, however, Vault only copies Validation Requirements if the chosen Validation Deliverable shares the same Validation Entity as the source Test Script. If an Admin has configured naming schemes, Vault increases the Run Number value of the new record by one (1) and assigns the same Name upon copying when the source test script’s deliverable and the target test script’s deliverable are the same. If the source test script’s deliverable and the target test script’s deliverable are different, Vault generates a new name and the run number begins from the default value configured for the Run Number field. If your Admin has not configured naming schemes, you must manually update the Run Number of the new Test Script.

Copying Test Protocols

An Admin may configure a user action to deep copy an existing Test Protocol in a particular state. This action may be named according to the lifecycle state the new Test Protocol will be transitioned to. For example, the action labeled Generate New Run may copy a protocol and transition it to the Initiated lifecycle state.

Upon initiating a deep copy of a Test Protocol from its Actions menu, a dialog appears to select test scripts. From the dialog, you can select the test scripts to copy to the new Test Protocol record and click Continue. Vault then prompts you to select an existing Validation Deliverable from the drop-down or create a new one before clicking Continue. Vault sends a notification if the operation is successful or fails.

When a user copies a Test Protocol, Vault copies all of the selected test scripts and their associated test steps, prompts, and requirements to a new record. Vault only copies Validation Requirements if the chosen Validation Deliverable shares the same Validation Entity as the source Test Protocol. Otherwise, the user receives a warning message that Vault will not copy the requirements. If configured by an Admin, Vault only copies requirements in a specific lifecycle state to the new Test Protocol record. The deep copy operation also creates equivalent Requirement Traceability Matrix records according to the linked requirements and test steps of the copied test scripts. If an Admin has configured naming schemes and the target deliverable is the same as the source test protocol’s deliverable, Vault increases the Run Number value of the new record by one (1) and populates the Copied From Protocol field upon copying.

Creating Test Scripts

Test Scripts contain test steps to be performed by one or more Executors to fulfill a Testing Deliverable for a validation activity. Executors perform test steps in the Test Execution Interface.

To create a Test Script:

  1. From a Testing Deliverable record, navigate to the Test Script section.
  2. Click Create.
  3. Fill in all basic field data, including the Title and Description.
  4. Select a Test Type. The options are IQ (Installation Qualification), OQ (Operational Qualification), or PQ (Performance Qualification). If you are creating a Test Protocol, you can select more than one Test Type.
  5. Select whether this test is a Regression Test.
  6. Fill in a Run Number. This value indicates the number of times that the test must be executed either due to a statistical requirement or due to multiple attempts due to discrepancies encountered during execution.
  7. Select an Author.
  8. Optional: Select one or more Co-Authors. Users assigned to this role are permitted to simultaneously author test scripts alongside Test Authors.
  9. Select a Lead Executor.
  10. If this Test Script was based on another script, select a Source Script.
  11. Optional: If available, select a value for Dry Run Required? This field indicates whether the test requires a dry run before proceeding to pre-approval.
  12. Click Save. If configured by an Admin, Vault automatically populates the Record Number field with the appropriate naming scheme.

Once created, you can advance the script to the In Authoring lifecycle state, where you can begin authoring test steps.

Test Scripts typically go through a pre-approval review workflow to proceed to a Pre-Approved state. Until the test is Pre-Approved, it is not available for execution. However, if configured by an Admin, you may perform the Start Dry Run action to dry run a Test Script before its pre-approval and execution. Once a Test Script is in the Pre-Approved state, it becomes available for execution by assigned Executors. At this point, you perform the Start Test Execution workflow action to assign roles and begin the test execution process.

Creating Steps

To begin test authoring:

  1. Navigate to a Test Script record in the In Authoring lifecycle state.
  2. Perform the Author Test action.
  3. From the Test Authoring Interface, proceed with creating setup steps or execution steps.
  4. Click Complete to finish authoring and change the test’s state to In Pre-Approval.

Creating Setup Steps

To create a setup test step:

  1. Navigate to a Test Script record in the In Authoring lifecycle state.
  2. Perform the Author Test action. Vault displays the Test Authoring Interface.
  3. In the navigation panel of the Test Authoring Interface, select the Setup Steps section.
  4. Click + Step.
  5. Add a Step Title.
  6. In the Prompts section, add or remove prompts as needed. The default prompts for a step are Procedure and Actual Results, but each can be removed with the delete icon.
  7. In the Settings section, select a value for the Attachment Setting:
    1. Attachments Allowed: Attachments are optional and it’s up to the Executor to decide if they want to provide objective evidence.
    2. Attachments Required: Executors must attach at least one file to complete the step.
    3. Attachments Disabled: Executors cannot add attachments to the step.
  8. Choose whether to allow comments with the Comment Allowed? radio button.
  9. Optional: Select an Executor. If an Executor was defined on the parent Test Script record, Vault pre-populates the Executor field.
  10. Click Save. When you save a test step, Vault reloads the Test Script if there is no step being edited. If there are other steps being edited, Vault only reloads the saved step. If the Test Script is not up to date, Vault displays a banner to reload the Test Script.

To add additional steps, you can either click + Steps after the last setup step at the bottom of the content panel, or copy an existing step.

Creating Execution Steps

To create an execution test step:

  1. In the navigation panel of the Test Authoring Interface, click Execution Steps.
  2. Click + Step.
  3. Add a Step Title.
  4. In the Prompts section, add or remove prompts as needed. The default prompts for a step are Procedure, Expected Results, and Actual Results, but each can be removed with the delete icon.
  5. In the Settings section, select a value for the Attachment Setting:
    1. Attachments Allowed: Attachments are optional and it’s up to the Executor to decide if they want to provide objective evidence.
    2. Attachments Required: Executors must attach at least one file to complete the step.
    3. Attachments Disabled: Executors cannot add attachments to the step.
  6. In the Settings seciton, choose whether to allow comments with the Comment Allowed? radio button.
  7. Optional: In the Settings section, select a value for the Verdict Setting:
    1. Verdict Required: Executors must select a verdict (Pass, Fail, N/A) to complete the step.
    2. Verdict Disabled: Executors cannot select a verdict for the step.
  8. Optional: Select an Executor. If an Executor was defined on the parent Test Script record, Vault pre-populates the Executor field.
  9. Click Save. When you save a test step, Vault reloads the Test Script if there is no step being edited. If there are other steps being edited, Vault only reloads the saved step. If the Test Script is not up to date, Vault displays a banner to reload the Test Script.
  10. Optional: Add requirements to be challenged by this step.

To add additional steps, you can either click + Step after the last setup step at the bottom of the content panel, or copy an existing step.

Test Step Prompts

You can define up to 15 prompts for each Test Step. You can remove prompts with the delete icon. You can choose whether the prompt should extend fully across the step section with the expand and contract icons if it is a text instruction or text response type prompt. Click the copy icon to duplicate a prompt. To reorder the prompts, drag them with the grip icon.

The default prompts for a step are Procedure, Expected Results, and Actual Results. If you delete the default prompts, you can bring them back using the Restore Defaults button:

  • Procedure: Describes what the Executor must do to complete the step. Click the pop-out icon to view and edit the field in a larger dialog.
  • Expected Results: Describes the expected outcome after completing the step. Click the pop-out icon to view and edit the field in a larger dialog. This is only available for execution steps.
  • Actual Results: An input prompt that allows the Executor to describe the results after testing.

Click Add Instruction to add an instructional prompt. The available instruction prompt options are:

  • Text Instruction: Additional instructions for the executor. Click the pop-out icon to view and edit the field in a larger dialog.

Click Add Response to add a response prompt. The available response prompt types are:

  • Text Response: An input prompt for the executor to provide text.
  • Date/Time Response: An input prompt for the executor to provide a date and time value.
  • Number Response: An input prompt for the executor to provide a numerical value with a unit of measure.

Adding Requirements

Execution steps should link to Requirements that they will challenge. Using the Requirements Burndown View, you can simply drag and drop a Requirement onto the Execution Step which challenges it. You can also link requirements to steps manually in either the Steps View or Requirements view:

  1. Navigate to the Requirements section in an existing execution step. The Requirements section does not appear until the execution step has been saved.
  2. Click Add.
  3. In the Filter Requirements dialog, select whether you want to view and select from All available requirements or filter the list to only requirements that are currently Not Challenged by another step.
  4. Click Continue.
  5. In the dialog, select the requirements you want to add. You can further filter the list using the drop-down menus at the top of the dialog to apply a custom filter.
  6. Click Ok.

The Requirement is now linked to the test step. When you add a Requirement in this section, Vault will update the Requirements Traceability Matrix linking the Requirement, Validation Entity, test script, and test step information. You can change the Requirements section display settings using its Actions menu to add or remove columns.

To remove a Requirement, select it and click - Remove.

Calculating Step Numbers

When you add new test steps between existing ones, Vault automatically calculates the step number and assigns it to the Step field. For example, if you add a test step between steps two (2) and three (3), Vault assigns 2.5 to the Step field of the new test step.

When authoring a test, rearranging steps can result in Step field values with long decimal values, which could be confusing to Executors. After making changes to the step order in a test, select Calculate Step Numbers from the Actions menu of the Test Script record to simplify the Step field values to the appropriate integers. This action may be configured as an entry action on In Pre-Approval and Pre-Approved lifecycle states to ensure the step numbers are more readable.

Dry Running Tests

When viewing a test in the In Authoring state, the Start Dry Run action may be available from the record’s Actions menu depending on your Vault’s configuration. You can select one or more Dry Run Executors to dry run a script by a Dry Run Due Date, in which they can identify scripting errors and provide comments on the test script and its test steps. Upon completion of a dry run, authors can see comments left by the user, resolve or reject feedback, and make adjustments to the Test Script prior to submitting it for pre-approval.

When an author initiates a dry run, Vault deep copies the source test script in addition to its test steps and additional prompts, but does not link it to the Test Protocol or Validation Deliverable associated with the source Test Script record. The operation creates a separate set of records for each assigned executor, overwriting the Executor field on the copied test script with the user selected to execute the dry run. Executing a dry run does not increment the Run Number of the source Test Script but copies the value to the dry run test script.

Vault notifies the user who initiated the action once all dry run copies have been successfully created, the workflow has started, and tasks have been assigned to the Dry Run Executors.

Reviewing Tests

When viewing a test in an In Pre-Approval state, Vault displays the test in a read-only version of the Test Authoring Interface. From here reviewers can review all authored content. Click Review to provide a verdict, justification, and eSignature. A rejection verdict may move the test back into an In Authoring state, where the Test Author and Co-Authors can make any necessary changes.

The Test Script Review Interface is also available via the Review Authored Test action on a Test Script record.

Generating a Printable View

If configured by an Admin, you can execute the Generate Printable View action on a Test Script record to display the Test Script and related records in a simplified format. This view can then be printed to a PDF or otherwise via your browser’s standard print dialog. This action’s label may vary depending on your Vault’s configuration.

Blank Field Value Behavior

In test authoring, the following behaviors apply to field values that are blank for any reason:

Field Functions As Applicable Step Types
Procedure Setting Procedure Enabled Setup Step, Execution Step
Procedure Fit to Width FALSE Setup Step, Execution Step
Actual Results Setting Actual Results Required Setup Step, Execution Step
Actual Results Fit to Width FALSE Setup Step, Execution Step
Expected Results Expected Results Enabled Execution Step
Expected Results Fit to Width FALSE Execution Step

To use the Test Authoring Interface, users require a security profile and a permission set which grants these minimum object-level and field-level permissions:

  • Validation Entity: Read
  • Validation Requirement Entity Version: Read
  • Validation Activity: Read
  • Validation Deliverable (Testing Deliverable object type): Read
  • Validation Requirement (all object types including base and custom): Read
  • Requirements Traceability Matrix: Read, Create, Edit
  • Test Script: Read, Create, Edit
  • Test Step (Setup Step object type and Execution Step object type): Read, Create, Edit, Delete
  • Test Step Additional Prompt (all standard object types): Read, Create, Edit, Delete
  • Unit of Measure: Read