Testbed details

In the Testing Center, a Spark service can be applied against a dataset to determine the calculated outputs for many records at a time. The main use cases for this feature are for regression testing, model simulation, and portfolio projection use cases.

After choosing a specific service from the Folder overview, select the icon from the service sidebar. From the Testbeds, select a Testbed name to view the Testbed details.

Testing Center terminology

A test case is a single test case record. A test case can be composed of multiple single value and table inputs.
A testbed contains multiple test cases.
After a testbed is run against a Spark service, the output is a testbed result. It should contain the same number of records as the testbed.

Testbed information

The top of the Testbed details page shows information about the testbed added, modified, last run, version, and test cases.

Add test cases

Add test cases to an existing testbed and increment the testbed version.

Select Add test cases.
Need the template file to add your test cases? Select Download it here.
Open the downloaded Excel file. The file includes a table with the input names as headings.
Follow the instructions in the file to add additional test cases.
Select Browse or drag the workbook into the upload area. Spark supports .xlsx and .xlsm files.
Spark will show the number of existing test cases and number of new test cases.
You can enter a new Testbed description.
Confirm the change by selecting Add.
The testbed will have a minor semantic version update, for example 1.2.0 to 1.3.0 and the test case count will increase.

If the new test cases already exist in the testbed, they will be removed and noted in the duplicated test case count.

Run testbed

Select ▶️ Run to run the testbed.

Select the Service version that you want to run the testbed against.
Select service type: Stay with the default selection of Neuron.
Select execution mode: Legacy execution is the current method of running tests. Batch execution mode is experimental - it should require less configuration for batch size and should process in a shorter period of time.
Enter a Testbed result name.
Optionally, enter a Testbed result description.
If your service has Sub functions with subservices, you can select which subservices to include.
Select Run.
1. You can monitor the progress in the modal.
2. Alternatively, you can Close the modal and monitor progress in Background activity. Once it is completed select the Run testbeds activity.
The Run testbed modal should indicate This testbed was successfully run. with some statistics about the testbed result.
To download the result immediately, select Download result. Follow the instructions in Download testbed result.

If you encounter an error relating to timeout and batch size, we suggest the following actions:

Edit the Version details Batch Size Override to a lower number. This reduces the number of test cases that are submitted at a time which reduces the likelihood of a timeout as fewer records will be processed together. The default batch size value is 100.
If you frequently encounter this across many Spark services, the batch size value can be adjusted for new services. Contact Support.

Testbed actions

Additional actions can be performed on a testbed from the ellipsis menu.

See Testbed actions.

Change how testbed results are displayed

Search: Enter a search term and the list will update.
Source: Filter for a particular testbed result source.
Source
Description
Spark
Testbed result was run from the Testing Center.
Coherent Assistant
Testbed result was created by Coherent Assistant Testbeds feature.
User upload
Testbed result was uploaded by any user.
Service version: Filter for a specific service version.
Sort: Change the sort order by selecting the List view headers.
Select the expand for a testbed result to view the Subservices, Service type, Validation errors, Output errors, Run time, Calculation time, Description.

Compare testbed results

Download a report that will indicate where there are mismatches between two sets of testbed results. The main use cases for this feature are for regression testing and analysis of model changes between service versions.

Select two testbed results.
Select Compare results.
In the modal, select the comparison options.
Choose download format. CSV is recommended as the most compact data format.
Format
Description
Excel (fast beta)
This is an experimental feature. This uses an alternative faster method to write a testbed result to Excel.
Excel
Return the comparison in a single Excel file.
Choose whether to return the full comparison or mismatches only. If you are performing regression testing against an approved set of results, selecting Mismatches only will reduce the file generation time and size.
Choose whether to return the inputs with the testbed comparison. Excluding the inputs will result in smaller file sizes and faster downloads.
Choose how you would like headings to be ordered. Data order sorts the mapped inputs and outputs in the order they appear in the workbook.
Select particular outputs to compare. By default, all outputs are included. Choosing a subset of the outputs makes it easier to review the results and will result in smaller file sizes and faster downloads.
Select Download.
Spark will show the mismatch results.
The download task initiates and can be monitored in Background activity.
Once the download is ready, it will automatically start to download.
If you missed the download, the download can be initiated again in Background activity.

Upload testbed result

To enable broader comparisons across testbeds, you can upload a testbed result to Spark and use it to Compare testbed results.

Select Upload testbed result.
Need the latest version of this testbed to add your testbed result? Select Download it here.
Open the downloaded Excel file. The file includes a table with the input data and output names as headings.
Follow the instructions in the file to populate the outputs for each test case.
Select Browse or drag the workbook into the upload area. Spark supports .xlsx and .xlsm files.
Enter a Testbed result name.
Optionally, enter a Testbed result description.
Spark will show the number of Test cases with results.
Select Update.
The new testbed result is added to the list with the Source User Upload.

Testbed result actions

Additional actions can be performed on a testbed result from the ellipsis menu.

Download testbed result

In the modal, choose the desired download options.
Choose download format. CSV is recommended as the most compact data format.
Format
Description
CSV
Return the testbed result in a ZIP file containing CSV files. Files are chunked at 25 MB.
CSV expanded tables
CSV plus expanded tables will write table inputs and outputs to separate CSV files making it easy to consume testbed result that include tables.
Excel
Return the testbed result in a single Excel file.
Excel (expanded tables)
Excel plus expanded tables will write table inputs and outputs to separate worksheets making it easy to consume testbed result that include tables. This feature will not work if the testbed contains:
A large number of test cases that include input or output tables.
Very large input or output tables.
Excel (fast beta)
A faster version of Excel (expanded tables).
Choose whether to return the inputs with the testbed result. Excluding the inputs will result in smaller file sizes and faster downloads.
Choose how you would like headings to be ordered. Data order sorts the mapped inputs and outputs in the order they appear in the workbook.
Select particular outputs to download. By default, all outputs are included. Choosing a subset of the outputs makes it easier to review the results and will result in smaller file sizes and faster downloads.
Select Download.
The download task initiates and can be monitored in Background activity.
Once the download is ready, it will automatically start to download.
If you missed the download, the download can be initiated again in Background activity.

Delete testbed result

Before a testbed result is deleted, Spark will request confirmation for the delete action. After confirmation, everything related to the testbed result will be deleted. There is no process to recover deleted testbed results.

Explore test case in API Tester

View the inputs for a particular test case in the API Tester. Verify the calculations for the particular use case.

From a downloaded testbed result file, identify the test case record you want to examine and find its UUID value.
Enter the test case UUID.
Select Submit.
The API Tester appears with the input data from the test case.

Aggregate testbed result

If you have a large testbed but are only interested in summaries of the output data, Spark can apply the summation directly.

The interface is designed to allow you to define all of your aggregations together at once.

The aggregation list pane on the left lists all the aggregation blocks.
Aggregation blocks are used to define aggregation operations. Within an aggregation block, multiple aggregation operations can be defined against the same filters and group by.

We start with the default aggregation block named Aggregation #1.
If you want to filter the data before the aggregation, select + for Filters.
1. Select whether to filter for an Input or Output field. Then select the Field name.
2. For tables, you must also select the column(s) to aggregate. Any columns that are not selected will copy their table entries from the first test case.
3. Select the filter Operator and enter a Value.
4. You can apply a compound filter by adding another filter condition.
5. To remove a filter, select the - button.
Select the aggregation(s) you want to perform.
1. Under Aggregations, select the Output name, and the aggregation Operator.
2. Add additional aggregation operations.
3. To remove an aggregation, select the - button.
If you want to group the by any input or output variables, select + for Group By.
1. Select whether to group by an Input or Output field. Then select the Field name.
2. You can apply a compound group by by adding another group by field.
3. To remove a group by, select the - button.
You can cancel the aggregation by selecting the Cancel button.
If you are ok with the aggregation, select the Save button.
The aggregation block name is renamed and the filter, aggregation, and group by fields are no longer editable.
You can add additional aggregation blocks by selecting New aggregation.
After you have completed the aggregation blocks, select Submit.
The aggregation task initiates and can be monitored in Background activity.
Once the download is ready, it will automatically start to download.
If you missed the download, the download can be initiated again in Background activity.

Each aggregation operation that is defined will have a separate worksheet in the Excel.

Last updated 14 days ago