Testbeds

In the Testing Center, a Spark service can be applied against a dataset to determine the calculated outputs for many records at a time. The main use cases for this feature are for regression testing, model simulation, and portfolio projection use cases.

After choosing a specific service from the Folder overview, select the icon from the service sidebar. The first screen in Testing Center is the testbeds screen.

Testing Center terminology

A test case is a single test case record. A test case can be composed of multiple single value and table inputs.
A testbed contains multiple test cases.
After a testbed is run against a Spark service, the output is a testbed result. It should contain the same number of records as the testbed.

Add new testbed

Import your own testbed

If a service does not have any testbeds, you will be asked to create one. If the service has testbeds already, you can add another testbed by selecting New testbed.
Select Add test cases.
You can upload either an Excel file or CSV file.
- Excel file:
  1. Need the template file to add your test cases? Select Download it here. The template is based on the inputs from the latest service version.
  2. Open the downloaded Excel file. The file includes a table with the input names as headings.
  3. Follow the instructions in the file to add test cases.
- CSV file:
  1. The CSV file must contain headers in the first row. These should correspond to the names of inputs in the Spark service.
  2. Add the corresponding test data under the headers.
  3. You can add a column for Description which is read by Spark to describe each test case.
- Download API calls Excel file can also be imported directly as a testbed.
Select Browse or drag the workbook into the upload area. Spark supports .xlsx, .xlsm, and .csv files.
Enter a Testbed name. By default, this is read from either inside the workbook or based on the filename.
Optionally, enter a Testbed description.
Select Upload.
The new testbed will be listed in the table.

Handle null values

To set an input as null:

For Excel testbed uploads, an empty cell will be considered as null.
For CSV testbed uploads, an entry that is null will be considered as null.

Similar to the Execute API, the treatment of null is different for single value and table inputs.

For single value inputs, a null, is equivalent to clearing the cell in Excel and numerically equal to 0.
If a table is set to a singular null, then the default value will be used for that input.

Randomized test case generation

Generate test cases randomly based on your input parameters. This is good for broad testing coverage.

If a service does not have any testbeds, you will be asked to create one. If the service has testbeds already, you can add another testbed by selecting New testbed.
Select Randomized test case generation.
Enter a Testbed name.
Select a Service version. By default, the latest service version is selected. The inputs generated will be based on the selected service version.
If your service has Sub functions with subservices, you can select which subservices to include.
Optionally, enter a Testbed description.
Select Next.
Enter the Number of test cases. The default number is 100.
For each input, Spark automatically identifies the range of values based on the Data Validation defined in Excel.
- For categorical inputs where the input range is finite, a subset of the field values can be included in the test case generation.
- For numerical inputs, there are options to define the minimum, maximum, and step size.
- Fields that have dynamic validation that depends on other fields can be automatically updated with the correct dependencies.
- If an Xinput is larger than 1 row x 1 column, then the default values will be used for every test case.
Select Generate test cases.
1. You can monitor the progress in the modal.
2. Alternatively, you can Minimize the modal and monitor progress in Background activity. Once it is completed, select the Generate testbed activity.
When the process is complete, a modal will appear to show a summary of the resulting testbed.
If you want to run this testbed, select Run. Follow the instructions in Run testbed.

Systematic test case generation

Generate test cases systematically to cover all scenarios based on your input parameters. This is good for broad testing coverage to try to ensure coverage across permutations and input levels.

If a service does not have any testbeds, you will be asked to create one. If the service has testbeds already, you can add another testbed by selecting New testbed.
Select Systematic test case generation.
Enter a Testbed name.
Select a Service version. By default, the latest service version is selected. The inputs generated will be based on the selected service version.
Optionally, enter a Testbed description.
Select the method in which the permutations are generated:
Method
Permutations
Example
All Permutations: Spark will generate every possible combination across the single value inputs.
Maximum number such that every combination is tested.
If A ∈ {1,2,3} and B ∈ {6,7}. The 6 combinations generated are 1,6, 1,7, 2,6, 2,7, 3,6, 3,7.
Single Change Permutations: With the default values as the base test case, for each single value input level, Spark will create an additional test case.
Minimum number necessary such that each single value input level is tested once.
If A ∈ {1,2,3} and B ∈ {6,7} and the default values are 1,6. the 4 combinations generated are 1,6, 2,6, 3,6, 1,7.
For each of the inputs, you can adjust the number of variations. Select the input name to open the Edit pane. The variations can be modified by changing the Generation type and parameters.
Generation type
Details
List: Generate test cases for all the listed options.
If there is a List type data validation for this input in Excel, those values will be shown as the starting list. Otherwise, the default value will be the only initial item in the list.
Remove variations by selecting the x button.
To add new variations, click into the input box and begin typing. Hit Enter to create an option.
The number of variations is the same as the number of options.
Random list: Systematically generate test cases from a randomly selected number of items in the listed options.
If there is a List type data validation for this input in Excel, those values will be shown as the starting list. Otherwise, the default value will be the only initial item in the list.
The test cases will systematically step through a randomly predetermined list of the items until the desired number of variations has been reached.
If there are more options than the desired number of variations, not every item will be used.
If there are fewer options than the desired number of variations, some items will be repeated
To remove a variation, click the x button.
To add new variations, click into the input box and begin typing. Hit Enter to create a variation.
You can also select the number of variations.
Number range: Generate test cases to cover all the defined steps within a range.
The Number range input type works similarly to "for loops" in many programming languages. Conceptually, think of skip counting: start at the value for min and skip count by the step size until arriving at the max (or the next number will be greater than the max).
Example: min: 0, max: 20, step size: 5.
The sequence would be 0, 5, 10, 15, 20, producing 5 variations.
If the max was 23, the sequence would still stop at 20 because the next number 25 exceeds the max.
Random number range: Generate test cases selected randomly from all the defined steps within a range.
This functions the same as Number range but instead selects the desired number of items from the total number range list and iterates through it to create its variations.
Similar to Random list, the test cases will systematically step through the same list of randomly selected items until the desired number of variations has been reached. If there are more options than the desired number of variations, not every item will be used. If there are fewer options than the desired number of variations, some items will be repeated.
Static table: For multi-cell range inputs.
For multi-cell range inputs, the only input type available is Table. This works similarly to the List input type for single-cell inputs, except that the value is the JSON-formatted string representing the table.
This table will be repeated for every record.
- Data validation in the Excel file is used to populate the initial set of variations for List and Range generation types.
- Fields that have dynamic validation that are dependent on other fields can be automatically generated with the correct dependencies but the variations can only be randomly selected.
The total number of estimated test cases generated is indicated at the bottom. If the test cases are excessive, the testbed may be truncated.
Select Generate test cases.
1. You can monitor the progress on the screen.
2. Alternatively, you can Minimize the screen and monitor progress in Background activity. Once it is completed select the Generate testbed activity.
The new testbed will be listed in the table.

Change how testbeds are displayed

Search: Enter a search term and the list will update.
Favorite testbeds: Favorite testbeds are displayed first.
Sort: Change the sort order by selecting the List view headers.
List view / Card view: The default Card view shows services and documents as tiles. List view is more compact and displays one item per line.

Testbed actions

Additional actions can be performed on a testbed from the ellipsis menu.

View testbed details

Select a testbed name to go to Testbed details.

Run testbed

Select ▶️ Run to run a testbed.

Select the Service version that you want to run the testbed against.
Select service type: Stay with the default selection of Neuron.
Select execution mode: Legacy execution is the current method of running tests. Batch execution mode is experimental - it should require less configuration for batch size and should process in a shorter period of time.
Enter a Testbed result name.
Optionally, enter a Testbed result description.
If your service has Sub functions with subservices, you can select which subservices to include.
Select Run.
1. You can monitor the progress in the modal.
2. Alternatively, you can Close the modal and monitor progress in Background activity. Once it is completed select the Run testbed activity.
The Run testbed modal should indicate This testbed was successfully run. with some statistics about the testbed result.
To download the result immediately, select Download result. Follow the instructions in Download testbed result.

If you encounter an error relating to timeout and batch size, we suggest the following actions:

Edit the Version details Batch Size Override to a lower number. This reduces the number of test cases that are submitted at a time which reduces the likelihood of a timeout as fewer records will be processed together. The default batch size value is 100.
If you frequently encounter this across many Spark services, the batch size value can be adjusted for new services. This can be done by contacting Support.

Favorite testbed, Unfavorite testbed

Favorite testbeds can be filtered on the home screen using the toggle.

Download testbed

Format

Description

Download as CSV

Return the testbed in a ZIP file containing CSV files. Files are chunked at 25 MB.

Download as Excel

Return the testbed in a single Excel file.

Download as Excel (expanded tables)

Excel plus expanded tables will write table inputs to separate worksheets making it easy to review test cases that include tables.

The download task initiates and can be monitored in Background activity.
Once the download is ready, it will automatically start to download.
If you missed the download, the download can be initiated again in Background activity.

Delete testbed

Before a testbed is deleted, Spark will request confirmation for the delete action. After confirmation, everything related to the testbed will be deleted. There is no process to recover deleted testbeds.

Last updated 14 days ago