Remote service input and output data formats

When Spark communicates with a remote service - sending or receiving data - it's crucial to adhere to the proper data formats to ensure the external code is executed and returned successfully. When developing the remote service, it is important to conform to the input and output data requirements. The design of the input and output formats aligns with the definitions in the aligns with Execute API (v4) "JSON array" Execute API (v4).

If you are using a Proxy service to intermediate the XConnector, it would need to be able to reformat the inputs and outputs to conform to the requirements of the Introduction to XConnector and Introduction to XConnector.

After reading this section, you should be able to understand the data formats that the Introduction to XConnector must follow to accept and return requests.

Request format from Spark XConnector to Remote service

This section describes the format of the HTTP request that is being sent from the Introduction to XConnector to the Proxy service and Introduction to XConnector. This is important because the Introduction to XConnector needs to be able to consume the input data to process the result of the code execution. The HTTP request from Spark takes the form of a POST request containing headers and a request body.

Request headers

The header information typically appears to the remote service as a collection of key value pairs. Usually the HTTP request header will incorporate Authorization key with a Bearer token.

Request body

The top-level is a JSON object, presenting a set of name/value pairs (also termed a "dictionary"). There is a JSON array called inputs that is used to denote all the input records to pass to the Introduction to XConnector.

The JSON should follow the format prescribed in Execute API (v4) Execute API (v4), however note that for XConnector, only the JSON array format is supported.

Data type serialization is described in Execute API (v3) request_data.

Example 1: Single value inputs

• Within inputs, the first record [ "key1", "key2" ] defines the string keys.

• Subsequent arrays are used to define the values for the subsequent input records.

Data	Input format
gP9iRaleAcio-MHpRkbvLxbZY	{ "inputs": [ [ "key1", "key2" ], [ "value11", "value21" ], [ "value12", "value22" ], [ "value13", "value23" ], ] }

Example 2: Tables of inputs

• Within inputs, the first record [ "Table" ] defines the string keys.

• Subsequent arrays are used to define the values for the subsequent input records.

Data	Input format
zuxt33WEgjC8-MHpRkbvLxbZY	{ "inputs": [ [ "Table1" ], [ [ [ "TableKeyA", "TableKeyB" ], [ "ValueA1", "ValueB1" ], [ "ValueA2", "ValueB2" ], [ "ValueA3", "ValueB3" ] ] ] ] }

Example 3: Multiple records of single values and tables

Data	Input format
	{ "inputs": [ [ "Course", "Semester", "Students" ], [ "ECON101", "2022 Spring", [ ["SID", "Grade"], [ 123, 90 ], [ 234, 76 ], [ 345, 54 ] ] ], [ "ECON102", "2023 Fall", [ ["SID", "Grade"], [ 234, 85 ], [ 567, 72 ], [ 890, 97 ] ] ], [ "STAT101" , "2023 Fall", [ ["SID", "Grade"], [ 234, 77 ], [ 456, 80 ], [ 678, 91 ] ] ] ] }

Response format from the Remote service to Spark XConnector

This section describes the format of the HTTP response that is returned from the Introduction to XConnector to the Introduction to XConnector. This is important because the Introduction to XConnector needs to be able to return the appropriate data as the result of the code execution. The HTTP response from the Introduction to XConnector should take the form of a POST request containing headers and a request body.

Response body

This is very similar to the format described for the Request body.

The top-level is a JSON object, presenting a set of name/value pairs (also termed a "dictionary"). There is a JSON array called outputs that is used to denote all the output records to pass back to the Introduction to XConnector.

The JSON should follow the format prescribed in Execute API (v4) Execute API (v4), however note that for XConnector, only the JSON array format is supported.

Data type serialization is described in Execute API (v3) request_data.

The response body from the remote service encapsulates the serialized data, adhering to the following JSON structure:

{
  "outputs": [
    // ... array of the returned results
  ],
  "warnings": [
    // ... array of warnings
  ],
  "errors": [
    // ... array of errors
  ],
  "process_time": [
    // ... array of single values of the computation time for each record
  ]
}

Outputs

outputs is a JSON array, where each element is a JSON array that represents one record of the response data. Outputs should follow the same structure denoted above for the Request body.

• The first record defines the string keys.

• Subsequent arrays can be returned corresponding to the processing of multiple records.

{
  "outputs": [
    [
      "spider", // returns string
      "Australia", // returns string
      "specimen" // returns table
    ],
    [
      "Spiders, with eight legs, spin silk webs to catch prey",
      "Australia is a vast island continent renowned for its diverse landscapes, unique wildlife, and vibrant cities.",
      [ // table rows
        [ "name", "poisonous"],
        [ "Sydney Funnel Web Spider", true],
        [ "Redback Spider", true]
      ]
    ]
  ]
}

Warnings

Warning messages can be returned from the Introduction to XConnector. It should be constructed with a JSON array containing a JSON array for every record of data with a warning. If a record does not have any warnings, return null.

{
  "warnings": [
    null,
    [
      {
        "source_path": "type",
        "message": "Unknown specimen type spader, skept type filter"
      }
    ],
    [
      {
        "source_path": "main_country",
        "message": "Unknown country Astralaya, skept country filter"
      }
    ]
  ]
}

Errors

Errors should be returned in a similar way to Warnings.

{
  "errors": [
    null,
    [
      {
        "error_category": "timeout",
        "error_type": "500",
        "additional_details": "500 Internal Server Error",
        "source_path": "database",
        "message": "Connection timeout"
      }
    ],
    [
      {
        "error_category": "unauthorized",
        "error_type": "401",
        "additional_details": "401 Unauthorized",
        "source_path": "access_management",
        "message": "Unauthorized token"
      }
    ]
  ]
}

Process time

The process time is used to return he amount of calculation time needed for each row of data in milliseconds. We would recommend every successful execution to elapse at least 1 millisecond.

{
  "process_time": [
    10,
    7,
    5
  ]
}

Example 1: Successful response

{
  "outputs": [
    [
      "spider",
      "Australia",
      "specimen"
    ],
    [
      "Spiders, with eight legs, spin silk webs to catch prey",
      "Australia is a vast island continent renowned for its diverse landscapes, unique wildlife, and vibrant cities.",
      [
        [ "name", "poison"],
        [ "Sydney Funnel Web Spider", true],
        [ "Redback Spider", true]
      ]
    ]
  ],
  "warnings": [ null ],
  "errors": [ null ],
  "process_time": [ 1 ]
}

Example 2: Response with warning

{
  "outputs": [
    [
      "specimen"
    ],
    [
      [
        [ "name", "poisonous"],
        [ "Cobra", true],
        [ "Black Mamba", true],
        [ "Python", false],
        [ "Redback Spider", true],
        [ "Tarantula", false],
        [ "Viper", true],
        [ "Black Widow", true],
        [ "Boa Constrictor", false],
        [ "Wolf Spider", false],
        [ "King Cobra", true],
        [ "Sydney Funnel Web Spider", true]
      ]
    ]
  ],
  "warnings": [
    [
      {
        "source_path": "type",
        "message": "Unknown specimen type spader, skept type filter"
      },
      {
        "source_path": "main_country",
        "message": "Unknown country Astralaya, skept country filter"
      }
    ]
  ],
  "errors": [ null ],
  "process_time": [ 1 ]
}

Example 3: Unsuccessful response

{
  "outputs": [],
  "warnings": [],
  "errors": [
    [
      {
        "error_category": "validation_error",
        "error_type": "500",
        "additional_details": "500 Internal Server Error",
        "source_path": "database",
        "message": "Connection timeout"
      } 
    ]
  ],
  "process_time": [ 1 ]
}