# Remote service input and output data formats When Spark communicates with a remote service - sending or receiving data - it's crucial to adhere to the proper data formats to ensure the external code is executed and returned successfully. When developing the remote service, it is important to conform to the input and output data requirements. The design of the input and output formats aligns with the definitions in the aligns with [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md) "JSON array" [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md#declare-inputs-using-json-arrays). If you are using a [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#proxy-service) to intermediate the `XConnector`, it would need to be able to reformat the inputs and outputs to conform to the requirements of the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#client-application) and [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service). After reading this section, you should be able to understand the data formats that the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service) must follow to accept and return requests. ## Request format from Spark XConnector to Remote service
This section describes the format of the `HTTP` request that is being sent from the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#spark-xconnector) to the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#proxy-service) and [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service). This is important because the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service) needs to be able to consume the input data to process the result of the code execution. The `HTTP` request from Spark takes the form of a `POST` request containing headers and a request body. ### Request headers The header information typically appears to the remote service as a collection of key value pairs. Usually the `HTTP` request header will incorporate `Authorization` key with a Bearer token. ### Request body The top-level is a JSON object, presenting a set of name/value pairs (also termed a "dictionary"). There is a JSON array called `inputs` that is used to denote all the input records to pass to the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service). The JSON should follow the format prescribed in [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md) [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md#api-request), however note that for `XConnector`, only the JSON array format is supported. Data type serialization is described in [Execute API (v3)](/spark-apis/execute-api/execute-api-v3.md) [Execute API (v3)](/spark-apis/execute-api/execute-api-v3.md#request_data). ### Example 1: Single value inputs * Within `inputs`, the first record `[ "key1", "key2" ]` defines the string keys. * Subsequent arrays are used to define the values for the subsequent input records.
DataInput format
gP9iRaleAcio-MHpRkbvLxbZY
{
  "inputs": [
    [ "key1", "key2" ],
    [ "value11", "value21" ],
    [ "value12", "value22" ],
    [ "value13", "value23" ],
  ]
}
### Example 2: Tables of inputs * Within `inputs`, the first record `[ "Table" ]` defines the string keys. * Subsequent arrays are used to define the values for the subsequent input records.
DataInput format
zuxt33WEgjC8-MHpRkbvLxbZY
{
   "inputs": [
    [ "Table1" ],
    [ 
      [ [ "TableKeyA", "TableKeyB" ], 
        [ "ValueA1", "ValueB1" ],
        [ "ValueA2", "ValueB2" ],
        [ "ValueA3", "ValueB3" ]
      ]
    ]
  ]
}
### Example 3: Multiple records of single values and tables
DataInput format
{
  "inputs": [
    [ "Course", "Semester", "Students" ],
    [ "ECON101", "2022 Spring", 
      [ ["SID", "Grade"],
        [ 123, 90 ],
        [ 234, 76 ],
        [ 345, 54 ]
      ]
    ],
    [ "ECON102", "2023 Fall",
      [ ["SID", "Grade"],
         [ 234, 85 ],
         [ 567, 72 ],
         [ 890, 97 ]
      ]
    ],
    [ "STAT101" , "2023 Fall",
      [ ["SID", "Grade"],
         [ 234, 77 ],
         [ 456, 80 ],
         [ 678, 91 ]
      ]
    ]
  ]
}
## Response format from the Remote service to Spark XConnector
This section describes the format of the `HTTP` response that is returned from the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service) to the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#spark-xconnector). This is important because the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service) needs to be able to return the appropriate data as the result of the code execution. The `HTTP` response from the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service) should take the form of a `POST` request containing headers and a request body. ### Response body This is very similar to the format described for the [#request-body](#request-body "mention"). The top-level is a JSON object, presenting a set of name/value pairs (also termed a "dictionary"). There is a JSON array called `outputs` that is used to denote all the output records to pass back to the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#spark-xconnector). The JSON should follow the format prescribed in [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md) [Execute API (v4)](/spark-apis/execute-api/execute-api-v4.md#api-response), however note that for `XConnector`, only the JSON array format is supported. Data type serialization is described in [Execute API (v3)](/spark-apis/execute-api/execute-api-v3.md) [Execute API (v3)](/spark-apis/execute-api/execute-api-v3.md#request_data). The response body from the remote service encapsulates the serialized data, adhering to the following JSON structure: ```json { "outputs": [ // ... array of the returned results ], "warnings": [ // ... array of warnings ], "errors": [ // ... array of errors ], "process_time": [ // ... array of single values of the computation time for each record ] } ``` #### Outputs `outputs` is a JSON array, where each element is a JSON array that represents one record of the response data. Outputs should follow the same structure denoted above for the [#request-body](#request-body "mention"). * The first record defines the string keys. * Subsequent arrays can be returned corresponding to the processing of multiple records. ```json { "outputs": [ [ "spider", // returns string "Australia", // returns string "specimen" // returns table ], [ "Spiders, with eight legs, spin silk webs to catch prey", "Australia is a vast island continent renowned for its diverse landscapes, unique wildlife, and vibrant cities.", [ // table rows [ "name", "poisonous"], [ "Sydney Funnel Web Spider", true], [ "Redback Spider", true] ] ] ] } ``` #### Warnings Warning messages can be returned from the [Introduction to XConnector](/xconnector/introduction-to-xconnector.md#remote-service). It should be constructed with a JSON array containing a JSON array for every record of data with a warning. If a record does not have any warnings, return `null`. ```json { "warnings": [ null, [ { "source_path": "type", "message": "Unknown specimen type spader, skept type filter" } ], [ { "source_path": "main_country", "message": "Unknown country Astralaya, skept country filter" } ] ] } ``` #### Errors Errors should be returned in a similar way to [#warnings](#warnings "mention"). ```json { "errors": [ null, [ { "error_category": "timeout", "error_type": "500", "additional_details": "500 Internal Server Error", "source_path": "database", "message": "Connection timeout" } ], [ { "error_category": "unauthorized", "error_type": "401", "additional_details": "401 Unauthorized", "source_path": "access_management", "message": "Unauthorized token" } ] ] } ``` #### Process time The process time is used to return he amount of calculation time needed for each row of data in milliseconds. We would recommend every successful execution to elapse at least `1` millisecond. ```json { "process_time": [ 10, 7, 5 ] } ``` ### Example 1: Successful response ```json { "outputs": [ [ "spider", "Australia", "specimen" ], [ "Spiders, with eight legs, spin silk webs to catch prey", "Australia is a vast island continent renowned for its diverse landscapes, unique wildlife, and vibrant cities.", [ [ "name", "poison"], [ "Sydney Funnel Web Spider", true], [ "Redback Spider", true] ] ] ], "warnings": [ null ], "errors": [ null ], "process_time": [ 1 ] } ``` ### Example 2: Response with warning ```json { "outputs": [ [ "specimen" ], [ [ [ "name", "poisonous"], [ "Cobra", true], [ "Black Mamba", true], [ "Python", false], [ "Redback Spider", true], [ "Tarantula", false], [ "Viper", true], [ "Black Widow", true], [ "Boa Constrictor", false], [ "Wolf Spider", false], [ "King Cobra", true], [ "Sydney Funnel Web Spider", true] ] ] ], "warnings": [ [ { "source_path": "type", "message": "Unknown specimen type spader, skept type filter" }, { "source_path": "main_country", "message": "Unknown country Astralaya, skept country filter" } ] ], "errors": [ null ], "process_time": [ 1 ] } ``` ### Example 3: Unsuccessful response ```json { "outputs": [], "warnings": [], "errors": [ [ { "error_category": "validation_error", "error_type": "500", "additional_details": "500 Internal Server Error", "source_path": "database", "message": "Connection timeout" } ] ], "process_time": [ 1 ] } ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.coherent.global/xconnector/remote-service-input-and-output-data-formats.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.