How to: Use Spark XConnector service to connect a remote service

This feature is only enabled on certain environments and tenants. Contact support for more information about this feature.

The process to set up a Spark XConnector service is similar to Call Spark services as functions (Xcall).

Configure the remote service

The sample remote service is publicly available without authentication. Please see Secure XConnector access with OAuth 2.0 tokens for information about secure remote services.

For this guide, a remote service has been configured with an Azure Function called species. The species remote service receives a table of animal types and returns a table of results for the species name, type, and countries.

Sample request

1. Run the curl statement in your application of choice. The request body follows the format in Remote service request format.

2. Remove entries under ["Type"] and run the command again.

3. The list of species returned should change.

curl --location 'https://func-xconnector-sample.azurewebsites.net/api/species' \
--header 'Content-Type: application/json' \
--data '{
  "inputs": [
    ["species"],
    [
      [
        ["Type"],
        ["Bird"],
        ["Mammal"],
        ["Reptile"]
      ]
    ]
  ]
}'

Sample response

The response follows the format Remote service response format.

{
  "outputs": [
    [ "results" ],
    [
      [
        ["name", "type", "countries"],
        ["Spotted Owl", "Bird", "USA,Canada"],
        ...,
        ["Golden Tiger","Mammal","India,Thailand"],
        ...,
        ["Slow Lizard","Reptile","Mexico,Argentina"],
        ...
      ]
    ]
  ],
  "warnings": [
    null
  ],
  "errors": [
    null
  ],
  "process_time": [
    1
  ]
}

The species Azure Function can be built with this C# project.

Coherent.Spark.Demos.XConnector-master.zip

archive · 14KB

Open

Create a downstream Spark service with the remote service call

species.xlsx

23KB

Open

In Call Spark services as functions (Xcall), downstream services contain encapsulated calculations in Data. For XConnector, it only needs to contain the references to the remote service and the input / output specifications.

1. Open the Excel file.

   • The workbook has two tabs.

     • XConnector: Defines the location of the remote service.

     • POST: Define the inputs and outputs to send to the remote service.

2. Review all of the mappings in the Name Manager, either:

   • Select the Formulas tab and select Name Manager OR,

   • Use the keyboard shortcut Ctrl+F3.

     Name	Value	Refers to	Comment
     Xinput_type	...	=POST!$B$2:$B$7	List of the species types. This range needs to be long enough to accept the maximum input length to the remote service.
     Xoutput_results	...	=POST!$E$2:$G$102	List of species received. This range needs to be long enough to accept the maximum output length from the remote service.
     XPROPERTY_DefaultEngineType	XConnector	=XConnector!$B$4	Sets the service type as XConnector in Spark.
     XPROPERTY_XConnectorServiceName	species	=XConnector!$B$3	Name of the remote service within the base URL.
     XPROPERTY_XConnectorTargetUrl	https://func-xconnector-sample.azurewebsites.net/api/	=XConnector!$B$2	Base URL for the remote resource. Must end with /.

3. Note that this Excel workbook contains no formulas. Its purpose is only to define the location and layout of the remote service request and response.

4. Create a folder called xconnectors if it does not exist.

5. Upload the Excel file for the downstream service into the folder.

6. Note that the API Tester and Execute API will not function directly on this service.

Create an upstream Spark service to invoke the downstream service

With Call Spark services as functions (Xcall), any Excel file can have access to the remote service. The upstream service makes the API request and receives the response from the downstream service.

species-xcall.xlsx

27KB

Open

Interact with XConnector in Excel

1. Open the upstream file species-xcall.xlsx in Excel. The file includes a CS.SPARK_XCALL function to xconnectors/species.

2. Load Coherent Assistant and sign in to the same environment and tenant.

3. Review all of the mappings in the Name Manager, either:

   • Select the Formulas tab and select Name Manager OR,

   • Use the keyboard shortcut Ctrl+F3.

     Name	Value	Refers to	Comment
     Xinput_type	...	=Sheet1!$B$2:$B$7	List of the species types. This range needs to be long enough to accept the maximum input length to the remote service.
     Xoutput_count	71	=Sheet1!$T$2	Number of species returned.
     Xoutput_results	...	=Sheet1!$O$2:$Q$102	List of species received. This range needs to be long enough to accept the maximum output length from the remote service.

4. Remove entries under the input type and the species returned will change.

Use the XConnector with an integration

When uploaded to Spark, the Excel file becomes an upstream service with access to the remote service.

1. Open the previously created xconnectors folder.

2. Upload the Excel file for the upstream service into the folder.

3. Use the API Tester to interact with the upstream service species-xcall. For this to work properly, you must add the request_meta xconnector_parse_mode = 1.

   1. In API request (inputs), select Raw view.

   2. In the request_meta, add a comma , to the end of the last entry and then add "xconnector_parse_mode" = 1.

4. Remove entries under the input type and the species returned will change.

5. If you would like to make any changes to the upstream service, edit the Excel file and re-test the CS.SPARK_XCALL function with Coherent Assistant. Upload the Excel file again to update the service.

Two levels of calculations take place:

1. The downstream service calls the remote service to query the species data in the Azure Function.

2. The upstream file receives the response and processes the response into the CS.SPARK_XCALL output template. Any calculations within the upstream file are also calculated, such as the Xoutput_count.

Sample request body

{
  "request_data": {
    "inputs": {
      "type": [
        {
          "type": "bird"
        },
        {
          "type": "reptile"
        }
      ]
    }
  },
  "request_meta": {
    "xconnector_parse_mode": 1
  }
}

Sample response body

HTTP 200 OK Content-Type: application/json

{
  "response_data": {
    "outputs": {
      "count": 24,
      "results": [
        {
          "name": "Spotted Owl",
          "type": "Bird",
          "countries": "USA,Canada"
        },
        ...,
        {
          "name": "Slow Lizard",
          "type": "Reptile",
          "countries": "Mexico,Argentina"
        },
        ...
      ]
    },
    "warnings": null,
    "errors": null
  },
  "response_meta": {
    ...
  }
}

Considerations

• If the request_meta "xconnector_parse_mode" = 1 is omitted, you will encounter issues handling any XConnector services that return tables.

• Although Remote service request and response format specification supports multiple request records and response records at a time, the upstream Spark integration can only process one request at a time. In other words, the length of the outputs, warnings, errors, process_time array should be 2 elements only.

• Array Xcall for scaling is not supported with XConnector.