Ctrlk
About CoherentAPI referenceSupportYouTube
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Secure XConnector access with OAuth 2.0 tokens

This feature is only enabled on certain environments and tenants. Contact support for more information about this feature.

For the implementation of XConnector, we recommend implementing security for both the proxy service and remote service using OAuth 2.0 JWT (JSON web token) access tokens issued by your Spark tenant from Keycloak. While your organization may leverage various modern identity providers that support OIDC or SAML for user authentication, the access token from Keycloak plays a central role in the security for XConnector.

After reading this section, you should be able to understand:

  • How the authentication flow can work via the proxy service or a direct call to the remote service.

  • The essential components in the Keycloak JWT needed to validate authentication.

Options for securing the XConnector implementation

In the Architecture and information flow, the proxy service is an optional component that can be used to facilitate the routing of XConnector calls to the remote service.

Implementation
Actions

If the remote service is meant to be publicly available without authentication.

The steps in this article are not required.

If you have implemented a proxy service for your XConnector implementation.

Implement security at the proxy service and to review Option 1: Secure the proxy service.

If the XConnector has been implemented as a direct call to the remote service.

Implement security at the remote service and to review Option 2: Secure the remote service.

Option 1: Secure the proxy service

It is crucial to safeguard your proxy service endpoints unless the remote services are intended for public access.

Spark utilizes credential-less API integration objects for authentication to the proxy service endpoint. These integrations segregate responsibilities between administrators and users, enabling administrators to establish a trust policy between Spark and the cloud provider via the provider's native authentication and authorization mechanism. Upon Spark's connection to the cloud provider, authentication and authorization are facilitated through this trust policy.

Additionally, administrators can specify an allowed list of endpoints accessible by the API integration object, restricting Spark's access to specific proxy services and resources. This capability empowers administrators to enforce organizational policies governing data egress and ingress.

Proxy service authentication flow

  1. Spark XConnector sends a request to the proxy service. The Authorization request header should include the bearer token.

  2. The proxy service validates the access token's signature against the your Keycloak tenant OpenID Connect metadata document, using predefined authentication in the validation policy.

  3. After successful validation, the proxy service forwards the request to the remote service using the remote service's API key or OAuth2 client credentials access token.

Example: Secure end-to-end data consumption from Azure Data Lake through Azure API Management

Diagram of Spark XConnector call to the function app.

  1. A client application initiates a request to the Spark XConnector. The Authorization request header should include the bearer token.

  2. Spark verifies the access token's authenticity by referencing the Keycloak tenant OpenID Connect Metadata document.

  3. Upon successful validation, Spark transfers the request to Azure API Management, functioning as the proxy service.

  4. Within Azure API Management:

    1. The initial step involves validating the request's IP address against the predefined range outlined in the ip-filter section of the validation policy.

    2. Following this, it authenticates the access token's signature using the Keycloak OpenID Connect metadata document, also specified in the same policy.

  5. Post-validation, the Azure API Management retrieves the API key for the Azure Function App (serving as the remote service) from Azure Key Vault, utilizing a passwordless managed system identity.

  6. The API key from step 5 is incorporated as the value of the x-functions-key header in the request. Subsequently, the Azure API Management solution forwards the request to the Azure Function App's HTTP Trigger.

  7. The Azure Function App's network security group confirms the request's IP address matches the Azure API Management (per the network security group rule) and validates the presence of the API key in the x-functions-key request header. Upon meeting all criteria, the Azure Function App employs the Azure Storage Files Data Lake client library for .NET to access data from Azure Data Lake, leveraging a passwordless managed system identity.

  8. The Azure Function App transfers data from the Azure Data Lake to Spark.

  9. Spark formats the retrieved data and delivers it back to the client application.

Option 2: Secure the remote service

Use of the proxy service is optional. Spark XConnector can also be set up directly to access the remote service.

Remote service authentication flow

  1. Spark XConnector sends a request to the remote service. The Authorization request header should include the bearer token. The request body should conform to Remote service request and response format.

  2. The remote service validates the access token's signature against the your Keycloak tenant OpenID Connect metadata document, using the custom validation functionality. Upon successful validation, the request is authenticated.

Example: Secure end-to-end data consumption from Azure Data Lake through direct call to the remote service

Spark XConnector can be used to securely provide access to data from Azure Data Lake without the need for a proxy service.

  1. A client application initiates a request to the Spark XConnector. The Authorization request header should include the bearer token.

  2. Spark verifies the access token's authenticity by referencing the Keycloak tenant OpenID Connect Metadata document.

  3. Upon successful validation, Spark transfers the request to the Azure Function App, functioning as the remote service.

  4. The Azure Function App's network security group validates the request's IP address corresponds to the Spark tenant. The Azure Function App employs custom validation functionality to authenticate the access token's signature using the Keycloak OpenID Connect metadata document.

  5. Upon meeting all criteria, the Azure Function App employs the Azure Storage Files Data Lake client library for .NET to access data from Azure Data Lake, leveraging a passwordless managed system identity.

  6. The Azure Function App transfers data from Azure Data Lake to Spark.

  7. Spark formats the retrieved data and delivers it back to the client application.

Keycloak access token validation

Sample Keycloak JWT access token (abbreviated)

From the Keycloak access token, your proxy service or custom token remote service must validate:

  • Payload claims

    Claim
    Description
    Required

    iss

    Identifies the token issuer, allowing the recipient to verify the token's authenticity and origin by matching it with the expected issuer.

    Yes

    aud

    Identifies the intended audience for the token, specifying who the token is intended for. From Spark the value is product-factory.

    No, but recommended

    scope

    Identifies the permissions or access rights granted to the bearer of the token.

    No, but recommended to use open id offline_access spark profile

  • Signature: In a JWT issued by Keycloak, the signature refers to the cryptographic signature added to the token to ensure its integrity and authenticity. It is validated using the public key found in the OpenID Connect metadata document published by Keycloak. This validation process helps confirm that the JWT has not been tampered with and can be trusted.

Last updated