Using data from data sources

Introduction §

In this guide we will explain how to find, get access to and use data from data sources published on an IOXIO data sharing service. We will be showing the process on the IOXIO Sandbox Dataspace , but this guide also works for most parts on for example IOXIO.io .

Prerequisites §

• Knowing what kind of data you want to access/use.
• An account and a group on one of the data sharing services, such as IOXIO Sandbox Dataspace or IOXIO.io .
• To access the data, you need to request access for your group. If you don’t yet know which data, you can find instructions to identify the source below, but you cannot connect to it before your group has been granted access.

Finding a data source §

The first step is to find a data source providing the kind of data you want to use. You can browse all available data sources by going to the Data sources section in the menu.

Search for the data relevant to you, e.g. using search words, or tags. In this example we’ve searched for data related to weather and will be using a source for the Meteorology/Weather definition.

To view all data sources available for it, open the details from the blue arrow next to it.

You will be presented with a list of all data sources available for different versions of that definition. On the right side, you will also find links to the OpenAPI Spec for the data product definition, as well as links to view it in Swagger UI or ReDoc.

In this case there’s a source available from the ioxio_dataspace_guides group. You can view the details by pressing the Open button.

Requesting access to the data source §

On IOXIO.io all data sources are set to require API tokens, in other data sharing services the data source may or may not require them. This particular source on the IOXIO Sandbox Dataspace requires API tokens. In order to be able to generate API tokens you need access control keys for the data source. You can for example contact the Technical contact by email and request access to the data source. In order for the access to be granted you will need to provide at least:

• The details of the data source you want to access, the simplest way is to mention the Data source identifier for the source, since it includes in a condensed form the needed details like the domain of the data sharing service (for example ioxio.io) the Data definition (Meteorology/Weather_v0.1), and the Data source name (ioxio_dataspace_guides).
• The name of the group you want the access to be granted to. This should be one of your groups found on the My groups page.

The data source owner might also have other conditions for granting access.

In many cases the access to the data source is taken care of beforehand as part of the coordination work for the project.

If you’re interested in trying out bidirectional communication yourself, you might also want to check out the Building a data source guide.

Access control keys and API tokens §

The access control system consisted of two parts:

• Access control keys: Each group, which is granted access to a data source, gets their own access control keys for the data source. These are long-lived keys that are shared between the IOXIO data sharing service and the group who has access to the source. These are used only for generating short lived API tokens for one particular data source.

  It’s important that these keys are kept well protected and they should only be used in server-to-server connections with the IOXIO data sharing service. They should not be shared with anyone else, not even with the owners of the data source. If they for any reason leak, or there’s a suspicion of it, they can easily be rotated by the click of a button.

• API tokens: API tokens are short-lived, typically valid for an hour from being generated, and used in the requests to fetch data products from a data source. They are validated by the IOXIO data sharing service (when the Dataspace verified API tokens option is used for the data source) and optionally validated by the data source.

  These can be treated as opaque strings for the practical matters of the group requesting data. Technically they are actually JWTs signed by the data sharing service. They prove that the holder of the token, for a limited time, has the right to request data from a particular source on behalf of a the group that generated them. These tokens can not be invalidated or revoked but automatically expire in a short time.

  They are forwarded by the IOXIO data sharing service to the data source so that it can (if it wants to) verify that the request came from the IOXIO data sharing service. The data source can also verify which group is requesting data and might also implement its own access control to restrict access based on that.

  API tokens should also be kept safe and only used in server-to-server connections as they can be used to request any data on behalf of the requesting group from the data source.

Using access control keys §

You can view all the access control keys for all the groups you are a member of in the Access control keys section:

On the Access control keys page you will find a listing of the data sources that each of your groups have access to. If you see nothing on this page, you’ve not been granted access yet, and you should go back to requesting access and wait for it to be granted before continuing.

Press the View button to see the access control keys for the source:

The page displays the two long term access control keys, together with buttons to reveal the key, rotate the key and copy the key.

The Show cURL example button will show you how to make a request to get a short-lived API token using one of the two access control keys:

The example here has all the necessary details pre-configured for this data source. It’s has filled in:

• The URL for the API token request endpoint on the particular data sharing service
• accessControlKey using one of the two long lasting access control keys (be careful since this shows it in clear text!)
• sub set to your group (the one who was granted access and is requesting the token)
• aud set to the DSI of this particular data source

You can try running the cURL command; if you don’t have cURL installed on your computer you can use the View API token request in Swagger UI link in the Resources sidebar, press the Try it out button and paste in the 3 values mentioned above and run it with the Execute button:

The response contains the field success, which is set to true to indicate all was OK, the apiToken, information on when it expires (expiresAt) as well as the aud and sub you submitted.

For your real integration you will want to automate this process, as the generated API tokens are valid for a somewhat short time (currently one hour at a time, but might be subject to change). You should set up some secure way of storing the access control keys and you might want to also cache or reuse the short lived API tokens if you are doing multiple requests to the same data source within the lifetime of the tokens.

To make it easier to test fetching data, the access control keys page also has a Generate API token button, that will use one of your access control keys to generate a fresh API token that you can easily copy & paste.

Requesting data §

The details page of the access control keys tell you the main parts of making the request:

I.e. it tells you that you need to make a POST request and include a valid API token in the X-API-Key header. The URL consists of the address to the product gateway for the particular data sharing service (for example https://gateway.ioxio.io or https://gateway.sandbox.ioxio-dataspace.com), the path consists of the name of the data definition, including the version number (for example /Meteorology/Weather_v0.1) and the name of the particular source is used as a query parameter (for example ?source=ioxio_dataspace_guides:test).

The format of the JSON payload to be sent to the data source in the request varies per data definition. There are multiple ways to check the format. You can check it in Swagger UI, ReDoc or check the OpenAPI Spec version of the definition.

We will here be using Swagger UI, as it also allows us to easily make a test request to the data source. Copy the generated API token and take note of the name of the source mentioned below the field of the generated token and open the Swagger UI link below the field.

The Swagger UI should open with the correct definition by default when opened from that link. Under the Request body heading you will see an Example Value and you can switch to the Schema to see more details of the expected fields, such as descriptions, limits etc.

Based on this, we now know what data we need to supply in the request.

To make a test request directly from Swagger UI, press the Try it out button:

Fill in the name of the source in the source query parameter, paste in the API token in the x-api-key header, do any necessary changes to the Request body and finally press the Execute button to make the request to the data source.

Below the Execute button you should now see the cURL version of the request, as well as the actual response:

You should now have successfully requested data from the data source, congratulations!

Below the response you will also find documentation of all the possible responses, incl. errors, by response code and can view the schema of them as well. You might also want to consider viewing the definition in the ReDoc interface or using the OpenAPI Spec file, possibly together with some tools or libraries.

Conclusion §

You should now have the necessary knowledge to find suitable data sources, request access to them if needed and use the access control keys to get short-lived API tokens that are included in the X-API-Key header.

You should also know how to check the details of the definitions to understand what data you need to send and the structure and meaning of the data you are receiving.

Next steps §

We recommend structuring your code so that using an alternative source only requires changing configuration for the name of the source and the access control key for it.

You might want to check out the IOXIO Dataspace example app to see an example of an app that is fetching data products, as well as handling the access control keys and API tokens.