Skip to main content

Using pagination

Overview​

Your DataGalaxy modules tend to contain thousands and sometimes millions of objects, depending on the size of your data sources.

Fetching all these objects at once using the DataGalaxy API could potentially cause performance issues and put a strain on the network traffic.

That's why most of the DataGalaxy API endpoints that return lists, such as GET /fields, are paginated.

How to use pagination ?​

limit parameter​

The limit parameter controls the maximum number of objects returned by the API.

Its default value is 20, meaning the API will return 20 results if you do not add the limit parameter to your request.
Its value can be set between 1 and 5000.

Let's take a simple case: you must fetch your Glossary objects but only 50 at a time.
In this case, you would use the limit parameter with a value of 50.

cURL request
curl -L -X GET 'https://yourapi.datagalaxy.com/v2/usages?versionId={versionId}&limit=50' -H 'Authorization: Bearer {token}'

page parameter​

The page parameter helps you fetch a specific result page.

By default, its value is 1, it will fetch the first page of the result, and the number of objects returned is determined by the limit parameter.

Now, following the previous section's use case, we can fetch the first 150 results, 50 at a time, using the page parameter:

  • first 50 results : GET /usages?versionId={versionId}&limit=50&page=1
  • results from 50 to 100 : GET /usages?versionId={versionId}&limit=50&page=2
  • results from 100 to 150 : GET /usages?versionId={versionId}&limit=50&page=3

next_page property​

The next_page property is found in the response payload and is a prebuilt URL pointing to the next page.
It saves all the parameters you set in your request.

It will be null if there is no next page.

Response payload
{
    "pages": 5,
    "total": 50,
    "total_sum": 250,
    "next_page": "https://yourapi.datagalaxy.com/v2/usages?versionId={versionId}&limit=50&page=2",
    "results": [ ... ]
}

With this in mind, fetching all your Usage objects can easily be done with the following modus operandi :

  1. Make a request to GET /usages?versionId={versionId}&limit=50 to fetch the first 50 results
  2. Retrieve the next_page property found in the response payload and use it as your next request's URL
  3. Repeat step 2 until next_page is null