More Okta gotchas: "Type mismatch exception. Could not convert parameter 'limit' to the required type."

This error has to do with how Okta implements pagination, which I have ... opinions on, but aren't really relevant to the problem at hand.

When you perform a list operation on an Okta resource, Okta notifies you that there are additional page results via the link header. The link header looks like this:

<https://OKTA_ORG_NAME.OKTA_BASE_URL/api/v1/apps?limit=20>; rel="self", 
<https://OKTA_ORG_NAME.OKTA_BASE_URL/api/v1/apps?after=0oa11gglcgm6HZBUR0h8&limit=20>; rel="next"

The self link refers to the current page of results, and next is the URI for your next page. If the next link is not present, you're at the end of the list.

If you're working in Python, you're probably using the Requests library doing something similar to this:

import json
import requests

def list_apps(org_name, base_url, token, limit=None):
    results = []
    more_pages = True
    params = {'limit':limit} if limit else {}
    headers = {'Authorization': 'SSWS %s' % token}
    apps_url = 'https://%s.%s/api/v1/apps' % (org_name, base_url)
    while more_pages:
        response = requests.get(apps_url, params=params, headers=headers)
        print("%s: %d" % (apps_url, response.status_code))
        if response.status_code >= 300:
            print(response.text)
            break;
        if response.content:
            results.extend(json.loads(response.content))
        more_pages = 'link' in response.headers and 'rel="next"' in response.headers['link']
        if more_pages:
            # not pictured: _next_page() parses the link header and returns the "next" URL
            apps_url = _next_page(response.headers['link'])
    return results

This works great if you don't pass in a limit. However, if you pass in a value for limit, you get this un-useful error:

{
    "errorCode":"E0000037",
    "errorSummary":"Type mismatch exception.  Could not convert parameter 'limit' to the required type.",
    "errorLink":"E0000037",
    "errorId":"REDACTED",
    "errorCauses":[]
}

Challenging assumptions

The error isn't immediately obvious. The Okta documentation explicitly tells you to treat the links in the link header as opaque. However, to solve this error, we have to pay attention to the actual link and understand how URL parameters get parsed by web servers.

As of today (13 October 2021), the next link is simply the API link with a after=... parameter and a limit=... parameter added. The limit will match whatever limit you provided in your original request (or 20 if you did not provide one):

https://OKTA_ORG_NAME.OKTA_BASE_URL/api/v1/apps?after=0oa11gglcgm6HZBUR0h8&limit=50

What do you think happens when you call requests.get(url='https://OKTA_ORG_NAME.OKTA_BASE_URL/api/v1/apps?after=0oa11gglcgm6HZBUR0h8&limit=50', params={'limit':'50'})?

You might see that url already contains limit=50 and ignore params, but instead what happens is that the values in the params map get added to whatever is in url and that's why it breaks.

Here's a demonstration to show you what I mean:

Python 3.9.6 (default, Aug 16 2021, 15:42:07) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> resp1 = requests.get('https://google.com/', params={'q':'foo'})
# PROTIP: Want to see exactly what was sent to the server? the response
# object has a `request` member that lets you see the details for the request that
# generated it! Here we look at the complete URL we tried to access.
>>> resp1.request.url
'https://www.google.com/?q=foo'
>>> resp2 = requests.get(resp1.request.url, params={'q':'foo'})
>>> resp2.request.url
'https://www.google.com/?q=foo&q=foo'

Understanding the error message

Okay, so why does it complain about type conversion? And now we get to how URL parsing works.

In early web days, form data was passed in the query string. The syntax for sending an array is to simply repeat the same key with a different value. So, for example, if you had /foo/bar?items=1&items=2, that would get parsed into an array named items with values [1, 2].

So, in our scenario above, when we specify a limit value and try to follow the next link, Okta sees that limit is [50, 50] instead of 50 and throws the type conversion error.

TL;DR: the fix

We solve this by emptying the params map before we iterate to the next page. Here's a working version of the loop:

    while more_pages:
        response = requests.get(apps_url, params=params, headers=headers)
        print("%s: %d" % (apps_url, response.status_code))
        if response.status_code >= 300:
            print(response.text)
            break;
        if response.content:
            results.extend(json.loads(response.content))
        more_pages = 'link' in response.headers and 'rel="next"' in response.headers['link']
        if more_pages:
            # not pictured: _next_page() parses the link header and returns the "next" URL
            params = {}
            apps_url = _next_page(response.headers['link'])

This does what we want: the next link is followed unaltered, and we get the next page (or a 429 response if we're hitting the API too hard!)

Hope this helps someone out there!