Making sense of nonsensical errors: Python X509 PEM lib edition

Are you trying to use Requests (or something built on Requests, like HVAC) with a server that uses a TLS certificate issued by a private CA?

Maybe you tried to follow these instructions in the HVAC docs and got a bizzare error like the following:

Traceback (most recent call last):
  File "/home/demo/.pyenv/versions/3.9.6/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
    context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
ssl.SSLError: [X509] PEM lib (_ssl.c:4293)

What the hell does that mean?

First, the quick answer for folx finding this via Google: this particular error means one or more TLS certificates in the bundle.pem you created is probably expired. This was the case in my instance, at least.

We can guess a little bit about the error from where it happened: context.load_verify_locations(). When you set the verify property on a session object to a file path instead of True or False, urllib will try to load any certificates it finds. So, we can infer from the method name that the exception is being thrown while the certificates are being read from disk.

What about PEM lib (_ssl.c:4293)? This message comes from native code, not a Python code. We know the filename (_ssl.c) and the line number (4293). With a little searching around, we can find _ssl.c in the cpython implementation here

Note that depending on your version of Python, the line number is going to vary. The above link is accurate as of Python 3.9.6. If you're getting an error on a different line number, you might need to do some more sleuthing.

I'm not going to go diving into the native code here; for me, knowing the error was coming from the loading routines--not the TLS connection handling routines--told me that the issue was with the certificates themselves.

In my case, what I had done was concatenate a bunch of PEM files. Unfortunately, the directory had a mix of current and expired certificates, and Python barfs if any of the certificates it finds are bad. Which, is kind of annoying (why not just skip it?) and, because it happens at such a low level, it ends up propagating a relatively meaningless error instead of something that would give a hint at the actual problem.

I hope this helps someone out there!