WTF AWS CLI? "No credentials found in credential_source referenced in profile"

I was recently debugging a bash script that was failing on an EC2 instance. The script was fairly straight forward and was using the AWS CLI to make some changes to the AWS config, however it was failing with the error "Error when retrieving credentials from Ec2InstanceMetadata: No credentials found in credential_source referenced in profile"

The error implies that the correct credentials cannot be found via the Metadata API, right? Apparently not.

Given that the error implies the credentials can't be found via the Metadata API, I started by querying it manually with curl:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/special-role-name

But the role was present and looked to be in order:

{
  "Code" : "Success",
  "LastUpdated" : "2021-04-28T20:55:11Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "SOME SECRET STUFF",
  "SecretAccessKey" : "MORE SECRET STUFF",
  "Token" : "A WHOLE LOT OF SECRET STUFF",
  "Expiration" : "2021-04-29T03:04:24Z"
}

So what gives? Googling the error message only returns 40 something results, and none of them are quite relevant. Weird. 

After an embarrassingly long time and too much fiddling around with ~/.aws/config I discovered there was a debug option for the AWS CLI.

And there it is. The smoking gun:

2021-04-28 21:27:31,601 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 426, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3/connectionpool.py", line 421, in _make_request
  File "http/client.py", line 1321, in getresponse
  File "http/client.py", line 296, in begin
  File "http/client.py", line 257, in _read_status
  File "socket.py", line 589, in readinto
socket.timeout: timed out

'Read timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"'

Ahah! So... it's not that it can't find the credentials. It can't even connect to the API to discover them. ( An issue relating to a HTTP proxy which is a story for another day )

A clearer error message would be much more useful. I've raised a Github Issue on the repo.