Identity Server 4 WindowsCryptographicException: Keyset does not exist

I get these these two exceptions in Identity Server 4 during a regular login to get a token.
It’s on a load balanced setup with 2 servers, so I’ve done the following:
They both get the same PFX for encrypting tokens. It works just fine on a single server. They have the password for the PFX file, and can seemingly encrypt a token correct correctly.