OK!
So I walked into this thinking “Easy, I got this.” Turns out there’s a whole lot more to it than even I thought.
The first issue is that (according to the man pages for OpenSSL, man 3 pem), OpenSSL is expecting the RSA key to be in PKCS#1 format. Clearly, this isn’t what ssh-keygen is working with. You have two options (from searching around).
If you have OpenSSH v. 5.6 or later (I did not on my laptop), you can run this:
ssh-keygen -f key.pub -e -m pem
The longer method of doing this is to break apart your SSH key into its various components (the blog entry I found some of this in accuses OpenSSH of being “proprietary”, I prefer to call it “unique”) and then use an ASN1 library to swap things around.
Fortunately for you, someone wrote the code to do this:
