You must be able to access the server in another way in order to do this. Mine was a VM so I was able to get into a console session through VMware vSphere.
Change the owner to your user name.
For good measure, choose Replace owner on subcontainers and objects under your name and Replace all child object permission entries with inheritable permission entries from this object
Now you can delete the key. After a restart you should be able to access your server via remote desktop again.