If you control your VM access via SSH keys, you will need to add the keys to the following path on Centos/Ubuntu/Rhel

 ~/.ssh/authorized_keys

The key should start with ssh-rsa and continue in a single line with the value of the public key.