Curious (Clojure) Programmer: One SSH Key to rule them all: Or how to make sure that one and only one key is used (even if you use 1Password SSH agent and ProxyJump)

Table of Contents

Context
What keys are actually used by an ssh client when connecting to the remote server
More gotchas
- Default keys and ssh_config keys
- ProxyJump and IdentitiesOnly
Tips and Tricks
References

Persuading your ssh client to use one and only one SSH key may be tricky. In any case, it may help to understand what keys/identities are actually used when you connect to the remote host.

Context

This year, I switched from using native SSH agent running on macOS to the 1Password SSH agent.

The 1Password SSH agent uses the SSH keys you have saved in 1Password to seamlessly integrate with your Git and SSH workflows. It authenticates your Git and SSH clients without those clients ever being able to read your private key.
In fact, your private key never even leaves the 1Password app. The SSH agent works with the SSH keys stored in 1Password, but never without your consent. Only SSH clients you explicitly authorize will be able to use your SSH keys until 1Password locks.

This, I thought, was great. However, it comes with some trade-offs and gotchas to be aware off.

What keys are actually used by an ssh client when connecting to the remote server

Let’s look at what happens when you try to connect to a server by using the -v flag:

ssh -v my-remote-server
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/jumar/.ssh/config
debug1: Reading configuration data /Users/jumar/.orbstack/ssh/config
debug1: /Users/jumar/.ssh/config line 6: Applying options for *
debug1: /Users/jumar/.ssh/config line 34: Applying options for my-remote-server
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 188.34.187.204 [188.34.187.204] port 22.
debug1: Connection established.
debug1: identity file /Users/jumar/.ssh/hetzner_id_ed25519.pub type -1
debug1: identity file /Users/jumar/.ssh/hetzner_id_ed25519.pub-cert type -1
...
debug1: Host '188.34.187.204' is known and matches the ED25519 host key.
debug1: Found key in /Users/jumar/.ssh/known_hosts:144
...
debug1: get_agent_identities: agent returned 9 keys
debug1: Will attempt key: ABC1 ED25519 SHA256:pAQNf578V9kGteiSDwrkCE3n4pVcpJcXNAe4r8njn2Q agent
debug1: Will attempt key: ABC2 ED25519 SHA256:YJXp5hicdCKD/HSyhK0g3vhq1j7GbYXolX1qioBoNz4 agent
debug1: Will attempt key: ABC3 ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c agent
debug1: Will attempt key: ABC4 ED25519 SHA256:Ir9UnK66tGuprvsZ6ZUWxUdiYyiMlfDPOR27X2zbFNI agent
debug1: Will attempt key: ABC5 ED25519 SHA256:40qdht50McdlYVC+QB9MgzgHHXD7yt4XHvSPZ7LVMRg agent
debug1: Will attempt key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
debug1: Will attempt key: BCD1 RSA SHA256:1nomqTYG7NMREQ0EAGPVZR//QOSE94qzZAbpRlLMnOA agent
debug1: Will attempt key: BCD2 RSA SHA256:OrUpsXrGBI03KjcfsCxB6VJCsCXtIqZmu4CYI89bu6w agent
debug1: Will attempt key: BCD3 RSA SHA256:DVe29VP1j0ETf2gQ7Bmz0fWd7D/eMGX0ATwWQw6OU+w agent
debug1: Will attempt key: /Users/jumar/.ssh/hetzner_id_ed25519.pub  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,[email protected],ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected]>
debug1: kex_input_ext_info: [email protected]=(0)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: ABC1 ED25519 SHA256:pAQNf578V9kGteiSDwrkCE3n4pVcpJcXNAe4r8njn2Q agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC2 ED25519 SHA256:YJXp5hicdCKD/HSyhK0g3vhq1j7GbYXolX1qioBoNz4 agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC3 ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC4 ED25519 SHA256:Ir9UnK66tGuprvsZ6ZUWxUdiYyiMlfDPOR27X2zbFNI agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC5 ED25519 SHA256:40qdht50McdlYVC+QB9MgzgHHXD7yt4XHvSPZ7LVMRg agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
debug1: Server accepts key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
Authenticated to 188.34.187.204 ([188.34.187.204]:22) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: client_input_hostkeys: searching /Users/jumar/.ssh/known_hosts for 188.34.187.204 / (none)
debug1: client_input_hostkeys: searching /Users/jumar/.ssh/known_hosts2 for 188.34.187.204 / (none)
debug1: client_input_hostkeys: hostkeys file /Users/jumar/.ssh/known_hosts2 does not exist
debug1: client_input_hostkeys: no new or deprecated keys from server
debug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LC_ALL = "en_US.UTF-8"
debug1: channel 0: setting env LC_TERMINAL = "iTerm2"
debug1: channel 0: setting env LC_TERMINAL_VERSION = "3.5.0beta11"
Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-73-generic aarch64)

`~/.ssh/config`

This is the relevant part of my ssh config file

Host *
»·# Use 1Password SSH agent: https://developer.1password.com/docs/ssh/get-started
  IdentityAgent "~/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock"

Host hetzner-cloud-devel
  Hostname 188.34.187.204
  User root
  IdentityFile ~/.ssh/hetzner_id_ed25519.pub

Why so many keys?

Notice this section where the ssh command shows all the keys that are considered:

debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: ABC1 ED25519 SHA256:pAQNf578V9kGteiSDwrkCE3n4pVcpJcXNAe4r8njn2Q agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC2 ED25519 SHA256:YJXp5hicdCKD/HSyhK0g3vhq1j7GbYXolX1qioBoNz4 agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC3 ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC4 ED25519 SHA256:Ir9UnK66tGuprvsZ6ZUWxUdiYyiMlfDPOR27X2zbFNI agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC5 ED25519 SHA256:40qdht50McdlYVC+QB9MgzgHHXD7yt4XHvSPZ7LVMRg agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
debug1: Server accepts key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
Authenticated to 188.34.187.204 ([188.34.187.204]:22) using "publickey".

It seems, I got very lucky - one more key and it would be rejected with Too many authentication failures error.

WHY? Because of MaxAuthTries:

man sshd_config
...
     MaxAuthTries
             Specifies the maximum number of authentication attempts permitted per connection.  Once the number of failures reaches half this value, additional failures are logged.  The default is 6.

Wait?! Didn’t we specify the ssh key that should be used explicitly?

When you looked at the ssh config shown above, you may have noticed there’s IdentityFile directive for hetzner-cloud-devel host.

Host hetzner-cloud-devel
...
  IdentityFile ~/.ssh/hetzner_id_ed25519.pub

But why is ssh then using all the other keys too?

This is a well-know problem/feature. Even if you specify they key explicitly (in the config file or via the -i command line option), SSH will still attempt to use all the keys stored in the SSH agent (if you use an agent). To force the usage of a specific key (only), you need to also use the IdentitiesOnly=yes option: https://superuser.com/questions/772660/howto-force-ssh-to-use-a-specific-private-key

Host hetzner-cloud-devel
...
  IdentityFile ~/.ssh/hetzner_id_ed25519.pub
  IdentitiesOnly yes

Or on the command line:

ssh -o "IdentitiesOnly=yes" -i <private key filename> <hostname>

More gotchas

Default keys and ssh_config keys

Even with ssh -o IdentitiesOnly=yes, you may still be surprised to see more keys being offered than you expected. Checking the docs for the IdentitiesOnly option in the manual page for ssh_config:

man ssh_config
...
  IdentitiesOnly
         Specifies that ssh(1) should only use the configured authentication identity and certificate files (either the default files, or those explicitly config‐
         ured in the ssh_config files or passed on the ssh(1) command-line), even if ssh-agent(1) or a PKCS11Provider or SecurityKeyProvider offers more identi‐
         ties.  The argument to this keyword must be yes or no (the default).  This option is intended for situations where ssh-agent offers many different identi‐
         ties.

Notice "should only use the configured authentication identity and certificate files (either the default files, or those explicitly configured in the ssh_config files or passed on the ssh(1) command-line)".

If you really don’t want to use the key(s) specified in the ssh config file, you can use ! for hosts that should be excluded: https://superuser.com/questions/859661/how-can-i-force-ssh-to-ignore-the-identityfile-listed-in-host-for-one-specif/859719#859719

Host * !special1 !special2
IdentityFile etc...

ProxyJump and IdentitiesOnly

So we have finally solved the "Too many authentication failures" error, right? Not so fast!

Imagine you need to connect from your laptop A (outside a private network) to a server C inside a private company network, that is only accessible through an intermediate "jump server" B which is on a public network (Internet).

What happens?

ssh -v -o IdentitiesOnly=yes -i ~/.ssh/my.private.key.pem -J [email protected] [email protected]
...
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l ec2-user -v -W '[%h]:%p' @my.jump.server.B
...
debug1: Executing proxy command: exec ssh -l ec2-user -v -W '[]:22' my.jump.server.B
debug1: identity file /Users/jumar/.ssh/my.private.key.pem type -1
...
debug1: Connecting to my.jump.server.B port 22.
debug1: Connection established.
...
debug1: get_agent_identities: agent returned 9 keys
debug1: Will attempt key: ABC1 ED25519 SHA256:pAQNf578V9kGteiSDwrkCE3n4pVcpJcXNAe4r8njn2Q agent
debug1: Will attempt key: ABC2 ED25519 SHA256:YJXp5hicdCKD/HSyhK0g3vhq1j7GbYXolX1qioBoNz4 agent
debug1: Will attempt key: ABC3 ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c agent
debug1: Will attempt key: ABC4 ED25519 SHA256:Ir9UnK66tGuprvsZ6ZUWxUdiYyiMlfDPOR27X2zbFNI agent
debug1: Will attempt key: ABC5 ED25519 SHA256:40qdht50McdlYVC+QB9MgzgHHXD7yt4XHvSPZ7LVMRg agent
debug1: Will attempt key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent
debug1: Will attempt key: BCD1 RSA SHA256:1nomqTYG7NMREQ0EAGPVZR//QOSE94qzZAbpRlLMnOA agent
debug1: Will attempt key: BCD2 RSA SHA256:OrUpsXrGBI03KjcfsCxB6VJCsCXtIqZmu4CYI89bu6w agent
debug1: Will attempt key: BCD3 RSA SHA256:DVe29VP1j0ETf2gQ7Bmz0fWd7D/eMGX0ATwWQw6OU+w agent

debug1: Will attempt key: /Users/jumar/.ssh/id_rsa
debug1: Will attempt key: /Users/jumar/.ssh/id_ecdsa
debug1: Will attempt key: /Users/jumar/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/jumar/.ssh/id_ed25519 ED25519 SHA256:tfqPDAez/t/ipw7WwzPHh4KYhDP4n5sZmjp7gD33BEE
debug1: Will attempt key: /Users/jumar/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/jumar/.ssh/id_xmss
debug1: Will attempt key: /Users/jumar/.ssh/id_dsa

debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: ABC1 ED25519 SHA256:pAQNf578V9kGteiSDwrkCE3n4pVcpJcXNAe4r8njn2Q agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC2 ED25519 SHA256:YJXp5hicdCKD/HSyhK0g3vhq1j7GbYXolX1qioBoNz4 agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC3 ED25519 SHA256:As31qZ4Rff4WbHnS6nikN84c+FRxbMERDnvYIgexE8c agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC4 ED25519 SHA256:Ir9UnK66tGuprvsZ6ZUWxUdiYyiMlfDPOR27X2zbFNI agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC5 ED25519 SHA256:40qdht50McdlYVC+QB9MgzgHHXD7yt4XHvSPZ7LVMRg agent
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: ABC6 ED25519 SHA256:2lfJ436WJ0Ds0ieG4gQM0l5itZu485gh/ELDvtuC1Qw agent

Received disconnect from my.jump.server.B port 22:2: Too many authentication failures
Disconnected from my.jump.server.B port 22
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Ah, crap!

It’s now trying again all the keys hold by the SSH agent (at least, when I’m using the 1Password agent). What’s worse, it’s also supposed to try all the keys found in the default ~/.ssh location; in my case there are 7 more keys but they are never tried because it reaches "Too many authentication failures" (6) before it can do that.

Fortunately, there’s still a solution, although more verbose: https://superuser.com/questions/1662242/use-specific-key-to-connect-to-jump-host-without-modifying-ssh-config-file The trick is to specify ProxyCommand manually and include -o IdentitiesOnly=yes -i … twice.

ssh -v -o ProxyCommand="ssh -o IdentitiesOnly=yes -i ~/.ssh/my.private.key.pem -W %h:%p [email protected]" -o IdentitiesOnly=yes -i ~/.ssh/my.private.key.pem ec2-user@"my.private.server.C"
...
debug1: Executing proxy command: exec ssh -o IdentitiesOnly=yes -i ~/.ssh/my.private.key.pem -W my.private.server.C:22 [email protected]
debug1: identity file /Users/jumar/.ssh/my.private.key.pem type -1
...
debug1: get_agent_identities: agent returned 9 keys
debug1: Will attempt key: /Users/jumar/.ssh/my.private.key.pem  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/jumar/.ssh/my.private.key.pem
Authenticated to my.private.server.C (via proxy) using "publickey".
Last login: ...

Notice "agent returned 9 keys" - but the ssh client didn’t attempt to try them at all; it only used the key passed explicitly via -i.

You can compare the explicit ProxyCommand to the implicit one auto-generated before when we used -J:

debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l ec2-user -v -W '[%h]:%p' @my.jump.server.B

I think the original trigger to try this "explicit ProxyCommand" option was ssh inline proxy jump using keys is not working

if one of the hosts doesn’t like the fact that multiple keys are being tried, you will need to translate the -J into long-form ProxyCommand

Tips and Tricks

Restarting SSH agent

You may need to restart the ssh agent, 1Password, or the whole OS to make sure all the changes are effective!

How to restart ssh service on macOS?

sudo launchctl unload /System/Library/LaunchDaemons/ssh.plist
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

Listing ssh keys with 1Password agent

Standard ssh-add -[lL] doesn’t work out of the box. You also need to set SSH_AUTH_SOCK variable: https://developer.1password.com/docs/ssh/agent/config/#learn-more

SSH_AUTH_SOCK=~/Library/Group\ Containers/2BUA8C4S2C.com.1password/t/agent.sock ssh-add -l

References

1Password SSH agent
- 1Password SSH agent
- Turn on the 1Password SSH agent.
Can’t SSH? You Might Have Too Many Keys
- Mentions MaxAuthTries setting in /etc/ssh/sshd_config
Listing SSH keys
- How to list keys added to ssh-agent with ssh-add?
- Listing SSH keys with 1Password agent
Display full settings of sshd
Howto force ssh to use a specific private key?
How can I force SSH to ignore the IdentityFile listed in "Host *" for one specific host?
- 'IdentitiesOnly' disables prompting ssh-agent, but still offers defaults and ssh_config’d keys.
How to restart ssh service on macOS?
Turn off the SSH Agent
ssh inline proxy jump using keys is not working
- if one of the hosts doesn’t like the fact that multiple keys are being tried, you will need to translate the -J into long-form ProxyCommand