Debugging a failed ProxyJump chain, hop by hop

ssh db-primary fails. Between you and that host are two bastions, three DNS lookups, three TCP connections, three host keys and three logins. OpenSSH tells you about it in one line. This guide is a method for turning that line into a hop and a step, without hammering a server that counts failed logins.

Five steps, at every hop #

Each hop in a ProxyJump chain goes through the same five steps, in order:

1. DNS. The name is resolved to an address.
2. TCP. A connection is opened to the SSH port.
3. SSH handshake. Both sides exchange version banners and agree on keys.
4. Host key. Your client checks the server’s key against the one it trusts.
5. Login. You authenticate.

A chain of three hosts is fifteen steps. Debugging is finding which one stopped.

One detail trips nearly everyone up: steps 1 and 2 for a hop happen on the hop in front of it. Your laptop resolves and dials the first bastion. The first bastion resolves and dials the second. So “it resolves fine on my laptop” tells you nothing about the target behind a bastion.

First, confirm the config you think you have #

ssh -G db-primary | grep -iE '^(hostname|user|port|proxyjump|identityfile) '

ssh -G prints the settings the client would use after every Host and Match block is applied, without connecting. A surprising share of jump problems end here: a wildcard block that set the wrong user, or a ProxyJump that never matched.

Then test one hop at a time #

Work outwards from your own machine.

ssh bastion true
ssh -J bastion inner-bastion true
ssh -J bastion,inner-bastion db-primary true

The first command that fails names the hop. Add -v to that one command only:

ssh -v -J bastion inner-bastion true

With -J, two ssh processes write debug lines to the same terminal. The lines that end the output belong to the hop that failed.

Read the error #

Two habits make this table work. Read the channel 0 line, not the one after it: a failed channel is always followed by a generic kex_exchange_identification or Connection closed, and that second line is noise. And read the user@host: prefix on a Permission denied: it says which hop refused you.

Test from the hop in front #

If the bastion cannot reach the next host, your laptop cannot help. Ask the bastion.

Does the name resolve there?

ssh bastion getent hosts db-primary.internal

Is the port open from there?

ssh bastion 'timeout 3 bash -c "</dev/tcp/10.0.3.4/22" && echo open || echo closed'

Does something that speaks SSH answer?

ssh bastion 'timeout 3 bash -c "exec 3<>/dev/tcp/10.0.3.4/22; head -n 1 <&3"'

A healthy server prints a banner such as SSH-2.0-OpenSSH_9.6. Silence means a firewall is swallowing packets. Anything else means another service owns the port.

Do not debug by logging in again #

Every failed login counts. MaxAuthTries defaults to six per connection, and fail2ban counts across connections and bans your address after a handful. Retrying with different keys to “see what happens” is how a login problem becomes a lockout.

Use what the failed attempt already told you. In ssh -v output, look for these lines:

debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/alice/.ssh/id_ed25519 ED25519 SHA256:...
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.

The server accepts only public keys, one key was offered, and it was refused. Now the question is precise: is that the key in the account’s authorized_keys, and is the user right? Permission denied (publickey): causes and fixes goes through the list.

If the output shows many Offering public key lines, your agent is spraying keys. Pin one:

Host db-primary
    IdentityFile ~/.ssh/id_ed25519_prod
    IdentitiesOnly yes

“administratively prohibited” #

This one is the bastion’s sshd saying no to forwarding. On the bastion:

sudo sshd -T | grep -iE 'allowtcpforwarding|permitopen'

allowtcpforwarding no or a permitopen list without your target is the answer. If a Match block applies to your user, test as that user:

sudo sshd -T -C user=alice,addr=203.0.113.7 | grep -iE 'allowtcpforwarding|permitopen'

A changed host key in the middle #

With ProxyJump, every hop’s key is checked on your computer, against your own known_hosts. When one changes, find out why before you remove it. A rebuilt server is the common reason. An interception is the other.

ssh-keygen -F 10.0.3.4          # what you trust now
ssh-keygen -R 10.0.3.4          # forget it, once you know why it changed

Compare the new fingerprint with one from the server’s console or your provisioning system, not with the one the connection offers you.

How Hop Doctor does this for you #

This whole method is what Hop Doctor in Gatesys SSH runs when a connect fails. It is part of the free plan.

• It names the hop and the step. The report is a row of checks for every hop: DNS, TCP, SSH, Host key, Login.
• It keeps the bastion’s own words. connect failed: Connection refused and administratively prohibited are evidence in the report, not lines that scrolled away.
• It checks from the hop in front. The checks run inside the failed connect, while the earlier hops are still up: a read-only getent hosts on the bastion, one bare TCP connect to read the greeting. They are capped at three seconds and written to the audit log.
• It never logs in again. It reads the attempt that already failed, so it adds nothing to MaxAuthTries or to fail2ban.
• It offers the fix as a diff. The user, the key, the order of methods, the port or a jump host, shown before and after, with Apply & retry.

With Pro, Explain puts the report into two or three plain sentences. The checks and fixes need no model and no account.

FAQ #

Why does ssh say “kex_exchange_identification: Connection closed by remote host”? #

The TCP path opened but the SSH handshake never started. Through a jump host this usually follows a channel 0: open failed line, which holds the real reason. On a direct connection it points to MaxStartups, a TCP wrapper, a load balancer, or a service on that port that is not SSH.

What does “channel 0: open failed: administratively prohibited” mean? #

The jump host’s SSH server refused to open a forwarding channel. AllowTcpForwarding is off, or PermitOpen does not include the host and port you asked for.

Where is the target’s name resolved when I use ProxyJump? #

On the jump host. Your computer resolves only the first hop. Test with ssh bastion getent hosts name.

How do I see which hop refused my login? #

Read the prefix: deploy@db-primary: Permission denied (publickey) names the account and the host. If there is no prefix, run the hops one at a time as shown above.

Does ssh -v show both hops? #

Yes, interleaved, because -J starts a second ssh process for the jump. Test each hop on its own first and the output is much easier to read.