Following up on SSH through jump hosts

I got interesting comments on my previous post about SSH through jump hosts, that made me think a bit.

The first one of them suggested to use a syntax that unfortunately doesn't work on anything other than zsh, but had the nice idea to add a login for each jump server. Sadly, this only works when there is only one hop, for reasons similar to those we will explore below.

The second one suggested "/" would not be so good a separator, because it would fail in svn+ssh urls and such, which is a good point that I didn't think of only by lack of such use case. The simplicity of the resulting configuration was nice enough.

This decided me to take a shot at implementing something that would allow adding both login and port number for each jump server.

The first implementation detail to set was which characters to use for login, port number and hop separators. I chose respectively "%" (@ is unfortunately impossible to use because ssh would take it, leaving out any hop before the character), ":" (pretty standard) and "+" (not allowed in DNS, and not too illogical as such a separator, I thought).

The result can look like dark magic when you are not savvy with sed and regular expressions:

Host *+*
ProxyCommand ssh $(echo %h | sed 's/+[^+]*$//;/+/!{s/%%/@/;s/:/ -p /};s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/') PATH=.:\$PATH nc -w1 $(echo %h | sed 's/^.*+//;/:/!s/$/ %p/;s/:/ /')

The syntax you can use to connect through jump hosts is the following:

ssh login1%host1:port1+login2%host2:port2+host3:port3 -l login3

We'll see further down why we can't put a login for the final destination host. Or if you want to skip the boring implementation details, you can also skip to the end, where a slightly more compact version lies: while writing this post, I got some optimization ideas.

Let me try to split the sed syntax to make it a little more understandable:

s/+[^+]*$//; # Remove all characters starting from the last "+" in the string (i.e. keep the n - 1 first hops)
/+/!{ # If the string doesn't contain a "+" after the previous command (i.e. there is only one hop remaining), do the following, otherwise, skip until the closing curly brace
s/%%/@/; # Replace "%" with "@" (% is doubled because of the ProxyCommand)
s/:/ -p / # Replace ":" with " -p ". Combined with the previous command, we rewrite "login%host:port" as "login@host -p port"
};
s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/ # Rewrite "hop1+hop2+login%lasthop" as "hop1+hop2+lasthop -l login"

The second sed goes like this:

s/^.*+//; # Remove everything up to the last occurrence of "+" in the string (i.e. only keep the last hop)
/:/! # If there is no ":" in the string, do the following, otherwise, skip the next statement
s/$/ %p/; # Add " %p" at the end of the line
s/:/ / # Replace ":" with " ". These last three instructions prepare a "host port" combination for use with nc.

Let's see what the ProxyCommand looks like for some examples, skipping PATH setting and nc option for better readability:

host1+host2
ssh host1 nc host2 %p
login1%host1+host2:port2
ssh login1@host1 nc host2 port2
login1%host1:port1+host2:port2
ssh login1@host1 -p port1 nc host2 port2

Now, maybe you start to see why we can't put a login on the last hop: not only does it make no sense for nc, but also the main ssh process that runs the ProxyCommand and will actually talk to the remote host won't have any knowledge of it.

From the above examples, it also appears obvious why we replace "login%host:port" with "login@host -p port" : so that the ssh command in the ProxyCommand gets the proper arguments for login and port. Note we could also replace with "host -p port -l login" for the same effect.

With even more hops, this is what happens:

host1+host2+host3
ssh host1+host2 nc host3 %p
login1%host1:port1+host2:port2+host3
ssh login1%host1:port1+host2:port2 nc host3 %p

Each of these ProxyCommands will trigger another ProxyCommand quite looking like our first few examples.

In the first hops, if we'd just replace all "%" with "@", in cases where logins are given everywhere, we'd end up with a ProxyCommand like the following:

ssh login%host1+login%host2 nc host3 %p

As we saw above, we can't use a login on the last hop, which means this ProxyCommand wouldn't work as expected and is why we have to rewrite "login1%host1+login2%hop2" as "login%host1+host2 -l login2" with the last instruction in the first sed.

In the end, this is what happens:

login1%host1+login2%host2+host3
ssh login1%host1+host2 -l login2 nc host3 %p
login1%host1:port1+login2%host2:port2+host3:port3
ssh login1%host1:port1+host2:port2 -l login2 nc host3 %p

In this last example, you see the ":port2" part is not changed into "-p port2". Actually, it would still work with the latter form : the ProxyCommand ssh login1%host1:port1+host2 -p port2 -l login2 would itself have ssh login1@host1 -p port1 nc host2 %p as ProxyCommand, in which %p would be replaced by port2, given as argument to -p.

Based on this and with further small optimizations, we can slightly shorten our ProxyCommand to the following :

Host *+*
ProxyCommand ssh $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:/ -p /') PATH=.:\$PATH nc -w1 $(echo %h | sed 's/^.*+//;/:/!s/$/ %p/;s/:/ /')

sed instructions "decryption" is left as an exercise to the reader ;).

Update : modified the shortened version to fix the issue spotted in the comments.

2009-04-11 23:25:50+0900

p.d.o

Both comments and pings are currently closed.

22 Responses to “Following up on SSH through jump hosts”

  1. nate Says:
    2009-04-12 06:37:38+0900

    I donno.

    All of that looks very wrong. It makes my head spin. There must be a better solution…

    Couldn’t you use a real language like Python or whatnot?

    For a while now Python has had the very nice ‘subprocess’ module that executes commands and handles arguements without a shell.. making lots of things much simplier. Also can handle redirecting stdin, stdout, stderr easy.

    For example I use it to deal with situations were I may run into maliciously created file names. With bash or shell environments the only effective way to deal with potentially malicious filenames is to use nulls for line seperators, but only a few commands (like find, sort, xargs) support it and you only see that support in GNU extentions so they are very non-portable.

    But with subprocess I can pass all sorts of crappy filenames, including file names with newlines or other characters that have significance in a shell environment, directly to commands as arguements and so far everything handles it quite well.

    Then it there are all sorts of modules for dealing with networking stuff. I donno.

    Your a much better programmer then me, I wouldn’t of tried to do what your doing.. it would make my head implode and I would always be worried about making a small error that would cause some sort of exploit or whatnot.

  2. glandium Says:
    2009-04-12 07:54:36+0900

    nate: The setup from this post goes into a ProxyCommand in $HOME/.ssh/config, so you either have to use a one-liner or an external script. The ProxyCommand is executed with $SHELL -c by default. There is not much to exploit either.

  3. phocean Says:
    2009-04-12 09:44:17+0900

    I found a tip a while ago, to make up the proxy even without netcat :
    ProxyCommand ssh host1 ‘exec 3<>/dev/tcp/{host2}/22;(cat <&3 & ); cat >&3’
    I would like to integrate it to your tip, but I won’t do it soon as it would take me hours…

  4. glandium Says:
    2009-04-12 11:28:09+0900

    phocean: ProxyCommand ssh $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:/ -p /’) "exec 3<>/dev/tcp/$(echo %h | sed ’s/^.*+//;/:/!s/$/\/%p/;s/:/\//’);(cat <&3 & ); cat >&3" should do the trick, but relies on the remote shell supporting /dev/tcp/host/post.

    Excerpt from the bash manpage:

    NOTE: Bash, as packaged for Debian, does not support using the /dev/tcp and /dev/udp files.

    See http://bugs.debian.org/65172 to understand why.

  5. th9 Says:
    2009-04-12 15:25:32+0900

    That last version of yours eats my memory and runs away…^C

  6. glandium Says:
    2009-04-12 16:03:58+0900

    th9: the one in the comments ?

  7. th9 Says:
    2009-04-12 17:00:26+0900

    glandium, the one in the article

  8. glandium Says:
    2009-04-12 18:23:19+0900

    th9: there is nothing in that setup that should eat memory… did you just copy/paste it ? wordpress may have replaced some characters with “fancier” versions, so you need to be extra careful.

  9. th9 Says:
    2009-04-12 19:15:16+0900

    Yes I copied it and removed PATH=.:\$PATH since I don’t need that and had to replace single quotes “‘”.. then CPU goes on full charge till the RAM is full and computer starts swapping, then it becomes unresponsive.. ps -A showed a lot of ssh processes, looked likes like its infinite loop problem. The version at the beginning of article forks fine (after replacing single quotes). It shouldn’t matter but I’m not on Debian :)

  10. th9 Says:
    2009-04-12 20:21:25+0900

    Oh, it actually works if I specify login for a jumphost (e.g. login%host1+host2).. otherwise it returns %h infinitely.. so the problematic place is %% between login and host.

  11. th9 Says:
    2009-04-13 11:30:09+0900

    Thanks a lot! This one was really helpful :) now I just have to figure out how to get X trough the tunnel and I’m set.

  12. glandium Says:
    2009-04-13 13:08:09+0900

    th9: -X or -Y depending on your case, as usual.

  13. th9 Says:
    2009-04-13 15:13:55+0900

    glandium, unfortunately it doesn’t work like that (can’t open display: ).. it might be related to that X is started with -nolisten tcp, but haven’t figured out yet how to check it.

  14. glandium Says:
    2009-04-13 18:48:56+0900

    th9: try adding -v and checking what it tells you about xauth.

  15. th9 Says:
    2009-04-13 19:18:04+0900

    glandium, thanks for advice

    With -X it complains about missing xauth data:
    Warning: untrusted X11 forwarding setup failed: xauth key data not generated
    Warning: No xauth data; using fake authentication data for X11 forwarding.
    debug1: Requesting X11 forwarding with authentication spoofing.

    With -Y does not:
    debug1: Requesting X11 forwarding with authentication spoofing.

    Although I have set XAuthLocation /usr/bin/xauth it still doesn’t work.
    With increased verbosity this follows:
    debug1: Requesting X11 forwarding with authentication spoofing.
    debug2: channel 0: request x11-req confirm 0
    debug2: client_session2_setup: id 0
    debug2: channel 0: request pty-req confirm 1
    debug2: channel 0: request shell confirm 1
    debug2: callback done
    debug2: channel 0: open confirm rwindow 0 rmax 32768
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: PTY allocation request accepted on channel 0
    debug2: channel 0: rcvd adjust 2097152
    debug2: channel_input_status_confirm: type 99 id 0
    debug2: shell request accepted on channel 0

    I don’t know actually what to make out of this.

  16. th9 Says:
    2009-04-15 15:36:17+0900

    ok, reinstalled and now it works as expected.. I guess it was some subtle configuration issue. Thanks!

  17. ossblog Says:
    2009-04-19 13:15:08+0900

    Usare SSH attraverso proxy…

    Può capitare di avere accesso ad alcune macchine solo dopo essersi loggati attraverso una terza macchina.
    Ssh dispone di una comoda funzionalità, ProxyCommand, utile in casi simili per semplificare ed automatizzare tutta la procedura di login. Mi…

  18. Usare SSH attraverso proxy « Best Web News Blog Says:
    2009-04-19 13:27:06+0900

    […] L’articolo originale mostra tutte le decisioni che hanno portato alla precedente linea di comando ed alla spiegazione del funzionamento dei comandi lanciati all’interno di ProxyCommand. […]

  19. Usare SSH attraverso proxy | Cicoira.it Says:
    2009-04-19 13:41:13+0900

    […] L’articolo originale mostra tutte le decisioni che hanno portato alla precedente linea di comando ed alla spiegazione del funzionamento dei comandi lanciati all’interno di ProxyCommand. […]

  20. Usare SSH attraverso proxy - marko’s weblog Says:
    2009-04-20 08:58:00+0900

    […] L’articolo originale mostra tutte le decisioni che hanno portato alla precedente linea di comando ed alla spiegazione del funzionamento dei comandi lanciati all’interno di ProxyCommand. […]

  21. » Usare SSH attraverso proxy Says:
    2009-04-20 09:59:44+0900

    […] L’articolo originale mostra tutte le decisioni che hanno portato alla precedente linea di comando ed alla spiegazione del funzionamento dei comandi lanciati all’interno di ProxyCommand. […]

  22. Who» Who Says:
    2009-06-19 20:17:57+0900

    […] L’articolo originale mostra tutte le decisioni che hanno portato alla precedente linea di comando ed alla spiegazione del funzionamento dei comandi lanciati all’interno di ProxyCommand. […]