When configuring an SSH bastion or even a machine, it's often a good idea to disable the TCP port forwarding option, which allows you to forward a network port between your SSH client and SSH server. The need is often to isolate networks from each other, and the creation of TCP tunnels would increase the attack surface.
I had never been interested in the details of how TTYs and the SSH protocol work. I foolishly thought that disabling the AllowTcpForwarding option in sshd_config would correctly disable the creation of a TCP tunnel between your workstation and the SSH server. Leaving only the user free to execute commands on the remote server.
I was surprised to discover in the sshd_config manual that this AllowTcpForwarding parameter is not a security option and is easily bypassed when shell mode is enabled. Which is my case...
AllowTcpForwarding: Specifies whether TCP forwarding is permitted... Note that disabling TCP forwarding does not improve security unless users are also denied shell access, as they can always install their own forwarders.
https://man7.org/linux/man-pages/man5/sshd_config.5.html
The initial problem
Bypass SSH network forwarding via session channels
One thing I should have known is that even with the TCP forwarding option disabled, it's still possible to connect using SCP or SFTP. Binary file transfer works. So if it's possible to transfer binary files, is it possible to do more?
To be clear about the components I'm mentioning in this blog post, I'm using OpenSSH version 8. The SSH protocol makes it possible to chat on several different channels in a single TCP stream. If you look in the manual, 7 different channel types are listed (their names are self-explanatory):
- agent-connection: Mostly disable because of security issue
- direct-tcpip, forwarded-tcpip: Disable by AllowTcpForwarding or DisableForwarding option!
- x11-connection: Disable by default
- session:command, session:shell, session:subsystem: Use to execute command, that's what interests us here
As you can see, there is no reference to SFTP in the SSH channels. If we take a look on SSHd log during an SFTP authentification using WinSCP:
Jul 01 16:15:09 arfevrier.fr sshd[2121795]: Starting session: subsystem 'sftp' for xxxxx from 81.x.x.x port 24542 id xxxx
Same thing directly using SCP command:
Jul 01 16:18:36 arfevrier.fr sshd[2122086]: Starting session: command for xxxxx from 81.x.x.x port 24575 id xxxx
So file transfer directly uses a session command to operate.
/usr/bin/scp for SCP. And in sshd_config we can read:
Subsystem sftp /usr/lib/openssh/sftp-server
We can do the same! If SFTP uses binaries to exchange data, users with shell access can do the same. We can use the SSH as a proxy server using netcat:
$ ssh arfevrier.fr "echo $SHELL"
/bin/bash
$ ssh arfevrier.fr "nc google.fr 80"
GET / HTTP/1.1
Host: google.fr
User-Agent: mycustomUserAgent/1.0
Accept: */*
HTTP/1.1 301 Moved Permanently
Location: http://www.google.fr/
Content-Type: text/html; charset=UTF-8
...
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
...
<A HREF="http://www.google.fr/">here</A>.
</BODY></HTML>
As soon as we're able to initiate a TCP connection via netcat, you can pass and exchange any type of flow. So we can perform port forwarding:
$ socat tcp-listen:12345,reuseaddr,fork exec:'ssh arfevrier.fr nc google.fr 80'
Here we forward remote port google.fr:80 on our local machine port 12345.
To sum up, we're able to execute commands on the server and use the session as a flow exchange. So we can use the SSH connection as a bidirectional pipe. I've already covered this behavior in my article on proxytunnel.
Arnaud
In the case of bastion, the aim is to bypass the bastion by accessing our target machine directly. With SSH, the ProxyCommand parameter can be used to establish an SSH connection through standard outputs.
If Netcat is not available on the bastion you can directly use TCP bash feature (thanks to Andras Kovacs's StackOverFlow anwser):
Detection
Know which commands are executed via session channels
Knowing that the user is able to pass any type of flow and that it may be encrypted, the idea is to be able to know which commands the user has executed. Unfortunately, even at its highest level of verbosity, OpenSSH does not indicate the commands received by the user. It is therefore impossible via OpenSSH logs to know which commands have been executed. There is two solution to anwser the problem:
- Log all commands requested at SSH session startup. That's what I'm interested in. I'd like to know which sessions do port forwarding at startup.
- Log all user command perform on the system. This is more a system audit issue. This goes beyond the SSH perimeter.
Let's start with solution 2. A first solution is to use Auditd, properly configured (thanks to https://blog.shichao.io/2015/04/22/auditing_user_tty_and_root_commands_with_auditd_on_ubuntu.html) it can be used to find out what keys the user is typing into the console. But in the same way, it works for TTY shells but not for commands executed without them.
We can also try to use Auditd to log any commands executed on the system. I won't go into this part in detail, as it's really focused on linux administration. And therefore out of the SSH context. But it can be a solution.
Another system level package is acct which allow to trace executed user command with lastcomm command.
Browsing StackOverFlow for ideas, I found the SSHlog tool, which provides a daemon that lists user commands:
sshlogBut another solution that I find more interesting and that relies on another OpenSSH feature is the ForceCommand parameter.
Forces the execution of the command specified by ForceCommand, ignoring any command supplied by the client and ~/.ssh/rc if present. The command is invoked by using the user's login shell with the -c option. ... The command originally supplied by the client is available in the SSH_ORIGINAL_COMMAND environment variable. ...
https://man7.org/linux/man-pages/man5/sshd_config.5.html
This is not realy interesting in our case because we don't want to retrict user command execution. I find over the internet lot of script trying to log the user command on then execute it (for exemple https://jms1.net/ssh-record.shtml).
I don't really like it, because it can easily break. You need to make sure that the commands the user executes escape the right characters or support multi-line, for example.
Let's go back to solution 1. In my case, I prefer simply list all the commands that are requested when the SSH session is started. I assume that if the user wants to automate command execution, he'll do so at session startup. Even if, we agree, it is possible to automate interaction in a shell.
An important thing to know is that when the user wishes to execute an SSH command, it will always be done in the shell defined for the user.
Take a look at my /etc/passwd file:
$ cat /etc/passwd | grep arnaud
arnaud:x:1000:1000:Arnaud,,,:/home/arnaud:/bin/bash
Bash is my default shell. Which mean that every command I start with SSH, will be run with bash. This behavior cannot be modified by the remote user. When using bash, this means above all that the system file /etc/bash.bashrc is loaded every time the shell is started.
One tweak we need to add is to enable ForceCommand. Because if we don't define this variable in the config, the environment variable SSH_ORIGINAL_COMMAND will not be set. And we want to be set so that bash.bashrc is able to log it.
In your sshd_config, define:
ForceCommand $SSH_ORIGINAL_COMMAND
And then, let's add a logging feature on .bashrc:
$ echo 'logger -p user.notice "$SSH_CONNECTION $SSH_ORIGINAL_COMMAND"' >> /etc/bash.bashrc
Now on every new shell or command session you will have a log in /var/log/syslog. This is also working with non-TTY shell and with SCP, SFTP protocol.
Jul 1 22:35:21 arfevrier.fr arnaud: x.x.x.x 26314 x.x.x.x 22 /usr/lib/openssh/sftp-server
Jul 1 22:51:34 arfevrier.fr arnaud: x.x.x.x 26366 x.x.x.x 22 sh -c echo todo > /dev/null
Here are two different log examples, the first resulting from an sftp connection, and the second when executing a command.
We've seen how to bypass the port forwarding feature used in an SSH server. But when using a shell session, it is possible to restrict and audit the commands executed by users. The solution I propose is a first step in securing a bastion. It allows you to quickly find out which users are using ProxyCommands through the bastion. It's important to understand that when you give users access to an operating system, the attack surface is enormous and so are the possible rebounds. In such cases, the second step is to use tools such as Auditd to keep track of all user actions at system call level. This is the only way to keep track of all actions performed on the server. As Auditd is a passive tool, you need to raise alerts, for example in a SIEM, and actively react on any suspicious behavior.
