Bastion host
A bastion host (or jump host) is a server acting as a secure gateway between the internet and the private network inside the cloud. It does not have any applications apart from a firewall and other security-related software. The implementation in this guide is based on Ubuntu 18.04. Some options and best practices are listed in the end of the chapter.
Contents
The bastion host is a server (compute instance) having a floating IP address associated with it, as well as a private IP address in the private network. The typical functionality of a bastion host include
SSH proxy
Firewall
Security logging
The steps how to set up and configure these functions on a bastion server are:
Configure the client to use SSH proxy by creating an SSH config file
Configure SSH restrictions, firewall and SSH attack protection on the bastion host
Create a restricted security group and assign to remote hosts
We assume that there are two server instances available, and for simplicity using the same key pair (see https://pannet.atlassian.net/wiki/spaces/DocEng/pages/524321364). One of the servers, which needs to have a floating IP address assigned to it, will be a bastion host called <bastion>
and the other a remote host called <remote>
. The security groups must be set to allow SSH (TCP on port 22) on both hosts.
Configure SSH on client
The SSH proxy commands are issued from the client, and are conveniently configured in the local SSH config file.
SSH config
On the client, open the file ~/.ssh/config
for editing with
nano ~/.ssh/config
The file will be created if it does not already exist. However, the directory needs to be created first with mkdir ~/.ssh
) Assuming that the same key pair used for accessing all hosts in the tenant, this can be specified in ~/.ssh/config
on the client as
1
2
Host *
IdentityFile <user-site-key>.pem
It is also necessary to set the configuration file permissions properly by
sudo chmod 600 ~/.ssh/config
With this configuration, the key file does not have to be specified explicitly in the commands. In case the hosts use different key pairs, the IdentityFile
declarations can be moved into individual Host
sections.
Using nicknames for the bastion and remote hosts, the syntax in ~/.ssh/config
is
1
2
3
4
5
6
7
8
Host <bastion-nickname>
User ubuntu
HostName <bastion-ip>
Host <remote-nickname>
User ubuntu
HostName <remote-ip>
ProxyJump <bastion-nickname>
where User
is ubuntu
on Ubuntu and HostName
is the IP addresses of the bastion and remote hosts. The bastion host has a public, and the remote servers private addresses. The nickname is some memorable name of the servers, for example the names given at the time of their creation. Save and close the file. The settings take effect immediately without restarting anything.
An example of an SSH configuration file is shown in Figure 1.
The remote host can now be accessed simply by
ssh <remote-nickname>
which in the example would be
ssh raven
Adding connectivity to a new instance through the bastion host is now done by adding a record to ~/.ssh/config
that contains <remote-user>
and <remote-ip>
(as HostName
) and if necessary the key file (as IdentityFile
). That is, no further configuration needs to be done on the bastion host itself.
Settings on bastion host
On the server designated to be the bastion host, the applications and settings are only aiming at hardening it from unauthorized access. This includes the SSH server, firewall and attack prevention configurations. The configuration work is performed over SSH.
It is a good approach to implement the settings step by step, logging out and logging back in again after each step to verify that a connection still can be established. Should SSH be blocked by some mistaken setting, the entire server needs to be rebuilt or recreated with OpenStack and the configuration work started from the beginning.
SSH server
The SSH server configuration file /etc/ssh/sshd_config
should be modified for best performance. Open the file for editing with
sudo nano /etc/ssh/sshd_config
First, the authentication method should be enforced in /etc/ssh/sshd_config
by rejecting password authentication. The entry already exists in the configuration file and should be set to
1
PasswordAuthentication no
This removes the vulnerability exposed by the password field, where an intruder can find the password by repetitive guessing and trial.
Next, for all users apart from internal tenant users - since only Ubuntu is used here, the username is ubuntu
- enter the following settings to the end of the file sshd_config
:
1
2
3
4
5
6
7
Match User *,!ubuntu
GatewayPorts no
AllowAgentForwarding no
X11Forwarding no
PermitTunnel no
ForceCommand echo 'This account can only be used for SSH port forwarding'
AllowTcpForwarding yes
Save the changes and close the file. The new configuration can be tested to check that the service will start properly with the new settings with
sudo sshd -t
When everything is consistent, there is no output printed. If there is any configuration error, for example a space after the comma in *,!ubuntu
, an error message is printed to the terminal.
Restarting the SSH service can be done without interrupting the ongoing session by the command
sudo /etc/init.d/ssh restart
The sequence of SSH commands and outputs are shown in Figure 2.
Configure firewall
Even when the bastion host have restrictive security group settings, the additional protection of a separate firewall is recommended (see also https://pannet.atlassian.net/wiki/spaces/DocEng/pages/501022858).
Uncomplicated firewall (UFW), which is part of the Ubuntu distribution, is by default set to deny all incoming traffic and allow all outgoing traffic. Note that since the bastion host is configured over SSH, enabling the firewall without having set a rule to allow SSH will block this connection. The current set of rules (in optional verbose
mode) is displayed with
sudo ufw status verbose
When UFW is disabled, it will only show Status: inactive
. Note that switching on the firewall may break the SSH connection and block further attempts. It is therefore crucial that the exception of SSH to the default rule of blocking incoming traffic is in place before starting it. The rule is set by
sudo ufw allow ssh
This produces the output (Rules updated) shown in Figure 3.
It is also important to apply rules to IPv6 regardless of whether it is being used or not. Open the firewall configuration file with a standard text editor, for example
sudo nano /etc/default/ufw
and set IPV6=yes
(Figure 4).
The firewall is enabled with
sudo ufw enable
which displays a warning prompt (Figure 5).
Type y
and press ENTER
to acknowledge. Now sudo ufw status verbose
should show a result like in Figure 6.
Blocking SSH attacks
Utilities such as Fail2Ban scan log files and blocks IP address that show suspicious behavior, such as a large number of failed SSH login attempts. Such attacks are known as brute force or dictionary-based attacks. Installation and configuration of fail2ban is described in https://pannet.atlassian.net/wiki/spaces/DocEng/pages/501022858
Security group on remote hosts
After the need for direct SSH connectivity to remote hosts has been removed by the SSH proxy, the security groups should be tightened to reflect this. The rule in the security group allowing SSH should now be complemented by the CIDR for the local network (that is, subnet). To create the new rule for the subnet, for example 192.168.0.0/24
, do
openstack security group create ssh-internal
openstack security group rule create --remote-ip 192.168.0.0/24 --dst-port 22 --protocol tcp --ingress ssh-internal
and replace the SSH security group on the remote server with this. The bastion host cannot be restricted with this security group, which only allows connections from the internal network.
Alternatively, the new rule can be created in Horizon (Figure 7).
The security group for the bastion host, however, needs to accept IP addresses from external networks, but access can here be limited to IP addresses from known clients.
SSH proxy
For an application consisting of multiple instances deployed in a tenant, a fundamental requirement is to have a secure management (SSH) interface to all of these instances. The bastion host performs this by establishing an SSH connection to any other instance from a hardened dedicated server, and removes the need for direct SSH connectivity to the back-end servers and storing keys or certificates on intermediary hosts. This also leads to a much improved economy in terms of floating IP addresses.
There are two functions in OpenSSH that can be used for this purpose: agent forwarding or SSH proxy. Agent forwarding is not recommended, since the SSH agent has access to the private key on the client, which makes the system vulnerable to interception by anyone with root access.
The SSH proxy command uses forwarding of the stdin
and stdout
streams between the bastion and the remote hosts. Suppose we can login to the bastion host from the client with
ssh -i <user-site-key>.pem <bastion-user>@<bastion-ip>
and from there to the remote host with
ssh -i <user-site-key>.pem <remote-user>@<remote-ip>
where the key pair <user-site-key>.pem
used to access both hosts is assumed to be the same for simplicity, but different key pairs for each host can easily be configured. We can then first login to the bastion host, and from there login to the remote host, or - using either ProxyCommand
or ProxyJump
- login to the remote host directly from the client.
ProxyJump
SSH proxy based on ProxyJump
(from OpenSSH version 7.3) has the compact syntax
ssh -J <bastion-user>@<bastion-ip> <remote-user>@<remote-ip>
In the local SSH configuration file ~/.ssh/config
, the syntax is ProxyJump <bastion-ip>
. or ProxyJump <bastion-nickname>
.
ProxyCommand
It is also possible to implement SSH proxy based on ProxyCommand
, which has the syntax
1
2
ssh -i <user-site-key>.pem -o ProxyCommand="ssh -W %h:%p <bastion-user>@<bastion-ip>"
<remote-user>@<remote-ip>
Note that there should be no blank spaces surrounding the equality sign =
and that the private IP address of the remote host is used. The proxy command can be added the file ~/.ssh/config
on the SSH client machine as
1
2
3
4
Host <remote-nickname>
User <remote-user>
HostName <remote-ip>
ProxyCommand ssh <bastion-user>@<bastion-ip> -W %h:%p
Just as with ProxyJump
, it is then possible to login to to the remote host from the client with
ssh <remote-nickname>
using the host’s private IP behind the bastion host.
Security best practices
To ensure a high level of integrity and security of the jump host, it should meet the following conditions:
Install a minimal distribution option
Apply patches and updates regularly
Do not host any protocols except for SSH - disable and/or remove unnecessary protocols, daemons, and services
Avoid password authentication and use certificates or key pair authentication, or stronger multi-factor or hardware token authentication (such as YubiKeys)
Never store SSH private keys on the host
Do not use SSH agent forwarding (which is vulnerable to SSH hijacking), but use SSH proxy instead
Set up a services such as
fail2ban
to prevent brute force attacksDo not disable strict host check. The SSH property
StrictHostKeyChecking
in/etc/ssh/ssh_config
should be set to
1
2
Host *
StrictHostKeyChecking yes
When the StrictHostKeyChecking
property is set to yes
, SSH will not automatically add host keys to ~/.ssh/known_hosts
, and then it is necessary to configure internal hosts with /etc/hosts.allow
and /etc/hosts.deny
files to control access.
Use a restrictive, host-based firewall on all Linux hosts as complement to OpenStack security groups
Include Host-based Intrusion Detection System (HIDS), for example OSSEC or TripWire
Collect and analyze security logs (stored off-side), and track configuration changes
Create at least one secondary jump host as backup in case of failure; use NAT forwarding to the jump host
To restrict user SSH access and prevent shell apart from the management user (in this example ubuntu
), edit the sshd_config
file with the following settings:
1
2
3
4
5
6
7
Match User *,!ubuntu
GatewayPorts no
AllowAgentForwarding no
X11Forwarding no
PermitTunnel no
ForceCommand echo 'This account can only be used for SSH port forwarding'
AllowTcpForwarding yes
Further details on how to implement some of these functions can be found in https://pannet.atlassian.net/l/c/D4wet2wk