A bastion host (or jump host) is a server acting as a secure gateway between the internet and the private network inside the cloud. It does not have any applications apart from a firewall and other security-related software. The implementation in this guide is based on Ubuntu 18.04. Some options and best practices are listed in the end of the chapter.

Contents

The bastion host is a server (compute instance) having a floating IP address associated with it, as well as a private IP address in the private network. The typical functionality of a bastion host include

• SSH proxy

• Firewall

• Security logging

The steps how to set up and configure these functions on a bastion server are:

• Configure the client to use SSH proxy by creating an SSH config file

• Configure SSH restrictions, firewall and SSH attack protection on the bastion host

• Create a restricted security group and assign to remote hosts

We assume that there are two server instances available, and for simplicity using the same key pair (see https://pannet.atlassian.net/wiki/spaces/DocEng/pages/524321364). One of the servers, which needs to have a floating IP address assigned to it, will be a bastion host called <bastion> and the other a remote host called <remote>. The security groups must be set to allow SSH (TCP on port 22) on both hosts.

Configure SSH on client

The SSH proxy commands are issued from the client, and are conveniently configured in the local SSH config file.

SSH config

On the client, open the file ~/.ssh/config for editing with

nano ~/.ssh/config

The file will be created if it does not already exist. However, the directory needs to be created first with mkdir ~/.ssh) Assuming that the same key pair used for accessing all hosts in the tenant, this can be specified in ~/.ssh/config on the client as

It is also necessary to set the configuration file permissions properly by

sudo chmod 600 ~/.ssh/config

With this configuration, the key file does not have to be specified explicitly in the commands. In case the hosts use different key pairs, the IdentityFile declarations can be moved into individual Host sections.

Using nicknames for the bastion and remote hosts, the syntax in ~/.ssh/config is

where User is ubuntu on Ubuntu and HostName is the IP addresses of the bastion and remote hosts. The bastion host has a public, and the remote servers private addresses. The nickname is some memorable name of the servers, for example the names given at the time of their creation. Save and close the file. The settings take effect immediately without restarting anything.

An example of an SSH configuration file is shown in Figure 1.

Figure 1. SSH configuration file with ProxyJump.

The remote host can now be accessed simply by

ssh <remote-nickname>

which in the example would be

ssh raven

Adding connectivity to a new instance through the bastion host is now done by adding a record to ~/.ssh/config that contains <remote-user> and <remote-ip> (as HostName) and if necessary the key file (as IdentityFile). That is, no further configuration needs to be done on the bastion host itself.

Settings on bastion host

On the server designated to be the bastion host, the applications and settings are only aiming at hardening it from unauthorized access. This includes the SSH server, firewall and attack prevention configurations. The configuration work is performed over SSH.

It is a good approach to implement the settings step by step, logging out and logging back in again after each step to verify that a connection still can be established. Should SSH be blocked by some mistaken setting, the entire server needs to be rebuilt or recreated with OpenStack and the configuration work started from the beginning.

SSH server

The SSH server configuration file /etc/ssh/sshd_config should be modified for best performance. Open the file for editing with

sudo nano /etc/ssh/sshd_config

First, the authentication method should be enforced in /etc/ssh/sshd_config by rejecting password authentication. The entry already exists in the configuration file and should be set to

This removes the vulnerability exposed by the password field, where an intruder can find the password by repetitive guessing and trial.

Next, for all users apart from internal tenant users - since only Ubuntu is used here, the username is ubuntu - enter the following settings to the end of the file sshd_config:

Save the changes and close the file. The new configuration can be tested to check that the service will start properly with the new settings with

sudo sshd -t

When everything is consistent, there is no output printed. If there is any configuration error, for example a space after the comma in *,!ubuntu, an error message is printed to the terminal.

Restarting the SSH service can be done without interrupting the ongoing session by the command

sudo /etc/init.d/ssh restart

The sequence of SSH commands and outputs are shown in Figure 2.

Figure 2. Sequence of SSH commands.

Configure firewall

Even when the bastion host have restrictive security group settings, the additional protection of a separate firewall is recommended (see also https://pannet.atlassian.net/wiki/spaces/DocEng/pages/501022858).

Uncomplicated firewall (UFW), which is part of the Ubuntu distribution, is by default set to deny all incoming traffic and allow all outgoing traffic. Note that since the bastion host is configured over SSH, enabling the firewall without having set a rule to allow SSH will block this connection. The current set of rules (in optional verbose mode) is displayed with

sudo ufw status verbose

When UFW is disabled, it will only show Status: inactive. Note that switching on the firewall may break the SSH connection and block further attempts. It is therefore crucial that the exception of SSH to the default rule of blocking incoming traffic is in place before starting it. The rule is set by

sudo ufw allow ssh

This produces the output (Rules updated) shown in Figure 3.

Figure 3. Updating SSH rules in UFW.

It is also important to apply rules to IPv6 regardless of whether it is being used or not. Open the firewall configuration file with a standard text editor, for example

sudo nano /etc/default/ufw

and set IPV6=yes (Figure 4).

Figure 4. UFW configuration file.

The firewall is enabled with

sudo ufw enable

which displays a warning prompt (Figure 5).

Figure 5. Enabling UFW.

Type y and press ENTER to acknowledge. Now sudo ufw status verbose should show a result like in Figure 6.

Figure 6. UFW status showing default rules and exception for SSH.

Blocking SSH attacks

Utilities such as Fail2Ban scan log files and blocks IP address that show suspicious behavior, such as a large number of failed SSH login attempts. Such attacks are known as brute force or dictionary-based attacks. Installation and configuration of fail2ban is described in https://pannet.atlassian.net/wiki/spaces/DocEng/pages/501022858

Security group on remote hosts

After the need for direct SSH connectivity to remote hosts has been removed by the SSH proxy, the security groups should be tightened to reflect this. The rule in the security group allowing SSH should now be complemented by the CIDR for the local network (that is, subnet). To create the new rule for the subnet, for example 192.168.0.0/24, do

openstack security group create ssh-internal

openstack security group rule create --remote-ip 192.168.0.0/24 --dst-port 22 --protocol tcp --ingress ssh-internal

and replace the SSH security group on the remote server with this. The bastion host cannot be restricted with this security group, which only allows connections from the internal network.

Alternatively, the new rule can be created in Horizon (Figure 7).

Figure 7. SSH rule restricted to internal network.

The security group for the bastion host, however, needs to accept IP addresses from external networks, but access can here be limited to IP addresses from known clients.

SSH proxy

For an application consisting of multiple instances deployed in a tenant, a fundamental requirement is to have a secure management (SSH) interface to all of these instances. The bastion host performs this by establishing an SSH connection to any other instance from a hardened dedicated server, and removes the need for direct SSH connectivity to the back-end servers and storing keys or certificates on intermediary hosts. This also leads to a much improved economy in terms of floating IP addresses.

There are two functions in OpenSSH that can be used for this purpose: agent forwarding or SSH proxy. Agent forwarding is not recommended, since the SSH agent has access to the private key on the client, which makes the system vulnerable to interception by anyone with root access.

The SSH proxy command uses forwarding of the stdin and stdout streams between the bastion and the remote hosts. Suppose we can login to the bastion host from the client with

ssh -i <user-site-key>.pem <bastion-user>@<bastion-ip>

and from there to the remote host with

ssh -i <user-site-key>.pem <remote-user>@<remote-ip>

where the key pair <user-site-key>.pem used to access both hosts is assumed to be the same for simplicity, but different key pairs for each host can easily be configured. We can then first login to the bastion host, and from there login to the remote host, or - using either ProxyCommand or ProxyJump - login to the remote host directly from the client.

ProxyJump

SSH proxy based on ProxyJump (from OpenSSH version 7.3) has the compact syntax

ssh -J <bastion-user>@<bastion-ip> <remote-user>@<remote-ip>

In the local SSH configuration file ~/.ssh/config , the syntax is ProxyJump <bastion-ip>. or ProxyJump <bastion-nickname>.

ProxyCommand

It is also possible to implement SSH proxy based on ProxyCommand, which has the syntax

Note that there should be no blank spaces surrounding the equality sign = and that the private IP address of the remote host is used. The proxy command can be added the file ~/.ssh/config on the SSH client machine as

Just as with ProxyJump, it is then possible to login to to the remote host from the client with

ssh <remote-nickname>

using the host’s private IP behind the bastion host.

Security best practices

To ensure a high level of integrity and security of the jump host, it should meet the following conditions:

• Install a minimal distribution option

• Apply patches and updates regularly

• Do not host any protocols except for SSH - disable and/or remove unnecessary protocols, daemons, and services

• Avoid password authentication and use certificates or key pair authentication, or stronger multi-factor or hardware token authentication (such as YubiKeys)

• Never store SSH private keys on the host

• Do not use SSH agent forwarding (which is vulnerable to SSH hijacking), but use SSH proxy instead

• Set up a services such as fail2ban to prevent brute force attacks

• Do not disable strict host check. The SSH property StrictHostKeyChecking in /etc/ssh/ssh_config should be set to

• Use a restrictive, host-based firewall on all Linux hosts as complement to OpenStack security groups

• Include Host-based Intrusion Detection System (HIDS), for example OSSEC or TripWire

• Collect and analyze security logs (stored off-side), and track configuration changes

• Create at least one secondary jump host as backup in case of failure; use NAT forwarding to the jump host

To restrict user SSH access and prevent shell apart from the management user (in this example ubuntu), edit the sshd_config file with the following settings:

 Further details on how to implement some of these functions can be found in https://pannet.atlassian.net/l/c/D4wet2wk