A bastion host (or jump host) is a server acting as a secure gateway between the internet and the private network inside the cloud. It does not have any applications apart from a firewall and other security-related software. The implementation in this guide is based on Ubuntu 18.04. Some options and best practices are listed in the end of the chapter.
The bastion host is a server (compute instance) having a floating IP address associated with it, as well as a private IP address in the private network. The typical functionality of a bastion host include
The steps how to set up and configure these functions on a bastion server are:
Configure the client to use SSH proxy by creating an SSH config file
Configure SSH restrictions, firewall and SSH attack protection on the bastion host
Create a restricted security group and assign to remote hosts
We assume that there are two server instances available, and for simplicity using the same key pair (see https://pannet.atlassian.net/wiki/spaces/DocEng/pages/524321364). One of the servers, which needs to have a floating IP address assigned to it, will be a bastion host called <bastion> and the other a remote host called <remote>. The security groups must be set to allow SSH (TCP on port 22) on both hosts.
Configure SSH on client
The SSH proxy commands are issued from the client, and are conveniently configured in the local SSH config file.
On the client, open the file ~/.ssh/config for editing with
The file will be created if it does not already exist. However, the directory needs to be created first with mkdir ~/.ssh) Assuming that the same key pair used for accessing all hosts in the tenant, this can be specified in ~/.ssh/config on the client as
It is also necessary to set the configuration file permissions properly by
sudo chmod 600 ~/.ssh/config
With this configuration, the key file does not have to be specified explicitly in the commands. In case the hosts use different key pairs, the IdentityFile declarations can be moved into individual Host sections.
Using nicknames for the bastion and remote hosts, the syntax in ~/.ssh/config is
where User is ubuntu on Ubuntu and HostName is the IP addresses of the bastion and remote hosts. The bastion host has a public, and the remote servers private addresses. The nickname is some memorable name of the servers, for example the names given at the time of their creation. Save and close the file. The settings take effect immediately without restarting anything.
An example of an SSH configuration file is shown in Figure 1.
Figure 1. SSH configuration file with ProxyJump.
The remote host can now be accessed simply by
which in the example would be
Adding connectivity to a new instance through the bastion host is now done by adding a record to ~/.ssh/config that contains <remote-user> and <remote-ip> (as HostName) and if necessary the key file (as IdentityFile). That is, no further configuration needs to be done on the bastion host itself.
Settings on bastion host
On the server designated to be the bastion host, the applications and settings are only aiming at hardening it from unauthorized access. This includes the SSH server, firewall and attack prevention configurations. The configuration work is performed over SSH.
It is a good approach to implement the settings step by step, logging out and logging back in again after each step to verify that a connection still can be established. Should SSH be blocked by some mistaken setting, the entire server needs to be rebuilt or recreated with OpenStack and the configuration work started from the beginning.
The SSH server configuration file /etc/ssh/sshd_config should be modified for best performance. Open the file for editing with
sudo nano /etc/ssh/sshd_config
First, the authentication method should be enforced in /etc/ssh/sshd_config by rejecting password authentication. The entry already exists in the configuration file and should be set to
This removes the vulnerability exposed by the password field, where an intruder can find the password by repetitive guessing and trial.
Next, for all users apart from internal tenant users - since only Ubuntu is used here, the username is ubuntu - enter the following settings to the end of the file sshd_config:
Match User *,!ubuntu
ForceCommand echo 'This account can only be used for SSH port forwarding'
Save the changes and close the file. The new configuration can be tested to check that the service will start properly with the new settings with
sudo sshd -t
When everything is consistent, there is no output printed. If there is any configuration error, for example a space after the comma in *,!ubuntu, an error message is printed to the terminal.
Restarting the SSH service can be done without interrupting the ongoing session by the command
sudo /etc/init.d/ssh restart
The sequence of SSH commands and outputs are shown in Figure 2.
Uncomplicated firewall (UFW), which is part of the Ubuntu distribution, is by default set to deny all incoming traffic and allow all outgoing traffic. Note that since the bastion host is configured over SSH, enabling the firewall without having set a rule to allow SSH will block this connection. The current set of rules (in optional verbose mode) is displayed with
sudo ufw status verbose
When UFW is disabled, it will only show Status: inactive. Note that switching on the firewall may break the SSH connection and block further attempts. It is therefore crucial that the exception of SSH to the default rule of blocking incoming traffic is in place before starting it. The rule is set by
sudo ufw allow ssh
This produces the output (Rules updated) shown in Figure 3.
Figure 3. Updating SSH rules in UFW.
It is also important to apply rules to IPv6 regardless of whether it is being used or not. Open the firewall configuration file with a standard text editor, for example
sudo nano /etc/default/ufw
and set IPV6=yes (Figure 4).
Figure 4. UFW configuration file.
The firewall is enabled with
sudo ufw enable
which displays a warning prompt (Figure 5).
Figure 5. Enabling UFW.
Type y and press ENTER to acknowledge. Now sudo ufw status verbose should show a result like in Figure 6.
Figure 6. UFW status showing default rules and exception for SSH.
Blocking SSH attacks
Utilities such as Fail2Ban scan log files and blocks IP address that show suspicious behavior, such as a large number of failed SSH login attempts. Such attacks are known as brute force or dictionary-based attacks. Installation and configuration of fail2ban is described in https://pannet.atlassian.net/wiki/spaces/DocEng/pages/501022858
Security group on remote hosts
After the need for direct SSH connectivity to remote hosts has been removed by the SSH proxy, the security groups should be tightened to reflect this. The rule in the security group allowing SSH should now be complemented by the CIDR for the local network (that is, subnet). To create the new rule for the subnet, for example 192.168.0.0/24, do
and replace the SSH security group on the remote server with this. The bastion host cannot be restricted with this security group, which only allows connections from the internal network.
Alternatively, the new rule can be created in Horizon (Figure 7).
Figure 7. SSH rule restricted to internal network.
The security group for the bastion host, however, needs to accept IP addresses from external networks, but access can here be limited to IP addresses from known clients.
For an application consisting of multiple instances deployed in a tenant, a fundamental requirement is to have a secure management (SSH) interface to all of these instances. The bastion host performs this by establishing an SSH connection to any other instance from a hardened dedicated server, and removes the need for direct SSH connectivity to the back-end servers and storing keys or certificates on intermediary hosts. This also leads to a much improved economy in terms of floating IP addresses.
There are two functions in OpenSSH that can be used for this purpose: agent forwarding or SSH proxy. Agent forwarding is not recommended, since the SSH agent has access to the private key on the client, which makes the system vulnerable to interception by anyone with root access.
The SSH proxy command uses forwarding of the stdin and stdout streams between the bastion and the remote hosts. Suppose we can login to the bastion host from the client with
where the key pair <user-site-key>.pem used to access both hosts is assumed to be the same for simplicity, but different key pairs for each host can easily be configured. We can then first login to the bastion host, and from there login to the remote host, or - using either ProxyCommand or ProxyJump - login to the remote host directly from the client.
SSH proxy based on ProxyJump (from OpenSSH version 7.3) has the compact syntax
Note that there should be no blank spaces surrounding the equality sign = and that the private IP address of the remote host is used. The proxy command can be added the file ~/.ssh/config on the SSH client machine as
Just as with ProxyJump, it is then possible to login to to the remote host from the client with
using the host’s private IP behind the bastion host.
Security best practices
To ensure a high level of integrity and security of the jump host, it should meet the following conditions:
Install a minimal distribution option
Apply patches and updates regularly
Do not host any protocols except for SSH - disable and/or remove unnecessary protocols, daemons, and services
Avoid password authentication and use certificates or key pair authentication, or stronger multi-factor or hardware token authentication (such as YubiKeys)
Never store SSH private keys on the host
Do not use SSH agent forwarding (which is vulnerable to SSH hijacking), but use SSH proxy instead
Set up a services such as fail2ban to prevent brute force attacks
Do not disable strict host check. The SSH property StrictHostKeyChecking in /etc/ssh/ssh_config should be set to
When the StrictHostKeyChecking property is set to yes, SSH will not automatically add host keys to ~/.ssh/known_hosts, and then it is necessary to configure internal hosts with /etc/hosts.allow and /etc/hosts.deny files to control access.
Use a restrictive, host-based firewall on all Linux hosts as complement to OpenStack security groups
Include Host-based Intrusion Detection System (HIDS), for example OSSEC or TripWire
Collect and analyze security logs (stored off-side), and track configuration changes
Create at least one secondary jump host as backup in case of failure; use NAT forwarding to the jump host
To restrict user SSH access and prevent shell apart from the management user (in this example ubuntu), edit the sshd_config file with the following settings:
Match User *,!ubuntu
ForceCommand echo 'This account can only be used for SSH port forwarding'