Deployment

This is a guide to the best practices for deploying Hardware Isolation Layer (HIL). It documents many of the assumptions that the HIL development team made in terms of how HIL would be used, how it would be deployed, and the configuration of the hardware systems it would be managing.

For instructions on installing HIL, take a look at Installation. For a general overview, please take a look at Introduction.

Nodes

BIOS/Firmware

We recommend coming up with a standard list of BIOS settings for your nodes and documenting them. These could vary between pools of different types, like virtualization based (which could benefit more from hyperthreading) vs HPC (which might not).

Settings useful to document would include:

  • BIOS boot order
  • Hyperthreading
  • Whether turbo boosting is enabled

Boot order

A provider should consistently set up all nodes to have the same order. It is recommended that this order consist of: cdrom, disk, PXE. By placing CDROM first, the datacenter administrator can easily boot the system locally in order to perform maintenance tasks like upgrading the BIOS. Having disk come before PXE allows customers who install an OS to have it boot first, reducing boot times and avoiding accidental overwriting of their installed OS.

  • Note: In the future, we anticipate a feature (see issue 303) where customers can explicitly request PXE booting for the next boot via an API call, however the permanent boot order would not likely be able to be changed as that is not supported by IPMI in a standard way.

Devices

For higher security, it may be desirable to disable CD/DVD access to the OS, only enabling it when needed for maintenance or other reasons. This is because an inadvertent CD/DVD left in the drive could allow a user to burn an image. Placed in another system or left in the tray, this image could then allow an attack on another system.

Firmware security

End users will have full control your node. If they are not trusted, it is extremely important to disable firmware update mechanisms before access is first given to it. The less reachable this is through remote access (ie - hardware jumpers vs. software settings) the more secure the system should be.

Ideally, hardware that does not trust the user to update firmware should be used. Examples of this are AMD Seamicro systems as well as HP Moonshot. Short of that, it is recommended that one lock down the firmware options as much as possible in case access were ever gained to the boot screens or IPMI interface. This includes setting a BIOS menu password as well as disabling any kind of privileged access from within the OS.

Firmware updates for the motherboard and for peripheral devices should be disabled via hardware (ideally) or software mechanisms, as they could represent a vector by which a user (whether maliciously or unintentionally) could impact future users of a machine and possibly violate their security. It is important to note that administrators should also monitor the manufacturers’ websites for legitimate firmware upgrades and should regularly update nodes themselves, as these could contain security or other fixes. Here is an incomplete list of firmwares to which these guidelines apply:

  • BIOS Should be configured not to allow updates from within the OS, since even the root user is untrusted.
  • Hard disks
  • Network cards
  • GPUs
  • Other peripherals

Networks

Switch configuration

  • In order to deploy HIL, at least one switch supported by one of HIL’s drivers is required. See the list of supported switches

  • HIL uses the ssh command to connect to the switches. As a consequence, administrators will need to connect to the switch once via ssh, as the user HIL will run as. This is because otherwise ssh will display the usual confirmation dialog for new hosts:

    Are you sure you want to continue connecting (yes/no)?

…which the HIL drivers do not handle.

  • Please make sure that there are no unwanted VLANs on a switchport when it is registered in HIL. There are more details about this in the description for supported switches.

VLANs

The network administrator will need to pre-allocate a set of VLANs to dedicate to HIL’s management. These are kept within the vlan section of the config file.

Headnodes

Currently, the system or systems that run the HIL service (sometimes informally called the HIL master) also run the Headnode virtual machines. These can be used to access and administrate user networks, whether public or private. To facilitate this, all VLANs that are allocatable by HIL must be trunked using 802.1Q to HIL service systems. This port upon which these networks are shared is indicated using the trunk_nic member of the [headnode] section.

Multiple Switches

HIL supports networks spanning multiple switches via 2 mechanisms:

  1. Vendor-specific “stacking” features that make all of the switches appear to be one big one.
  2. By running a network cable between the two switches, and setting all VLANs belonging to HIL to be trunked (tagged) on the interfaces connecting those switches.

IMPORTANT: If you’re operating a multi-switch deployment using technique (2) above, you should be sure to configure the switches to prevent VLAN hopping attacks::

https://en.wikipedia.org/wiki/VLAN_hopping