Aug 20 ’10

Using Hardware Cryptographic Support With OpenSSH in Linux on System z

by Editor in z/Journal

Setting up secure, protected servers and network environments has become vital as companies seek to comply with regulations from governments and other organizations to implement specific levels of security compliance, including protection of network data traffic.

Common access methods for Linux servers such as Telnet, or other protocols for file transfer such as File Transfer Protocol (FTP), aren’t adequate in Internet environments and internal company networks because sensitive information such as passwords is transferred over the network in the clear. Using Secure Shell (SSH), Secure Copy (SCP), and Secure File Transfer Protocol (SFTP) increases security because encryption protects both passwords and data. Although data encryption is expensive and can severely impact system performance, the IBM System z provides hardware encryption facilities that can reduce these adverse effects.

OpenSSH is a free version of the SSH connectivity tools included in the Linux on System z distributions. Starting with OpenSSH version 4.4, OpenSSL dynamic engine loading is supported; it lets OpenSSH benefit from System z cryptographic hardware support if a specific option is used during the OpenSSH package build. This article describes experiences with such an OpenSSH package and demonstrates the value of hardware encryption support for SSH sessions and SCP file transfers. In our testing, we used a System z10 Enterprise Class (EC) with Novell SUSE Linux Enterprise Server (SLES) 11 for IBM System z Service Pack 1 (SP1), as this version of Linux contains the updated OpenSSH package.

Hardware Cryptographic Support of System z

System z provides two different types of hardware support for cryptographic operations: Central Processor Assist for Cryptographic Function (CPACF) and Crypto Express features.

The first type, CPACF, is incorporated into the central processors shipped with the System z, and was introduced with System z990 and System z890. CPACF supports several symmetric encryption algorithms:

CPACF algorithms execute synchronously and are for clear key operation (i.e., the calling application provides cryptographic keys in unencrypted format).

The second type uses installable Crypto Express features, either Crypto Express2 (z9 and z10 up to GA2) or Crypto Express3 (z10 GA3). A Crypto Express feature can be configured either as a cryptographic accelerator to perform clear key RSA operations at high speed or as a cryptographic coprocessor to perform symmetric and asymmetric operations (RSA) in clear key and secure key mode. Crypto Express operations are performed asynchronously outside the central processor; work is partially off-loaded, and CPU cycles for cryptographic operations are reduced compared with the equivalent execution of a pure software implementation.

To benefit from CPACF, you must install the free of charge LIC internal feature 3863 (Crypto Enablement);  this isn’t installed on the machine by default at delivery, but can be later installed without disrupting system operation. There’s no downside to installing CPACF, so it’s recommended for all systems.

To verify that CPACF is enabled, use the Support Element (SE) or Hardware Management Console (HMC) and look for “CP Assist for Crypto functions: Installed” in the CPC details panel (see Figure 1). Or, if you’re a Linux on System z user, you can easily check whether the feature is installed and which algorithms are supported. The icainfo command of library libica displays which CPACF functions are supported (see Figure 2).


Configuring the Crypto Express Feature for Linux on System z

The Crypto Express2 and Crypto Express3 features add hardware support for SSH session “handshake” acceleration. To benefit from this, you must enable Linux guests under z/VM for access to Crypto Express. The Logical Partition (LPAR) activation profile must contain at least one processor of a Crypto Express feature in the cryptographic candidate list and at least one usage domain index. Starting with System z10, LPAR activation profiles can be modified without deactivating the LPAR.

Linux Cryptographic Architecture Used by OpenSSH

OpenSSH uses OpenSSL to perform cryptographic operations. If the OpenSSH package is built using the “--with-ssl-engine” option, the OpenSSL library will use the ibmca cryptographic engine, if installed, to perform encryption operations. The ibmca engine uses the libica library to handle the requests; this library is aware of which algorithms the available hardware supports and passes requests to the hardware as appropriate, instead of performing the operation in software. Running under z/VM has no impact on the cryptographic architecture inside the Linux server other than requiring access to installed Crypto Express hardware via the z/VM directory for the Linux guest. Figure 3 shows the architecture.

Enabling OpenSSH hardware cryptographic support from Linux on System z requires installing the following software and driver packages. These are all part of the Linux on System z distribution and may be installed by default:

Preparing OpenSSL for ibmca Engine Use

The OpenSSH package shipped with SLES 11 SP1 automatically uses hardware cryptographic support, if OpenSSL is configured for dynamic loading of the ibmca engine.

SLES 11 SP 1 ships with dynamic engine support disabled. To enable it, you must modify the OpenSSL configuration file after installing all required software packages. A sample openssl.cnf.sample file is provided with the openssl-ibmca package and contains settings to enable ibmca. To customize OpenSSL to enable ibmca:

  1. Make a backup copy of openssl.cnf.
  2. Append the content of sample file /usr/share/doc/packages/openssl-ibmca/openssl.cnf.sample to the existing openssl.cnf.
  3. Move the line “openssl_conf = openssl_def” from the appended part to the top of the configuration file. The new configuration file should resemble Figure 4.
  4. Check the value of the “dynamic_path” variable and change it as necessary to the correct path for (this path varies, depending on the Linux distribution in use).


To verify whether dynamic engine support for ibmca is enabled, use the “openssl engine -c" command as shown in Figure 5, which will show ibmca status and a list of supported algorithms. To disable dynamic engine loading of ibmca, comment out the “openssl_conf = openssl_def” line at the top of openssl.cnf. Once dynamic engine loading of ibmca engine is enabled, any OpenSSH activity will automatically use any available hardware cryptographic support.


Verify CPACF Usage

To verify whether CPACF is used for a Linux server on System z, use the icastats tool, which shows the number of executed encryption requests the libica library handled. It distinguishes between requests CPACF executes and those executed by software fallback.

Figure 6 shows an example of icastats output. Here, CPACF executed AES and SHA requests; since no Crypto Express was available, RSA handshakes were executed in software.


Using OpenSSH With Hardware Crypto Support

Transferring large files using SCP shows the influence of using dynamic engine loading support of OpenSSL on OpenSSH, as SCP uses OpenSSH under the covers.

Transferring data from a PC to the host using the network—which is complex, with many hops between the PC and the host—showed no throughput increase by enabling ibmca, but host CPU usage during the transfer dropped dramatically, from about 70 percent to 15 percent. This suggested a network bottleneck; repeating the test using SCP within the Linux host, using “localhost” as the target address and /dev/null as the output, increased throughput when ibmca was enabled.

Since CPACF doesn’t support all ciphers and Message Authentication Codes (MACs), it’s important to select appropriate ciphers and MACs; SCP allows specification of both.

We performed more tests using various ciphers: Triple DES, AES-128, AES-192 and AES-256, all of which benefit from CPACF. The MD5 MAC is always executed in software on System z, since CPACF doesn’t support it, whereas SHA-1 can also benefit from CPACF. Figure 7 shows the results of operations with and without dynamic engine loading support active using TDES and AES ciphers and MD5 and SHA-1 MACs. The user time of the execution of the scp command is an  indication of the CPU cycles consumed. It’s clear that CPACF dramatically decreases CPU consumption for TDES and AES. MD5 is the faster MAC in software, but with CPACF, SHA-1 contributes to another 30 percent gain. Using CPACF clearly frees up cycles that can be used for other workloads.


Determine Ciphers/MAC Via Profile

Most SSH or SCP users won’t explicitly specify the cipher or MAC, so it’s worth considering modifying the configuration files for the Linux SSH client (ssh_config) and SSHD server (sshd_config) to default to using algorithms that will benefit from hardware crypto support. Using the Ciphers and MACs keyword lets you specify the algorithms that will benefit from hardware at the top of the search order: AES or TDES as the top cipher, and SHA as the top MAC. If the hardware is System z9, choose TDES or AES-128 rather than AES-192 or AES-256 because the latter two aren’t supported by CPACF until z10.

Influence of Crypto Express

Having a Crypto Express feature helps during session initialization for asymmetric RSA requests. The SCP test represents a relatively long-running session with only one RSA handshake, so the effect of an active Crypto Express feature was minimal (a difference of 0.01 seconds for user time). The benefit of a Crypto Express feature for OpenSSH is greater if a high number of short-running sessions are established simultaneously, as is common with some Web applications.


Using hardware encryption support in combination with OpenSSH can save significant CPU cycles, leading to better performance. At a minimum, CPU load will decrease for encryption workloads. Tests indicated a user time reduction for TDES by a factor 15 and a factor of 11 for AES. Selecting SHA as MAC also was beneficial. Use of CPACF is free of charge and its enablement is strongly encouraged. If a Crypto Express feature is available, make it available to Linux systems that can benefit from it.