OpenSSL SSL_sendfile performance with nginx on Linux

OpenSSL’s implementation of SSL_sendfile uses kernel TLS encryption to optimize away user space copying of file data before sending on a socket.

So from:

  • FILE –> <user-space application> –> socket

to just

  • FILE –> socket

This is sort of an encryption version of the sendfile system call.

Running nginx and curling

nginx-1.23.2>strace -f ./objs/nginx -c ./nginx.conf ...
...
[pid 408582] setsockopt(13, SOL_TCP, TCP_ULP, [7564404], 4) = 0
...
[pid 408582] setsockopt(13, SOL_TLS, TLS_TX, "\4\0034\0$F\273F\266\232\"\25M\v\33\257\31\366\252\210Q)>\225\200\216\235\300\341c\300T"..., 56) = 0
[pid 408582] openat(AT_FDCWD, "/home/rmorrison/data/www/rand_1MB.bin", O_RDONLY|O_NONBLOCK) = 14
[pid 408582] newfstatat(14, "", {st_mode=S_IFREG|0664, st_size=1048576, ...}, AT_EMPTY_PATH) = 0
[pid 408582] fadvise64(14, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
[pid 408582] write(3, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 260) = 260
[pid 408582] sendfile(3, 14, [0] => [1048576], 1048576) = 1048576
[pid 408582] write(5, "127.0.0.1 - - [21/Feb/2023:17:27"..., 112) = 112
[pid 408582] close(14)                  = 0
...

See “Kernel TLS” guide to implementing applications with kTLS.

sendfile has been around for a long time, but Linux kernel support for doing encryption in the kernel was only added in 2015. OpenSSL has opted to only add support for kTLS and the SSL_sendfile function in their 3.x+ series of releases, meaning OpenSSL 1.x+ doesn’t currently have support.

Kernel Support

First things first, will need a kernel with kTLS support. “Kernel TLS offload” was introduced in version 4.13. Only some cipher suites are available for use with kTLS, and cipher support is somewhat varied by OS/version.

The mainline support for ciphers as of this post seems to be: link

const struct tls_cipher_size_desc tls_cipher_size_desc[] = {
	CIPHER_SIZE_DESC(TLS_CIPHER_AES_GCM_128),
	CIPHER_SIZE_DESC(TLS_CIPHER_AES_GCM_256),
	CIPHER_SIZE_DESC(TLS_CIPHER_AES_CCM_128),
	CIPHER_SIZE_DESC(TLS_CIPHER_CHACHA20_POLY1305),
	CIPHER_SIZE_DESC(TLS_CIPHER_SM4_GCM),
	CIPHER_SIZE_DESC(TLS_CIPHER_SM4_CCM),
};
...

This post from the nginx team is replete with information on enabling kTLS support.

On Linux ensure the tls module is loaded with either (as root):

modprobe tls

or add to /etc/modules to ensure mod_tls is loaded on every boot:

>cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
tls

Can verify after is loaded with lsmod command:

>lsmod | grep tls
tls                   102400  0

Building OpenSSL

The OpenSSL version I was testing with:

openssl version
OpenSSL 3.0.8-dev 1 Nov 2022 (Library: OpenSSL 3.0.8-dev 1 Nov 2022)

Building OpenSSL with kTLS support is interesting? This might be out of date information but I found it to be true that OpenSSL does not enable kTLS support by default for their tool chain, and it must be configured with a custom openssl.cnf configuration. Just something to be wary of if using openssl commands on the command line (eg. openssl s_server with the -sendfile option). For example to run s_server with kTLS enabled and using a custom build of OpenSSL:

OPENSSL_CONF=<custom_openssl.cnf file> \
LD_LIBRARY_PATH=<path to custom openssl> \
openssl s_server \
  -key server.key \
  -cert server.crt \
  -accept 12345 \
  -www \
  -sendfile

NOTE the OPENSSL_CONF and LD_LIBRARY_PATH env vars in the command above.

I added the following lines to a generic openssl.cnf file:

+ openssl_conf = my_openssl_conf

+ [ my_openssl_conf ]
+ ssl_conf = my_ssl_conf

+ [ my_ssl_conf ]
+ ktls = my_ktls_conf

+ [ my_ktls_conf ]
+ Options = KTLS

This also means a custom OpenSSL library will have to be built and linked with nginx since by default, most versions available on systems will either not have the feature (if OpenSSL version < 3.0.0) or it won’t have been enabled by default.

To configure OpenSSL to build with kTLS support, (in the OpenSSL repo dir) run:

openssl>./config -d enable-ktls

Can verify after run feature is enabled with:

openssl>./configdata.pm -o
...
Enabled features:
...
    ktls
...

Then build as usual:

openssl>make -j$(nproc)
...

If there’s issues with building/configuring/testing kTLS support with OpenSSL please see this GitHub Issue which has lots of troubleshooting advice from OpenSSL dev’s and users. It helped me a lot.

Building nginx and enabling

To configure building nginx with a custom built OpenSSL library (in the nginx source tree)

nginx-1.23.2>./configure \
  --with-http_ssl_module \
  --with-openssl=<path_to_custom_openssl_src_tree> \
  --with-openssl-opt=enable-ktls

I wanted pcre and h2 support as well so my specific configure line was:

nginx-1.23.2>./configure \
  --with-pcre-jit \
  --with-pcre \
  --with-http_v2_module \
  --with-http_ssl_module \
  --with-openssl=/home/rmorrison/archive/openssl \
  --with-openssl-opt=enable-ktls

Then just build as usual.

One thing to note in the nginx code is how nginx tests if kTLS is enabled on the socket with the BIO_get_ktls_send function:

BIO_get_ktls_send() returns 1 if the BIO is using the Kernel TLS data-path for sending. Otherwise, it returns zero. BIO_get_ktls_recv() returns 1 if the BIO is using the Kernel TLS data-path for receiving. Otherwise, it returns zero.

To configure the server to enable SSL_sendfile with kTLS (with an ssl listener)

sendfile on;
ssl_conf_command Options  KTLS;

Here’s an abridged version of my local test config:

worker_processes 1;
daemon off;
events {
    worker_connections 1024;
}
http {
    error_log /var/tmp/nginx/error.log error;
    include            mime.types;
    default_type       application/octet-stream;
    sendfile           on;
    read_ahead         1;
    tcp_nopush         on;
    tcp_nodelay        on;
    keepalive_timeout  3600;
    keepalive_requests 10000;
    server {
        listen                    12345 ssl;
        ssl_certificate           /home/rmorrison/data/conf/certs/my_default.crt;
        ssl_certificate_key       /home/rmorrison/data/conf/certs/my_default.key;
        ssl_conf_command Options  KTLS;
        ssl_protocols             TLSv1.2 TLSv1.3;
        server_name               myserver;
        access_log                /var/tmp/nginx/access.log;
        client_body_temp_path     /var/tmp/nginx/client_body_temp;
        proxy_temp_path           /var/tmp/nginx/proxy_temp;
        location / {
            root  /home/rmorrison/data/www/;
            index index.html;
        }
    }
}

Load Testing

Testing over localhost with hurl on a Xeon Gold Server:

>cat /proc/cpuinfo | grep 'model name' | uniq
model name	: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz

Testing is meant to demonstrate relative performance differences with and without SSL_sendfile.

Note:

hurl is my tool, but I checked with wrk and got similar numbers. I chose hurl so I could tune the number of requests per connection per run.

Running

hurl was run with:

hurl 'https://localhost:12345/<resource>' --silent --threads=4 --parallel=4 --seconds=10 --calls=<num_calls>

Where the variables were:

  • the resource sizes (1kB, 8kB, …)
  • the number of requests (calls) per connection (1, 10, 20, …)

For example

hurl 'https://localhost:12345/rand_8kB.bin' --silent --threads=4 --parallel=4 --seconds=10 --calls=10
| RESULTS:             ALL
| fetches:             82047
| max parallel:        4
| bytes:               7.010106e+08
| seconds:             10.00
| mean bytes/conn:     8544.01
| fetches/sec:         8202.24
| bytes/sec:           7.008004e+07
| HTTP response codes: 
| 200 -- 82047

img

img

img

img

img

img

img

Summary

The results are mostly what I expected from a sendfile like optimization, in that the speed up becomes more pronounced as the time to connect/handshake fades into the background and more time is spent in the symmetric cryptographic sending and receiving of file data. With larger files (> 1MB) the results can be dramatically better with ~60-70% improved throughput.

What’s interesting also is performance of SSL_sendfile with smaller files (1-8kB) is a little worse than just SSL_write. I haven’t dug into why yet, but might be something to keep track of and maybe avoid if size can be read ahead of serving.

References

Written on February 21, 2023