hyperthreading disabled, striping over 4 NVMe devices, stripe size 64k, use the
psync ioengine.

This is slightly slower than with the libaio engine.
QAT is slower than AES-NI

fio --ioengine=psync --iodepth=1 --rw=randread --direct=1 --end_fsync=1 --bs=64k --numjobs=56 --time_based --runtime=10 --group_reporting --name=job --filename=/dev/mapper/rhel_cidic1-striped
fio --ioengine=psync --iodepth=1 --rw=randwrite --direct=1 --end_fsync=1 --bs=64k --numjobs=56 --time_based --runtime=10 --group_reporting --name=job --filename=/dev/mapper/rhel_cidic1-striped

raw device:
READ: bw=10.8GiB/s
busy: 1.771817
idle: 97.563791
irq:  0.388178

WRITE: bw=4248MiB/s
busy: 0.826378
idle: 98.723663
irq:  0.189774

dm-crypt, aes-ni, 4096:
READ: bw=10.5GiB/s
busy: 7.353838
idle: 91.028772
irq:  0.771635

WRITE: bw=4255MiB/s
busy: 11.838650
idle: 88.081557
irq:  0.445179

dm-crypt, qat, 4096:
READ: bw=1761MiB/s
busy: 3.424129
idle: 92.964313
irq:  1.871788

WRITE: bw=1765MiB/s
busy: 3.328441
idle: 92.982500
irq:  1.916009

dm-crypt, aes-ni, 512:
READ: bw=10.5GiB/s
busy: 10.126000
idle: 88.375505
irq:  0.693940

WRITE: bw=4250MiB/s
busy: 11.185064
idle: 85.575740
irq:  0.441334

dm-crypt, qat, 512:
READ: bw=290MiB/s
busy: 3.361823
idle: 93.425560
irq:  1.766294

WRITE: bw=250MiB/s
busy: 3.040340
idle: 93.771378
irq:  1.770084