UPDATE: Vertica Test Results with Microcode Patches for the Meltdown and Spectre Security Flaws

This blog post was authored by Michael Leuchtenburg.

Introduction

Vertica engineers tested database performance with and without Meltdown and Spectre mitigations on both Haswell and Skylake microprocessors. As compared to no mitigation, with the fastest settings, we found a minimal, average 1% slowdown on Haswell, and an average 25% slowdown on Skylake. These results are different because of the mitigation techniques required for each microarchitecture.

 

Background

In early January, Intel announced two speculative execution security vulnerabilities announced for their processors—Meltdown and Spectre:

 

• Meltdown: Meltdown allows multiple processes on a processor to access the contents of another process’s memory. This vulnerability has only been verified on Intel processors: CVE-2017-5754
• Spectre: Spectre affects processors that implement branch prediction and speculative execution. This vulnerability may allow processes to read and modify the data cache.

Two variants of Spectre has been verified on several modern processors: Variant 1 (CVE-2017-5753) and Variant 2 (CVE-2017-5715). Variant 1 is fixed with a kernel patch. You cannot disable this patch, and there is no measureable performance impact from this patch.

Intel moved the announcement forward because multiple independent discoveries and partial public disclosures made it impossible to delay until their planned date. This resulted in scattershot and slow release of fixes from vendors.

Intel quickly released microcode patches that supported disabling the relevant features of the branch predictors. In some cases, more than just the relevant features are disabled, due to the limits on how much can be changed with a microcode update.

Some of the branch predictor features now need to be controlled—turned on and off—by the operating system, which required changes to Linux and Windows. Fortunately, Linux kernel developers had secretly built this ability into the kernel, allowing Linux to take advantage of the microcode patches to mitigate the attacks. Due to the short timeline, the Linux changes weren’t quite finished at the time of the announcement. Red Hat backported some early versions of these patches to the kernels used in RHEL 6.x and 7.x.

You can read more about the history of the Meltdown and Spectre disclosures, and the reasons why it was hurried, in the article Keeping Spectre Secret.

The first versions of microcode patches from Intel caused instability in many systems. Intel described the instability as “increased reboots”, aka crashes. Vertica heard reports from customers about crashes on systems with the microcode patches. At that time, we also found that the performance impact of fully mitigating the vulnerabilities was very large, as much as a 3x increase in execution time for the worst affected queries that we tested, as described in Vertica Test Results for Operating System Patches for Meltdown and Spectre Security Flaws. Intel stopped shipping the microcode patches, as did vendors.

Today

Since then, there have been several improvements in the situation. Intel fixed the stability problems with the microcode patches. Kernel engineers have also made improvements to the performance impact of the mitigations.

On pre-Skylake processors, an alternative to the Indirect Branch Restricted Speculation (IBRS) option for protecting kernel data structures against Spectre attacks is using the Retpoline (retp) mitigation strategy.

Using Retpoline is a mitigation strategy developed by the Google Project Zero team that worked on the Spectre attacks. The idea is to avoid using indirect JMP or CALL instructions that would be vulnerable to Spectre. Instead, use a RET instruction to call a function. RET is normally used for returning from a call, so this is a somewhat unintuitive workaround. This approach works because, in pre-Skylake processors, the branch prediction mechanism doesn’t touch RET instructions.

To read more about Retpoline, check out these links:

• https://support.google.com/faqs/answer/7625886
• https://software.intel.com/sites/default/files/managed/1d/46/Retpoline-A-Branch-Target-Injection-Mitigation.pdf

Red Hat started shipping retpoline-enabled kernels in March. With that change, the default for pre-Skylake processors is now

pti=1 ibrs=0 retp=1 ibpb=1

Skylake processors have a new branch prediction method that results in retpoline not being sufficient to protect against Spectre attacks. On these processors, the default settings are:

pti=1 ibrs=1 retp=0 ibpb=1

The first high-end Skylake server processors shipped in summer 2017 in the Intel Scalable Xeon line.

Performance Tests

Vertica engineers have retested with the new kernels and microcode. The results are significantly improved. On our clusters with Haswell (which is pre-Skylake) processors, the mean performance impact with the default security settings is only a 1% slowdown, with a worst-case slowdown of 4%. With IBRS on this cluster, the mean slowdown is 118%, with a worst-case of 246%.

On Skylake systems, the outlook is not as rosy because we must use IBRS. Fortunately, it’s not quite as bad as IBRS on Haswell. In the default configuration, as compared to having all mitigations disabled, we saw a mean slowdown of 25%, with a worst-case slowdown of 80%.

It’s clear that with IBRS enabled, the impact is still very large. Vertica engineers believe that there is more work for Red Hat to do in backporting the changes to the Linux kernel to support IBRS. In particular, with hyperthreading, IBRS is enabled for both threads on a core if IBRS is enabled for either thread. With IBRS=1, IBRS is supposed to be turned on for all kernel threads but off for all userspace threads. However, if a userspace thread is running on the same core, it also has IBRS enabled. The result is that the userspace thread runs with many important branch prediction features disabled. The results can be seen with a trivial userspace program. See the fib, which you can build with the following commands:

gcc -O3 -march=native -o fib
echo 0 > /sys/kernel/debug/ibrs_enabled
time ./fib 100000000
echo 1 > /sys/kernel/debug/ibrs_enabled
time ./fib 100000000

In our tests, we saw a slowdown of as high as 10x for fib.c. In the upstream Linux kernel, when a kernel thread goes idle, IBRS is disabled on that core. Without this, IBRS is on for all userspace programs as well, with negative consequences for performance. This implies that we can expect to see further improvements in performance in the future.

Given the improvement since the tests were performed in January, customers should consider upgrading to the latest patch of their operating system. Because of the difference in test results between Skylake and Haswell, customers building new clusters should explore whether they will see better overall performance on the newer chips or should stick with the older ones.

FAQs

Q: What mitigation settings does Vertica recommend?

A: Use the default Linux kernel settings for pti, ibrs, ibpb, and retp. They address the security issue with the minimal impact on performance.

Q: Do you plan to build Vertica using retpoline?

A: Not in the immediate future. To implement retpoline properly, every other program on your system will also need to be rebuilt.

Q: What does the performance look like on AMD?

A: Vertica engineers have not run any tests on AMD hardware yet.

Detailed Test Results

The following graphs show the detailed test results. All tests were run with the current development version of Vertica.

Haswell

The Haswell cluster was configured as follows:

• 4 nodes
• 2 x Xeon E5-2698 v3 @ 2.3GHz
• 252GB RAM
• 13TB SAS disk array
• Red Hat 7.5 with kernel version 3.10.0-862.el7.x86_64

The query run-time results for the Haswell cluster were as follows:

Skylake

The Skylake cluster was configured as follows:

• 3 nodes
• 2 x Xeon Scalable Gold 5120 @ 2.2GHz
• 384GB RAM
• 3.8TB local NVMe
• CentOS 7.4 with kernel version 3.10.0-693.21.1.el7.x86_64

This is somewhat different than the Haswell cluster. Since all comparisons are of this cluster to itself, there are no problems with the differences between clusters.

The query run-time results for the Skylake cluster were as follows: