High CPU usage and degraded performance of VxVM due to the use of block-level queueing

book

Article ID: 100054475

calendar_today

Updated On:

Cause

For some workloads, specifically small random reads, the block-level queuing used by VxVM Volumes can restrict throughput and degrade performance. VxVM does not fully implement Multi-Queue Block IO Queueing Mechanism (blk-mq). The current implementation of VxVM request-mode converts incoming 'request' into a 'bio', then processes the 'bio' and submits it to the linux block layer. 

This issue has been observed after applying VRTSvxvm-8.0.0.1800 on a system running RHEL 8.6, but isn't limited to these versions.

 

Resolution

The current workaround is to disable the usage of block-level queueing using the tunable attribute 'vol_use_rq'. This can be done with the following command. 
Note: The command can be executed at any time, but the change will only propagate after all VxVM diskgroups have been deported and imported. A system reboot will also work.
# vxtune vol_use_rq 0

To verify the current setting use the following command.
# vxtune vol_use_rq

The setting is persistent across reboots, and can be seen in the file 'vxtunables'.
# grep vol_use_rq /etc/vx/vxtunables
vol_use_rq=0;


For a long term solution the VxVM development team is reviewing the OS block layer code to make appropriate updates. The required changes are complex and reside in a critical code path which is needed to process all IO requests. Work is being done to plan for the complex code changes and perform extensive testing to avoid  potential code regressions.

 

Related article from RedHat Support: https://access.redhat.com/solutions/6983206

Issue/Introduction

By default VxVM will use block-level queueing to handle IO requests which can lead to degraded performance under certain conditions.

Additional Information

JIRA: STESC-7524