Bugcheck 0xA/0x1E reported for llt.sys when using Teamed NIC for Infoscale Cluster

book

Article ID: 1000766205

calendar_today

Updated On:

Description

For Infoscale 9.x on Windows platform, when clusters are configured for LLT over Ethernet using Windows NIC Teaming with Load Balancing and Failover mode (LBFO), the cluster nodes may experience intermittent reboots with bugchecks 0xA (IRQL_NOT_LESS_OR_EQUAL) and/or 0x1E (KMODE_EXCEPTION_NOT_HANDLED) pointing to llt.sys 

Cause

From Infoscale 9.x, the LLT module on Windows uses the NDIS 6 network framework.

While leveraging this framework, to avoid copying data on every send, it lets multiple in-flight packets share one underlying data buffer, tracked by a reference count and drawn from kernel memory pools. Under the parallel-transmit load of a Teamed NIC configuration, a narrow race occurs wherein a transmit thread takes a pointer to a shared buffer just before incrementing its reference count, while the original packet is freed on another processor — dropping the count to zero and returning the still-referenced buffer to the pool, where it is overwritten.

The duplicated packet's buffer-base pointer now points into recycled memory, so the buffer length it computes underflows to a gigabyte-scale value. As a result, the kernel faults building that impossible memory descriptor (MDL) at DISPATCH_LEVEL causing the bugcheck 0xA on transmit, while the same pool corruption hands back a bogus allocation on receive mode resulting in the bugcheck 0x1E.

The crash is intermittent, load-dependent and is specific to Teamed NIC configuration only.

Clusters using LLT over UDP or LLT over Ethernet on standalone (non-teamed) NICs are not exposed to this issue.

Resolution

To address this issue, the below options can be explored

Option #1: Reconfigure LLT to use LLT over UDP (IPv4)

Option #2: If LLT over UDP is not possible then reconfigure LLT over Ethernet but with Standalone (non-Teamed) NICs

By mapping each LLT link to an individual NIC instead of a Teamed NIC, LLT redundancy will still be retained as LLT supports multiple heartbeat links. 

Issue/Introduction

For Infoscale 9.x on Windows platform, cluster nodes configured with NIC Teaming for LLT links may experience intermittent reboots with Bugcheck 0xA/0x1E pointing to llt.sys