Error Message

The dmesg file, associated with the crash dump, reports:

[3772231.930881] Kernel panic - not syncing: GAB: Port h halting system due to client process failure at [14:1145]

Based on analysis of the core dump file, memory has been depleted. This can be seen by checking the thread backtrace of the had daemon and displaying the memory status of the machine:

crash> bt ffff91a00d161080

PID: 38004  TASK: ffff91a00d161080  CPU: 0   COMMAND: "had"

 #0 [ffff91a00d4f3608] __schedule at ffffffff833b78d8

 #1 [ffff91a00d4f3670] __cond_resched at ffffffff82cdd526

 #2 [ffff91a00d4f3688] _cond_resched at ffffffff833b7f50

 #3 [ffff91a00d4f3698] shrink_page_list at ffffffff82dde1b7

 #4 [ffff91a00d4f37d0] shrink_inactive_list at ffffffff82ddf376

 #5 [ffff91a00d4f3898] shrink_lruvec at ffffffff82ddfe55

 #6 [ffff91a00d4f3998] shrink_zone at ffffffff82de0286

 #7 [ffff91a00d4f39f0] do_try_to_free_pages at ffffffff82de0780

 #8 [ffff91a00d4f3a68] try_to_free_pages at ffffffff82de0cac

 #9 [ffff91a00d4f3b00] __alloc_pages_nodemask at ffffffff82dd4611

#10 [ffff91a00d4f3c30] alloc_pages_current at ffffffff82e25a48

#11 [ffff91a00d4f3c78] __get_free_pages at ffffffff82dce81e

#12 [ffff91a00d4f3c88] _llt_kmem_alloc at ffffffffc10cbb79 [llt]

#13 [ffff91a00d4f3ca8] _llt_kmem_zalloc at ffffffffc10cbf14 [llt]

#14 [ffff91a00d4f3cc8] llt_ioctl at ffffffffc10cc57d [llt]

#15 [ffff91a00d4f3d38] do_vfs_ioctl at ffffffff82e71988

#16 [ffff91a00d4f3db8] sys_ioctl at ffffffff82e71c21

#17 [ffff91a00d4f3df8] unload_network_ops_symbols at ffffffffc08e5948 [falcon_lsm_pinned_16407]

#18 [ffff91a00d4f3e68] unload_network_ops_symbols at ffffffffc0a24948 [falcon_lsm_pinned_16604]

#19 [ffff91a00d4f3ed8] unload_network_ops_symbols at ffffffffc0a44948 [falcon_lsm_pinned_16703]

#20 [ffff91a00d4f3f50] system_call_fastpath at ffffffff833c539a

    RIP: 00007fd2efb304a7  RSP: 00007ffe69218f10  RFLAGS: 00000212

    RAX: 0000000000000010  RBX: 0000000000000001  RCX: 0000000100000000

    RDX: 00005563024db980  RSI: 0000000080086c1f  RDI: 000000000000000d

    RBP: 00007ffe69217fb0   R8: 0000000001000000   R9: 0000000000001800

    R10: 0000000000000067  R11: 0000000000000202  R12: 00005563019e0b80

    R13: 00007ffe6921e820  R14: 0000000000000000  R15: 0000000000000000

    ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b

crash> kmem -i

                 PAGES        TOTAL      PERCENTAGE

    TOTAL MEM  10230675        39 GB         ----

         FREE    60639     236.9 MB    0% of TOTAL MEM

         USED  10170036      38.8 GB   99% of TOTAL MEM

.

.

Refer to the operating system vendor for further analysis into memory allocated and whether further memory can be added.