Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

clayfreeman · 2026-03-13 19:14:38

After upgrading to linux-hardened 6.18.17.hardened1-1, my machine started kernel panicking on boot:

[   19.761411] kernel BUG at mm/slub.c:925!
[   19.761688] ------------[ cut here ]------------
[   19.762503] ------------[ cut here ]------------
[   19.762508] kernel BUG at mm/slub.c:925!
[   19.762519] Oops: invalid opcode: 0000 [#1] SMP PTI
[   19.762529] CPU: 27 UID: 0 PID: 334 Comm: (exec-inner) Not tainted 6.18.17-hardened1-1-hardened #1 PREEMPT(full)
[   19.762538] Hardware name: Dell Inc. PowerEdge R650xs
[   19.762541] RIP: 0010:kmem_cache_free+0x652/0x690
[   19.762556] Code: 61 fc ff ff 84 c0 0f 84 01 fd ff ff eb e9 48 89 d6 4c 89 ef e8 af e0 ff ff 48 89 c2 48 85 c0 0f 85 71 fd ff ff e9 a3 fb ff ff <0f> 0b 0f 0b 84 c0 0f 84 8f fe ff ff eb bf 41 39 55 30 44 0f 42 ca
[   19.762560] RSP: 0018:ff89147d882077a0 EFLAGS: 00010287
[   19.762567] RAX: 0000000000000000 RBX: ff3c3f8d29de7100 RCX: 00000000000000c0
[   19.762571] RDX: 00000000000000c0 RSI: ffffffe66ab57100 RDI: 6bbf64668b259d00
[   19.762574] RBP: ff89147d88207800 R08: 0000000000000000 R09: 00000000000000c8
[   19.762577] R10: 0000000000000000 R11: 00000000000007e0 R12: ffffffffaa62a0bf
[   19.762580] R13: ff3c3f87801fd2c0 R14: ffd1bdd99aa77980 R15: 0000000000000001
[   19.762583] FS:  0000000000000000(0000) GS:ff3c3fa712b17000(0000) knlGS:0000000000000000
[   19.762588] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.762591] CR2: 0000001b91bf76c8 CR3: 00000015de19e003 CR4: 0000000000771ef0
[   19.762595] PKRU: 55555554
[   19.762597] Call Trace:
[   19.762601]  <TASK>
[   19.762604]  ? preempt_count_add+0x55/0xd0
[   19.762619]  exit_mmap+0x1bf/0x3c0
[   19.762633]  __mmput+0x4f/0x160
[   19.762643]  begin_new_exec+0x4e3/0xd10
[   19.762649]  ? load_elf_phdrs+0x6b/0xb0
[   19.762658]  load_elf_binary+0x31c/0x18a0
[   19.762666]  ? __kernel_read+0x266/0x320
[   19.762673]  ? __kernel_read+0x266/0x320
[   19.762679]  ? preempt_count_add+0x7c/0xd0
[   19.762687]  ? _raw_read_lock+0x17/0x40
[   19.762698]  ? _raw_read_unlock+0x1c/0x40
[   19.762707]  bprm_execve+0x2ac/0x510
[   19.762718]  do_execveat_common.isra.0+0x194/0x1a0
[   19.762729]  __x64_sys_execve+0x38/0x50
[   19.762739]  do_syscall_64+0x84/0x750
[   19.762767]  ? clear_bhb_loop+0x30/0x80
[   19.762775]  ? clear_bhb_loop+0x30/0x80
[   19.762781]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   19.762787] RIP: 0033:0x6868c91c468b
[   19.762796] Code: Unable to access opcode bytes at 0x6868c91c4661.
[   19.762799] RSP: 002b:00007528e163f188 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
[   19.762805] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00006868c91c468b
[   19.762808] RDX: 0000001b91bf33d0 RSI: 00007528e163f3c0 RDI: 0000001b91bf3850
[   19.762810] RBP: 00007528e163f300 R08: 0000000000000020 R09: ae25f7cc08447ae8
[   19.762813] R10: 0000000000000021 R11: 0000000000000206 R12: 0000001b91bfb3d0
[   19.762816] R13: 0000000000000000 R14: 00000000fffffff7 R15: 0000000000000000
[   19.762822]  </TASK>
[   19.762824] Modules linked in:
[   19.762829] ---[ end trace 0000000000000000 ]---
[   19.765364] Oops: invalid opcode: 0000 [#2] SMP PTI
[   19.831333] kernel BUG at mm/slub.c:925!
[   19.844755] pstore: backend (erst) writing error (-28)
[   19.844757] RIP: 0010:kmem_cache_free+0x652/0x690
[   19.844760] Code: 61 fc ff ff 84 c0 0f 84 01 fd ff ff eb e9 48 89 d6 4c 89 ef e8 af e0 ff ff 48 89 c2 48 85 c0 0f 85 71 fd ff ff e9 a3 fb ff ff <0f> 0b 0f 0b 84 c0 0f 84 8f fe ff ff eb bf 41 39 55 30 44 0f 42 ca
[   19.844762] RSP: 0018:ff89147d882077a0 EFLAGS: 00010287
[   19.844764] RAX: 0000000000000000 RBX: ff3c3f8d29de7100 RCX: 00000000000000c0
[   19.844765] RDX: 00000000000000c0 RSI: ffffffe66ab57100 RDI: 6bbf64668b259d00
[   19.844766] RBP: ff89147d88207800 R08: 0000000000000000 R09: 00000000000000c8
[   19.844767] R10: 0000000000000000 R11: 00000000000007e0 R12: ffffffffaa62a0bf
[   19.844768] R13: ff3c3f87801fd2c0 R14: ffd1bdd99aa77980 R15: 0000000000000001
[   19.844769] FS:  0000000000000000(0000) GS:ff3c3fa712b17000(0000) knlGS:0000000000000000
[   19.844770] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.844771] CR2: 00006868c91c4661 CR3: 00000015de19e003 CR4: 0000000000771ef0
[   19.844772] PKRU: 55555554
[   19.844773] Kernel panic - not syncing: Fatal exception
[   20.912928] Shutting down cpus with NMI
[   21.276565] Kernel Offset: 0x29000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   21.394060] ---[ end Kernel panic - not syncing: Fatal exception ]---

Downgrading to linux-hardened 6.17.13.hardened1-3 resolves the issue.

Is anybody able to help me diagnose?

Last edited by clayfreeman (2026-03-13 19:25:33)

cryptearth · 2026-03-13 19:30:40

system specs would be helpful - what cpu EXACTLY? multi-socket board?

clayfreeman · 2026-03-13 19:37:55

2x Intel(R) Xeon(R) Silver 4309Y CPUs and 4x Kingston 9965694-026.B00G in a Dell PowerEdge R650xs with a 019H6N motherboard.

cryptearth · 2026-03-13 19:47:37

according to intel the cpu are specced for ddr4-2667 but the memory is ddr4-3200
check the bios for ram timings - could be an overclock issue

clayfreeman · 2026-03-13 19:51:42

While running on linux-hardened 6.17.13.hardened1-3:

	Speed: 3200 MT/s
	Configured Memory Speed: 2666 MT/s
	Speed: 3200 MT/s
	Configured Memory Speed: 2666 MT/s
	Speed: 3200 MT/s
	Configured Memory Speed: 2666 MT/s
	Speed: 3200 MT/s
	Configured Memory Speed: 2666 MT/s

Do you think there's a chance that linux-hardened 6.18.17.hardened1-1 ends up with incorrectly-clocked memory somehow?

cryptearth · 2026-03-13 23:56:38

you missed the point: do not rely on what ANY os gets you - but reboot the machine into its bios/uefi and check there what the ram is set to
the point is: whoever configured the system didn't look into the cpu spec sheet: intel limits them to ddr4-2667 - likely for a good reason
the ram, however, is designed to run at ddr4-3200 - which is the upper jedec limit and above the cpu spec
now there're three options:
- run minimum jedec: that's ddr4-1600 and would waste both the cpu and ram
- run at cpu max of 2667 - but this has to be supported by the ram - if it doesn't have a profile for 2667 it may become unstable (the oroblem you already encounter)
- run over cpu spec at full 3200 - which maybe the bios lets you to set as target but the cpu unlikely can handle - or requires some overclock setting - which xeons usually don't like

4th option: get two new cpus which support the ram up to 3200

it could be that the ram and cpu are just inoperable with eachother - and that whoever not read the specs made a costly mistake
the point it ran well until now isn't because of linux - but it shows up now because the most recent kernel somehow triggers the seemingly instability - but it's still a hsrdware incompatibility

clayfreeman · 2026-03-15 20:19:10

The BIOS does not allow me to select a different speed, and locks me to 2666 MT/s.

clayfreeman · 2026-03-15 20:43:02

Switching to linux 6.19.8-arch1-1 resolves the issue, which seemingly indicates a bug in the hardening patchset.

bediger4000 · 2026-03-17 17:28:02

For what it's worth, I upgraded to linux-hardened 6.19.8.hardened1-2 (from 6.17.13-hardened) this morning and got a kernel panic. I recovered because I also have linux-lts installed, and the GRUB menu let me boot that kernel, 6.18.18-1-lts . I do have intel-ucode 20260227-1 package installed.

Dell PowerEdge R530, serial no. 7XM3942. DMI base board: Dell Inc. 03XKDV, version A01, a Dell, like clayfreeman reported.

CPU Info from dmidecode:

Socket Designation: CPU1
Type: Central Processor
Family: Xeon
Manufacturer: Intel
ID: F2 06 03 00 FF FB EB BF
Signature: Type 0, Family 6, Model 63, Stepping 2
Version: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
Voltage: 1.3 V
External Clock: 9600 MHz
Max Speed: 4000 MHz
Current Speed: 2600 MHz
Status: Populated, Enabled
Upgrade: Socket LGA2011-3

Socket Designation: CPU2
Type: Central Processor
Family: Xeon
Manufacturer: Intel
ID: F2 06 03 00 FF FB EB BF
Signature: Type 0, Family 6, Model 63, Stepping 2
Version: Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
Voltage: 1.3 V
External Clock: 9600 MHz
Max Speed: 4000 MHz
Current Speed: 2600 MHz
Status: Populated, Enabled
Upgrade: Socket LGA2011-3

I'm not sure what other info to provide. This is the first kernel panic I've had since buying the machine in 2020. According to ipmitool, it was assembled in 2014, so it is getting old. I'm certain the rest of the software and configuration is so idiosyncratic as to prevent explanation.

seth · 2026-03-17 21:52:53

https://github.com/anthraxx/linux-hardened/issues

[   19.765364] Oops: invalid opcode: 0000 [#2] SMP PTI
[   19.831333] kernel BUG at mm/slub.c:925!
[   19.844755] pstore: backend (erst) writing error (-28)
[   19.844757] RIP: 0010:kmem_cache_free+0x652/0x690

https://github.com/anthraxx/linux-hardened/commits/6.19 seems to have some related patches, can you post the panic?

@clayfreeman, is this still a thing w/ 6.19.8 for you?

loqs · 2026-03-17 23:25:27

seth wrote:

https://github.com/anthraxx/linux-hardened/commits/6.19 seems to have some related patches

1a2b4b7 slub: add multi-purpose random canaries looks to be the relevant patch. Which has moved from line 9.25 in 6.18.17-hardened1 to line 921 in 6.19.8-hardened1.

Last edited by loqs (2026-03-18 00:49:47)

Arch Linux

#1 2026-03-13 19:14:38

Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#2 2026-03-13 19:30:40

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#3 2026-03-13 19:37:55

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#4 2026-03-13 19:47:37

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#5 2026-03-13 19:51:42

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#6 2026-03-13 23:56:38

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#7 2026-03-15 20:19:10

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#8 2026-03-15 20:43:02

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#9 2026-03-17 17:28:02

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#10 2026-03-17 21:52:53

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

#11 2026-03-17 23:25:27

Re: Upgrade to linux-hardened 6.18.17.hardened1-1 results in kernel panic

Board footer