(the lockups are caused by excessive "integrity: Checksum failed" messages) [ 4770.972041] device-mapper: integrity: Checksum failed at sector 0x1172d7 [ 4771.002682] device-mapper: integrity: Checksum failed at sector 0x11a237 [ 4771.033192] device-mapper: integrity: Checksum failed at sector 0x1172c7 [ 4771.063490] device-mapper: integrity: Checksum failed at sector 0x1199cf [ 4831.095393] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4831.108089] device-mapper: integrity: Checksum failed at sector 0x1199df [ 4831.137297] rcu: 9-....: (0 ticks this GP) idle=7fa/1/0x4000000000000002 softirq=2186/2186 fqs=15010 [ 4831.137298] rcu: (detected by 16, t=60043 jiffies, g=24633, q=375) [ 4831.137518] raid5_end_read_request: 9866 callbacks suppressed [ 4831.137521] md/raid:md127: read error corrected (8 sectors at 1153480 on dm-3) [ 4831.167832] device-mapper: integrity: Checksum failed at sector 0x1199e7 [ 4831.168028] md/raid:md127: read error corrected (8 sectors at 1153496 on dm-3) [ 4891.199009] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 4891.221308] device-mapper: integrity: Checksum failed at sector 0x118d4f [ 4891.251366] rcu: 9-....: (1923 ticks this GP) idle=7fa/1/0x4000000000000002 softirq=2190/2190 fqs=15013 [ 4891.251367] rcu: (detected by 16, t=60054 jiffies, g=24641, q=351) [ 4891.281621] device-mapper: integrity: Checksum failed at sector 0x118d47 [ 4891.311941] Sending NMI from CPU 16 to CPUs 9: [ 4891.312107] NMI backtrace for cpu 9 [ 4891.312108] CPU: 9 PID: 3215 Comm: kworker/9:43 Not tainted 5.0.0-rc6-next-20190212 #1 [ 4891.312108] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 4891.312109] Workqueue: dm-integrity-metadata integrity_bio_wait [dm_integrity] [ 4891.312110] RIP: 0010:try_to_wake_up+0x3e4/0x4b0 [ 4891.312111] Code: 04 84 c0 0f 84 42 fe ff ff 49 8b 8c 2f 90 09 00 00 48 8b 11 eb 1c f6 c2 08 75 75 48 89 d6 48 89 d0 48 83 ce 08 f0 48 0f b1 31 <48> 39 c2 74 61 48 89 c2 f7 c2 00 00 20 00 75 dc 44 89 c7 44 89 44 [ 4891.312111] RSP: 0018:ffff94ebb7443c80 EFLAGS: 00000046 [ 4891.312112] RAX: 0000000080200000 RBX: ffff94ea88151700 RCX: ffff94e883d68000 [ 4891.312112] RDX: 0000000080200000 RSI: 0000000080200008 RDI: ffff94ea88151730 [ 4891.312113] RBP: ffff94e9b7700000 R08: 0000000000000014 R09: 0000000000021640 [ 4891.312113] R10: 0002aeae5e1e0327 R11: 0000000000000000 R12: ffff94ea88152264 [ 4891.312114] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000021d80 [ 4891.312114] FS: 0000000000000000(0000) GS:ffff94ebb7440000(0000) knlGS:0000000000000000 [ 4891.312115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4891.312115] CR2: 00007f9da3d2a000 CR3: 000000010780e002 CR4: 00000000001606e0 [ 4891.312116] Call Trace: [ 4891.312116] [ 4891.312116] __wake_up_common+0x7a/0x190 [ 4891.312117] __wake_up_common_lock+0x7c/0xc0 [ 4891.312120] ep_poll_callback+0x146/0x2e0 [ 4891.312121] __wake_up_common+0x7a/0x190 [ 4891.312121] __wake_up_common_lock+0x7c/0xc0 [ 4891.312121] ? tick_sched_do_timer+0x80/0x80 [ 4891.312122] irq_work_run_list+0x49/0x70 [ 4891.312122] update_process_times+0x4a/0x60 [ 4891.312123] tick_sched_handle+0x22/0x60 [ 4891.312123] tick_sched_timer+0x37/0x70 [ 4891.312123] __hrtimer_run_queues+0x100/0x280 [ 4891.312124] hrtimer_interrupt+0x100/0x220 [ 4891.312124] ? sched_clock_cpu+0xc/0xb0 [ 4891.312125] smp_apic_timer_interrupt+0x6a/0x140 [ 4891.312125] apic_timer_interrupt+0xf/0x20 [ 4891.312125] [ 4891.312126] RIP: 0010:vprintk_emit+0x1dd/0x230 [ 4891.312126] Code: 0f 1f 40 00 84 d2 74 6b 0f b6 05 ee 1e 8e 01 48 c7 c2 e0 51 a1 9d 84 c0 74 09 f3 90 0f b6 02 84 c0 75 f7 e8 f5 0a 00 00 55 9d 9e e1 ff ff e8 d9 fd ff ff e9 52 ff ff ff 80 3d 75 f0 2b 01 00 [ 4891.312127] RSP: 0018:ffffa77cc5c5bc78 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4891.312128] RAX: 0000000000000000 RBX: 000000000000003c RCX: ffff94eaa846c500 [ 4891.312128] RDX: ffffffff9da151e0 RSI: 0000000000000002 RDI: ffffffff9da151f0 [ 4891.312129] RBP: 0000000000000202 R08: 0000000000000002 R09: 0000000000021640 [ 4891.312129] R10: 0002aeae5a82eccb R11: 000000000005eeec R12: 0000000000000001 [ 4891.312130] R13: 0000000000000001 R14: 0000000000008cef R15: ffffa77cc5c5bcd0 [ 4891.312130] printk+0x58/0x6f [ 4891.312130] integrity_metadata.cold.57+0x1e/0x30 [dm_integrity] [ 4891.312131] ? io_schedule_timeout+0x19/0x40 [ 4891.312131] ? wait_for_completion_io+0x134/0x180 [ 4891.312132] ? wake_up_q+0x70/0x70 [ 4891.312132] dm_integrity_map_continue+0x46c/0x810 [dm_integrity] [ 4891.312132] ? __switch_to_asm+0x40/0x70 [ 4891.312133] process_one_work+0x1a1/0x3a0 [ 4891.312133] worker_thread+0x30/0x380 [ 4891.312133] ? mod_delayed_work_on+0x90/0x90 [ 4891.312134] kthread+0x112/0x130 [ 4891.312134] ? __kthread_parkme+0x70/0x70 [ 4891.312135] ret_from_fork+0x35/0x40 [ 4891.313171] md/raid:md127: read error corrected (8 sectors at 1150280 on dm-3) [ 4891.343503] device-mapper: integrity: Checksum failed at sector 0x11a247 [ 4891.343756] md/raid:md127: read error corrected (8 sectors at 1150272 on dm-3) [ 4891.374968] md/raid:md127: read error corrected (8 sectors at 1153504 on dm-3) [ 4891.406010] device-mapper: integrity: Checksum failed at sector 0x11a23f [ 4891.406194] md/raid:md127: read error corrected (8 sectors at 1155648 on dm-3) [ 5043.457125] INFO: task md127_resync:2492 blocked for more than 120 seconds. [ 5043.469781] device-mapper: integrity: Checksum failed at sector 0x1199d7 [ 5043.470026] md/raid:md127: read error corrected (8 sectors at 1155640 on dm-3) [ 5043.501825] Not tainted 5.0.0-rc6-next-20190212 #1 [ 5043.501825] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5043.501828] md127_resync D 0 2492 2 0x80000080 [ 5043.533106] device-mapper: integrity: Checksum failed at sector 0x118d3f [ 5043.533256] md/raid:md127: read error corrected (8 sectors at 1153488 on dm-3) [ 5043.564339] Call Trace: [ 5043.564348] ? __schedule+0x24e/0x860 [ 5043.595654] clocksource: timekeeping watchdog on CPU9: Marking clocksource 'tsc' as unstable because the skew is too large: [ 5043.595896] md/raid:md127: read error corrected (8 sectors at 1150264 on dm-3) [ 5043.627546] ? raid5_sync_request+0x1d1/0x390 [raid456] [ 5043.627550] schedule+0x28/0x70 [ 5043.658893] clocksource: 'hpet' wd_now: ce4a88f9 wd_last: 2152db77 mask: ffffffff [ 5043.658894] clocksource: 'tsc' cs_now: 2aef51c1bd75c cs_last: 2ae96f5a60a8e mask: ffffffffffffffff [ 5043.658896] tsc: Marking TSC unstable due to clocksource watchdog [ 5043.692219] md_do_sync.cold.81+0x74a/0x95e [ 5043.692222] ? 0xffffffff9c000000 [ 5043.725674] device-mapper: integrity: Checksum failed at sector 0x118d37 [ 5043.758362] ? __switch_to_asm+0x34/0x70 [ 5043.758368] ? finish_wait+0x80/0x80 [ 5043.792802] device-mapper: integrity: Checksum failed at sector 0x11a22f [ 5043.793073] md/raid:md127: read error corrected (8 sectors at 1150256 on dm-3) [ 5043.823971] ? md_register_thread+0xd0/0xd0 [ 5043.823974] md_thread+0x94/0x150 [ 5043.855880] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. [ 5043.887377] kthread+0x112/0x130 [ 5043.887379] ? __kthread_parkme+0x70/0x70 [ 5043.887381] ret_from_fork+0x35/0x40 [ 5103.726621] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [ 5103.751607] sched_clock: Marking unstable (5039698344099, 4158842470)<-(5044919649138, -1063788713) [ 5103.781913] rcu: 9-....: (1912 ticks this GP) idle=7fa/1/0x4000000000000002 softirq=2192/2192 fqs=15004 [ 5103.781914] rcu: (detected by 4, t=60057 jiffies, g=24645, q=5186) [ 5103.812493] md/raid:md127: read error corrected (8 sectors at 1155624 on dm-3) [ 5103.848221] NMI backtrace for cpu 4 [ 5103.848223] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.0.0-rc6-next-20190212 #1 [ 5103.848224] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 5103.848225] Call Trace: [ 5103.848227] [ 5103.848232] dump_stack+0x46/0x60 [ 5128.552442] watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kworker/9:53:4131] [ 5128.567751] nmi_cpu_backtrace.cold.5+0x13/0x4e [ 5128.597844] Modules linked in: raid456 async_raid6_recov async_memcpy async_pq raid6_pq dm_integrity dm_bufio async_xor xor async_tx loop intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipmi_ssif intel_cstate iTCO_wdt intel_uncore ipmi_si iTCO_vendor_support hpilo ipmi_devintf intel_rapl_perf sg pcspkr hpwdt lpc_ich ioatdma acpi_power_meter ipmi_msghandler dca xfs libcrc32c mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci drm libahci sfc libata crc32c_intel serio_raw tg3 hpsa mdio mtd scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [ 5128.627997] ? lapic_can_unplug_cpu.cold.26+0x3b/0x3b [ 5128.657481] CPU: 9 PID: 4131 Comm: kworker/9:53 Not tainted 5.0.0-rc6-next-20190212 #1 [ 5128.688633] nmi_trigger_cpumask_backtrace+0xde/0xe0 [ 5128.718793] Hardware name: HP ProLiant DL360p Gen8, BIOS P71 07/01/2015 [ 5128.748696] rcu_dump_cpu_stacks+0x9c/0xca [ 5128.778489] Workqueue: events __sched_clock_work [ 5128.808750] rcu_sched_clock_irq.cold.69+0x29d/0x35e [ 5128.838870] RIP: 0010:smp_call_function_many+0x1ea/0x250 [ 5128.869566] ? tick_sched_do_timer+0x80/0x80 [ 5128.900296] Code: 8f 6e 00 3b 05 2b 8b 28 01 0f 83 99 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 20 38 14 9d 8b 51 18 83 e2 01 74 0a f3 90 8b 51 18 <83> e2 01 75 f6 eb c7 48 c7 c2 e0 1a 3f 9d 4c 89 f6 89 df e8 4e 90 [ 5128.930586] update_process_times+0x28/0x60 [ 5128.960666] RSP: 0018:ffffa77cc799bd68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5128.990783] tick_sched_handle+0x22/0x60 [ 5129.024097] RAX: 0000000000000004 RBX: ffff94ebb7462dc0 RCX: ffff94e9b7529280 [ 5129.057646] tick_sched_timer+0x37/0x70 [ 5129.087777] RDX: 0000000000000003 RSI: 0000000000000000 RDI: ffff94e9c7c014d8 [ 5129.117914] __hrtimer_run_queues+0x100/0x280 [ 5129.147723] RBP: ffffffff9c025130 R08: 0000000000027040 R09: ffffffff9c043cca [ 5129.147733] R10: fffff02e0421f140 R11: 0000000000000001 R12: 0000000000000000 [ 5129.147734] R13: 0000000000000001 R14: 0000000000000040 R15: 0000000000000001 [ 5129.147737] FS: 0000000000000000(0000) GS:ffff94ebb7440000(0000) knlGS:0000000000000000 [ 5129.178253] hrtimer_interrupt+0x100/0x220 [ 5129.208357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5129.238178] ? ktime_get+0x36/0xa0 [ 5129.268391] CR2: 00007f9da3d2a000 CR3: 000000010780e002 CR4: 00000000001606e0 [ 5129.298491] smp_apic_timer_interrupt+0x6a/0x140 [ 5129.328749] Call Trace: [ 5129.359223] apic_timer_interrupt+0xf/0x20 [ 5129.389316] ? optimize_nops.isra.2+0x80/0x80 [ 5129.419559] [ 5129.449318] ? sched_clock_tick_stable+0x1/0x10 [ 5129.479555] RIP: 0010:cpuidle_enter_state+0xb4/0x440 [ 5129.509737] on_each_cpu+0x28/0x40 [ 5129.509740] ? sched_clock_idle_wakeup_event+0x20/0x20 [ 5129.543090] Code: 24 0f 1f 44 00 00 31 ff e8 59 99 a5 ff 80 7c 24 13 00 74 12 9c 58 f6 c4 02 0f 85 61 03 00 00 31 ff e8 d0 8b ab ff fb 45 85 e4 <0f> 88 92 02 00 00 49 63 cc 4c 8b 3c 24 4c 2b 7c 24 08 48 8d 04 49 [ 5129.575112] text_poke_bp+0x68/0xdf [ 5129.605230] RSP: 0018:ffffa77cc1943e98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5129.635501] __jump_label_transform+0x112/0x120 [ 5129.666949] RAX: ffff94e9b7521d80 RBX: ffffffff9d32c020 RCX: 000000000000001f [ 5129.666952] RDX: 000004a44dcfd085 RSI: 0000000040277b67 RDI: 0000000000000000 [ 5129.697127] arch_jump_label_transform+0x26/0x40 [ 5129.727700] RBP: ffff94e9b752cb00 R08: 0000000000000002 R09: 0000000000021640 [ 5129.727703] R10: 0002af110b0bae1c R11: ffff94e9b7520e64 R12: 0000000000000004 [ 5129.757659] __jump_label_update+0x75/0xe0 [ 5129.787932] R13: ffffffff9d32c1b8 R14: 0000000000000004 R15: 0000000000000000 [ 5129.787936] ? cpuidle_enter_state+0x97/0x440 [ 5129.818079] static_key_disable_cpuslocked+0x59/0x90 [ 5129.848133] do_idle+0x1f1/0x230 [ 5129.878559] static_key_disable+0x16/0x20 [ 5129.909617] cpu_startup_entry+0x19/0x20 [ 5129.939847] process_one_work+0x1a1/0x3a0 [ 5129.969635] start_secondary+0x195/0x1e0 [ 5129.999827] worker_thread+0x30/0x380 [ 5130.030567] secondary_startup_64+0xb6/0xc0 [ 5130.066336] ? mod_delayed_work_on+0x90/0x90 [ 5130.097684] NMI backtrace for cpu 4 [ 5130.097685] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.0.0-rc6-next-20190212 #1