ssd cache recovery

eom.dev · 2022-12-21 16:00:52

I believe I lost my ssd cache during an unexpected power outage, but cannot find out how to recover it. When I plug in the ssd, I get the following journal entries:

Dec 21 15:51:59 archlinux kernel: usb 2-8: new SuperSpeed USB device number 5 using xhci_hcd
Dec 21 15:51:59 archlinux kernel: usb 2-8: New USB device found, idVendor=xxxx, idProduct=xxxx, bcdDevice=43.06
Dec 21 15:51:59 archlinux kernel: usb 2-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Dec 21 15:51:59 archlinux kernel: usb 2-8: Product: Rugged SSD Pro
Dec 21 15:51:59 archlinux kernel: usb 2-8: Manufacturer: LaCie
Dec 21 15:51:59 archlinux kernel: usb 2-8: SerialNumber: xxxxxxxxxxxxxxxx
Dec 21 15:51:59 archlinux kernel: scsi host16: uas
Dec 21 15:51:59 archlinux kernel: scsi 16:0:0:0: Direct-Access     LaCie    Rugged SSD Pro   xxxx PQ: 0 ANSI: 6
Dec 21 15:52:01 archlinux kernel: usb 2-8: USB disconnect, device number 5
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] tag#13 uas_zap_pending 0 uas-tag 1 inflight: CMD 
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] tag#13 CDB: Test Unit Ready 00 00 00 00 00 00
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Read Capacity(16) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Sense not available.
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Read Capacity(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Sense not available.
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] 0 512-byte logical blocks: (0 B/0 B)
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] 0-byte physical blocks
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Write Protect is off
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Mode Sense: 00 00 00 00
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Asking for cache data failed
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Assuming drive cache: write through
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes not a multiple of physical block size (0 bytes)
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of physical block size (0 bytes)
Dec 21 15:52:01 archlinux kernel: sd 16:0:0:0: [sdc] Attached SCSI disk
Dec 21 15:52:01 archlinux multipath[870]: sysfs_is_multipathed: error scanning /sys/block/sdc/holders

How do I repair the ssd's cache so that it appears as /dev/sdc again?

Slithery · 2022-12-21 16:14:15

I don't think that it's possible - the drive is FUBAR.

But post the result of a SMART test anyway (if it even runs), there may still be a very slim chance that your data is intact.

eom.dev · 2022-12-21 16:16:12

Slithery wrote:

I don't think that it's possible - the drive is FUBAR.
But post the result of a SMART test anyway (if it even runs), there may still be a very slim chance that your data is intact.

The data is intact. This is a bootable ssd that works as a standalone instance or a vm, and I can boot it fine through virtual box on a Mac, but archlinux does not find /dev/sdc.

I booted the device and used it to run S.M.A.R.T. tests on itself:

~ smartctl -a /dev/sdc -d scsi

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.0.12-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               LaCie
Product:              Rugged SSD Pro
Revision:             J583
Compliance:           SPC-4
User Capacity:        1,000,204,886,016 bytes [1.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:        Solid State Device
Logical Unit id:      0x300000ec12345600
Serial number:        xxxxxxxxxxxxx
Device type:          disk
Local Time is:        Wed Dec 21 09:01:16 2022 UTC
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===

Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging

For some clarity on my setup, I have three machines that can use this ssd, which boots archlinux:
* a MacBook that boots it as a virtual box vm (currently working)
* a desktop that boots the ssd installation as the primary os (currently working)
* an archlinux server that boots the ssd as a qemu virtual machine (not working because no /dev/sdc)

Last edited by eom.dev (2022-12-21 17:34:19)

eom.dev · 2022-12-21 17:24:20

When I boot from this SSD on the standalone desktop, journal messages indicate that the write cache on the device is enabled, so why can't a separate installation not read the cache?

Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] 4096-byte physical blocks
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Write Protect is off
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Mode Sense: 5f 00 00 08
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Preferred minimum I/O size 4096 bytes
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
Dec 21 09:14:10 archlinux kernel:  sdc: sdc1 sdc2 sdc3 sdc4
Dec 21 09:14:10 archlinux kernel: sd 6:0:0:0: [sdc] Attached SCSI disk
Dec 21 09:14:19 installation-0e9e743c7961 systemd-fsck[387]: /dev/sdc2: 8 files, 72/261627 clusters
Dec 21 09:15:06 installation-0e9e743c7961 kernel: sd 6:0:0:0: [sdc] Synchronizing SCSI cache

I had hoped that reactivating the cache here would magically self-repair the drive, but my rain dance failed.

Last edited by eom.dev (2022-12-21 17:35:48)

cfr · 2022-12-21 18:38:31

You probably want to look for the first difference between the fail and the success, but you've not provided enough context to compare them here.

Does another similar drive work properly across the board? That is, is the problem specific to this particular device?

eom.dev · 2022-12-21 19:33:36

cfr wrote:

You probably want to look for the first difference between the fail and the success, but you've not provided enough context to compare them here.
Does another similar drive work properly across the board? That is, is the problem specific to this particular device?

Going back in the journal shows the log messages from a previous day:

Dec 18 15:52:01 archlinux kernel: usb 2-8: new SuperSpeed USB device number 2 using xhci_hcd
Dec 18 15:52:01 archlinux kernel: usb 2-8: New USB device found, idVendor=xxxx, idProduct=xxxx, bcdDevice=43.06
Dec 18 15:52:01 archlinux kernel: usb 2-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Dec 18 15:52:01 archlinux kernel: usb 2-8: Product: Rugged SSD Pro
Dec 18 15:52:01 archlinux kernel: usb 2-8: Manufacturer: LaCie
Dec 18 15:52:01 archlinux kernel: usb 2-8: SerialNumber: xxxxxxxxxxxxx
Dec 18 15:52:01 archlinux kernel: scsi host16: uas
Dec 18 15:52:01 archlinux kernel: scsi 16:0:0:0: Direct-Access     LaCie    Rugged SSD Pro   xxxx PQ: 0 ANSI: 6
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] 4096-byte physical blocks
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] Write Protect is off
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] Mode Sense: 5f 00 00 08
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
Dec 18 15:52:03 archlinux kernel:  sdc: sdc1 sdc2 sdc3 sdc4
Dec 18 15:52:03 archlinux kernel: sd 16:0:0:0: [sdc] Attached SCSI disk

First difference is usb disconnect. I guess I should check my cables? ?

cfr · 2022-12-21 21:01:44

Cables seems a good place to start.

loqs · 2022-12-21 21:12:51

Please post the output of dmesg or full journal for a boot with the issue.

Last edited by loqs (2022-12-21 21:13:08)

eom.dev · 2022-12-21 22:59:28

loqs wrote:

Please post the output of dmesg or full journal for a boot with the issue.

Unfortunately it is working now so I cannot reproduce the error. There is only one thing I can think of that I did, but wow I cannot imagine it being correct:

Cfr's comment about a similar device reminded me that I had an old version of this project living on a usb drive that still exists. I attached it for comparison logs, it was successful, and then I reattached the ssd and it was successful as well. So it seems attaching an intermediate device did something to solve the issue? I am beyond baffled because this solved it on both servers...

cfr · 2022-12-22 02:17:21

You can get the journal from previous boots:

journalctl -b -1 --no-pager | curl -F 'file=@-' 0x0.st 
journalctl -b -2 --no-pager | curl -F 'file=@-' 0x0.st 
...

for the previous boot, the one before that and so on.

eom.dev · 2022-12-22 14:23:34

cfr wrote:

You can get the journal from previous boots:
for the previous boot, the one before that and so on.

I don't mean to be argumentative, but which boot is relevant? When booted as a standalone or vbox vm the system was not faulty. I shared the journal entries from the faulty system, which did not boot with the ssd attached and was not rebooted in between unsuccessful and successful cases.

I had been looking for that curl command btw - thanks!

eom.dev · 2022-12-22 17:22:56

Not out of the woods yet, it would seem. The ssd is recognized when connected, but seems to be having some I/O trouble. With the ssd booted as a qemu vm, the guest crashes after the following journal entries:

Dec 22 17:13:13 archlinux kernel: usb 2-8: USB disconnect, device number 7
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#12 uas_zap_pending 0 uas-tag 13 inflight: CMD 
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#12 CDB: Write(10) 2a 00 01 f1 e4 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#13 uas_zap_pending 0 uas-tag 14 inflight: CMD 
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#13 CDB: Write(10) 2a 00 01 f1 e8 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#14 uas_zap_pending 0 uas-tag 15 inflight: CMD 
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#14 CDB: Write(10) 2a 00 01 f1 ec 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#15 uas_zap_pending 0 uas-tag 16 inflight: CMD 
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#15 CDB: Write(10) 2a 00 01 f1 f0 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#12 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#12 CDB: Write(10) 2a 00 01 f1 e4 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: I/O error, dev sdd, sector 32629760 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: buffer_io_error: 13 callbacks suppressed
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078720, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078721, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078722, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078723, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078724, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078725, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078726, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078727, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078728, lost async page write
Dec 22 17:13:13 archlinux kernel: Buffer I/O error on dev sdd, logical block 4078729, lost async page write
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#13 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#13 CDB: Write(10) 2a 00 01 f1 e8 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: I/O error, dev sdd, sector 32630784 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#14 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#14 CDB: Write(10) 2a 00 01 f1 ec 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: I/O error, dev sdd, sector 32631808 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#15 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] tag#15 CDB: Write(10) 2a 00 01 f1 f0 00 00 04 00 00
Dec 22 17:13:13 archlinux kernel: I/O error, dev sdd, sector 32632832 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 32633856 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 32634880 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 32635904 op 0x1:(WRITE) flags 0x800 phys_seg 21 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 33837344 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 36971304 op 0x1:(WRITE) flags 0x800 phys_seg 7 prio class 2
Dec 22 17:13:13 archlinux kernel: device offline error, dev sdd, sector 45097232 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] Synchronizing SCSI cache
Dec 22 17:13:13 archlinux kernel: sd 17:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK

dmesg output:

[86529.124456] sd 17:0:0:0: [sdd] tag#14 CDB: Write(10) 2a 00 01 f1 ec 00 00 04 00 00
[86529.124459] I/O error, dev sdd, sector 32631808 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
[86529.124701] sd 17:0:0:0: [sdd] tag#15 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[86529.124706] sd 17:0:0:0: [sdd] tag#15 CDB: Write(10) 2a 00 01 f1 f0 00 00 04 00 00
[86529.124708] I/O error, dev sdd, sector 32632832 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
[86529.124934] device offline error, dev sdd, sector 32633856 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
[86529.125162] device offline error, dev sdd, sector 32634880 op 0x1:(WRITE) flags 0x4800 phys_seg 128 prio class 2
[86529.125378] device offline error, dev sdd, sector 32635904 op 0x1:(WRITE) flags 0x800 phys_seg 21 prio class 2
[86529.125422] device offline error, dev sdd, sector 33837344 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
[86529.125433] device offline error, dev sdd, sector 36971304 op 0x1:(WRITE) flags 0x800 phys_seg 7 prio class 2
[86529.125452] device offline error, dev sdd, sector 45097232 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
[86529.260229] sd 17:0:0:0: [sdd] Synchronizing SCSI cache
[86529.440206] sd 17:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK

It seems like the drive is having a hard time, but then why does this only occur on my linux machines?

I'm using a dell server, which is mentioned in this post and again has me concerned about the drive itself.

Last edited by eom.dev (2022-12-22 17:26:37)

PerfMonk · 2026-04-28 12:46:57

Is it an usb connected disk ?

If yes, did you consider switching the driver from uas to usb-storage ?
It's controlled by kernel parameters.

WorMzy · 2026-04-28 14:44:10

Please pay attention to the age of threads and refrain from necrobumping.

https://wiki.archlinux.org/title/Genera … bumping%22

Closing.

Arch Linux

#1 2022-12-21 16:00:52

ssd cache recovery

#2 2022-12-21 16:14:15

Re: ssd cache recovery

#3 2022-12-21 16:16:12

Re: ssd cache recovery

#4 2022-12-21 17:24:20

Re: ssd cache recovery

#5 2022-12-21 18:38:31

Re: ssd cache recovery

#6 2022-12-21 19:33:36

Re: ssd cache recovery

#7 2022-12-21 21:01:44

Re: ssd cache recovery

#8 2022-12-21 21:12:51

Re: ssd cache recovery

#9 2022-12-21 22:59:28

Re: ssd cache recovery

#10 2022-12-22 02:17:21

Re: ssd cache recovery

#11 2022-12-22 14:23:34

Re: ssd cache recovery

#12 2022-12-22 17:22:56

Re: ssd cache recovery

#13 2026-04-28 12:46:57

Re: ssd cache recovery

#14 2026-04-28 14:44:10

Re: ssd cache recovery

Board footer