Storage device USB or SD Card debugging
I was sent an SD card containing a partition table where one of the partitions contained a file containing captured video that the customer required. The only problem was the host computer kept throwing errors when trying to access the video file. I was asked to dig into the details.
Do no harm
The first thing I did was copy the raw contents of the entire SD card to a file. That way I can make mistakes on the image file without risking loosing the video file on the SD card. You need to be careful as most host computers will try to mount the file systems on the SD card. I have my Ubuntu box configured so auto-mount is disabled.
To find the SD card device file I used dmesg.
[3097298.239288] sd 154:0:0:0: [sdi] 15523840 512-byte logical blocks: (7.94 GB/7.40 GiB)
As you can see, in my case the device file was /dev/sdi. You may also see devices files like /dev/sdi1 and/dev/sdi2 (try cat /proc/partitions). These device files expose the partition contents. We want to copy the entire SD card, so I used /dev/sdi. Your computer will use a different device filename.
My first attempt was to use 'dd to do a raw copy of the entire SD card contents:
sudo dd bs=64M if=/dev/sdi of=sd.img
I saw errors and the copy stopped before all 8 GB were copied.
dd: reading `/dev/sdi': Input/output error 47+1 records in 47+1 records out 3168010240 bytes (3.2 GB) copied, 531.645 s, 6.0 MB/s
So, I tried again telling dd to ignore errors and fill the unreadable areas with zeros:
sudo dd bs=64M if=/dev/sdi of=sd.img conv=noerror conv=sync
with output
dd: reading `/dev/sdi': Input/output error 117+2 records in 119+0 records out 7985954816 bytes (8.0 GB) copied, 868.579 s, 9.2 MB/s
It is interesting the file size using dd is bigger than expected - 7985954816 bytes in the dd created copy where as the kernel reported the size as 7948206080 (15523840 512-byte logical blocks). I suspect this has to do with the last block being increased to match the 64M boundary of my block size setting.
I then used dmesg again to see what type of errors occurred:
[3099472.345548] Buffer I/O error on device sdi, logical block 773440 [3099472.345567] Buffer I/O error on device sdi, logical block 773441 [3099472.345575] Buffer I/O error on device sdi, logical block 773442 [3099472.345581] Buffer I/O error on device sdi, logical block 773443 [3099472.345588] Buffer I/O error on device sdi, logical block 773444 [3099472.345595] Buffer I/O error on device sdi, logical block 773445 [3099472.345602] Buffer I/O error on device sdi, logical block 773446 [3099472.345609] Buffer I/O error on device sdi, logical block 773447 [3099472.345616] Buffer I/O error on device sdi, logical block 773448 [3099472.345622] Buffer I/O error on device sdi, logical block 773449
This leads me to believe there are bad blocks on the SD card.
I read that dd_rescue does a better job at grabbing all the available data. The dd program will zero fill the block size (64M in the usage above) with zeros even if just 512 bytes is unreadable.
I used the following parameters with dd_rescue:
sudo dd_rescue -v -l 8gb-customer-sd-copy.log -A /dev/sdi 8gb-customer-dd-rescue-sd.img
If your system doesn't have dd_rescue, try
sudo apt-get install ddrescue
dd_rescue was very slow - be patient. I opened up another window and monitored the change in file size. When dd_rescue encounted a bad block, it took surprisingly long for the file size to start increating again. Here is some of the 424 lineos dd_rescue output:
dd_rescue: (info) expect to copy 7761920kB from /dev/sdi
dd_rescue: (info): about to transfer 0.0 kBytes from /dev/sdi to 8gb-customer-dd-rescue-sd.img
dd_rescue: (info): blocksizes: soft 65536, hard 512
dd_rescue: (info): starting positions: in 0.0k, out 0.0k
dd_rescue: (info): Logfile: 8gb-customer-sd-copy.log, Maxerr: 0
dd_rescue: (info): Reverse: no , Trunc: no , interactive: no
dd_rescue: (info): abort on Write errs: no , spArse write: never
dd_rescue: (info): ipos: 3093760.0k, opos: 3093760.0k, xferd: 3093760.0k
errs: 0, errxfer: 0.0k, succxfer: 3093760.0k
+curr.rate: 1kB/s, avg.rate: 8801kB/s, avg.load: 3.9%
>----------------.........................< 39% ETA: 0:08:50
dd_rescue: (info): problems at ipos 3093760.0k: Success
fall back to smaller blocksize
...
Bad block reading /dev/sdi: 6187775
...
dd_rescue: (info): read /dev/sdi (7761920.0k): EOF
dd_rescue: (info): Summary for /dev/sdi -> 8gb-customer-dd-rescue-sd.img:
dd_rescue: (info): ipos: 7761920.0k, opos: 7761920.0k, xferd: 7761920.0k
errs: 64, errxfer: 32.0k, succxfer: 7761888.0k
+curr.rate: 0kB/s, avg.rate: 825kB/s, avg.load: 0.6%
The good news is the file size is exactly what I expected: 7948206080
Who is changing my file system
Even when you mount a file system as read-only, it still may get written. Examples include updating the last time a file was accessed. So before doing anything to my sd.img copy of the SD card contents, I calculated an md5sum so I could check to see if sd.img got changed when I wasn't expecting that to happen.
sudo chown $USER:$USER *sd.img* md5sum sd.img > sd.img.md5sum
Mounting the file systems using kpartx
To find out the file system type on each partition, I used fdisk with the p - print partition table:
fdisk dd-rescue-sd.img
with output:
Disk dd-rescue-sd.img: 7948 MB, 7948206080 bytes
255 heads, 63 sectors/track, 966 cylinders, total 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
dd-rescue-sd.img1 * 16065 96389 40162+ c W95 FAT32 (LBA)
dd-rescue-sd.img2 96390 369494 136552+ 83 Linux
dd-rescue-sd.img3 369495 15518789 7574647+ c W95 FAT32 (LBA)
Then I did a
sudo apt-get install kpartx
and tried
sudo kpartx -a -v dd-rescue-sd.img find /dev/mapper
and got the output:
add map loop0p1 (252:0): 0 80325 linear /dev/loop0 16065 add map loop0p2 (252:1): 0 273105 linear /dev/loop0 96390 add map loop0p3 (252:2): 0 15149295 linear /dev/loop0 369495 ... /dev/mapper/loop0p1 /dev/mapper/loop0p2 /dev/mapper/loop0p3 ...
We can use /dev/mapper/loop0p3 as our device file.
First, lets try to repair the file system
sudo fsck.vfat -a -w /dev/mapper/loop0p3
I ran fsck a few time until the warning were gone.
Next, mount the repaired file system.
mkdir /tmp/mnt sudo mount /dev/mapper/loop0p3 /tmp/mnt tree /tmp/mnt
Output looks good. So I copied all the files to my development workstation hard disk
mkdir $HOME/sd-card-files
find /tmp/mnt -type f -exec cp {} $HOME/sd-card-files \;
ls -l $HOME/sd-card-files
and I see the recovered video files are there.