Recovering a Linux Software raid when a rebuild fails

In this scenario we will recover from a situation where both drives in a software raid 1 in Linux have failed.

First go through this guide: https://server-help.org/index.php/2019/04/17/rebuild-software-raid-linux/

If this process fails and the raid is not rebuilt proceed as follows.

In this we assume sda is the drive with our data but is failing, and sdb is new.

We also assume this server is running Cent OS 6, if using CentOS 7 or Ubuntu your steps may be different:

https://wiki.centos.org/TipsAndTricks/CreateNewInitrd

https://ubuntugenius.wordpress.com/tag/update-initramfs

Reboot into a rescue image.

If the ddrescue command is not installed on your rescue image:

1). wget http://ftp.us.debian.org/debian/pool/main/g/gddrescue/gddrescue_1.21-1_amd64.deb
2). dpkg -i gddrescue_1.21-1_amd64.deb

1). Run the following command:

ddrescue -f /dev/sda /dev/sdb /log.log

This process will take a very long time, especially if the drive is large or there are alot of errors (6-24 hours+)

Once successful proceed with removing the bad sda from the array and replacing with the cloned data on the good drive sdb.

1). mdadm –stop /dev/md1
2). mdadm –assemble /dev/md1 /dev/sdb1
3). dmsetup remove vg00-home
4). dmsetup remove vg00-usr
5). dmsetup remove vg00-var
6). mdadm –stop /dev/md3
7). mdadm –assemble /dev/md3 /dev/sdb3
8). vgchange -ay
9). fsck -y /dev/md1

10). fsck -y /dev/mapper/vg00-usr
11). fsck -y /dev/mapper/vg00-var
12). fsck -y /dev/mapper/vg00-home
13). mount /dev/md1 /mnt
14). mdadm –examine –scan –config=mdadm.conf >> /mnt/etc/mdadm.conf
15). mount /dev/vg00/usr /mnt/usr
16). mount /dev/vg00/var /mnt/var
17). mount /dev/vg00/home /mnt/home
18). mount –bind /proc /mnt/proc
19). mount –bind /dev /mnt/dev

20). mount –bind /sys /mnt/sys
21). chroot /mnt
22). ls /boot/ (to find your latest initramfs)
23). dracut -f /boot/initramfs-2.6.32-431.11.2.el6.x86_64.img initramfs-2.6.32-431.11.2.el6.x86_64