Even with the best RAID setups, the sad truth is that hard drives fail. A lot. One minute your server is happily spinning its platters, and the next everything is gone. You had a backup in place, right?

Situations like that have turned RAID setups into the defacto standard for servers and probably all machines of some importance. While a mirrored RAID setup is no substitute for a good backup (Sadly, it can only protect you from physical failures, not from malicious or foolish actions.) it can save you a lot of work. Nobody likes reinstalling operating systems, reconfiguring databases and servers and restoring data from backups when s/he can just replace a faulty drive, wait for the data to get synchronized and call it a day.

The problem any noob sysadmin faces is that RAID arrays are pretty easy to set up. People are happily using them all over the place, while they have never restored a degraded RAID array before. I know I am guilty of that too. The usual thinking is "Hey, it can't be too hard." It really isn't, but it is not the best idea to learn about it when disaster strikes. It is very easy to set up a virtual machine with a two-disk mirrored RAID 1 array and experiment there until you get it right. In this blog post, I will use the test VM I set up in our Ubuntu RAID article, but the principles and tools are generic and will work on any distribution.

Booting a machine with broken hard drives

Suppose you have a machine with two mirrored hard drives (RAID 1) and one of them dies. You shut it down, and replace the drive with a healthy one. Now you have a new unpartitioned hard drive and an old one that has all your data. If you got lucky, and it's the second drive that died, you might still be able to boot your machine -- the first drive is healthy and bootable after all. If your first drive died, then you will most likely have to use your rescue media to boot a system that has all the needed tools to recover your array.

In my case I just use the Ubuntu Server install CD. I choose the "Rescue a broken system" option and drop into a shell. Again, it should be perfectly possible to use any Linux rescue distribution, like Knoppix, Grml Live, etc, but the simple Ubuntu install CD does the job just fine.

Getting a feel of the broken system

Let us first look around our system and make sure things are really as we expect them to be. You can't be too careful when you partition hard disks.

For starters, let's see which of the hard drives is the broken one. I'll use GNU parted to view and edit partitions since my machine uses a GPT partitioning scheme. If you are using a plain MBR partition table, you can happily use fdisk. I'd even recommend the simple GUI cfdisk offers. Anyway, what I like about parted is that it seems to work with most partitioning schemes out there. Here it is checking the /dev/sda partitions:

~ # parted /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
print
Error: /dev/sda: unrecognised disk label  
(parted) quit                                                             
quit

Oops! "Unrecognized disk label" is parted's way of saying that drive hasn't been partitioned yet. It's probably our newly-replaced drive. Let's check the other one:

~ # parted /dev/sdb
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit kb                                                          
unit kb
(parted) print                                                            
print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 8589935kB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name  Flags
 1      1049kB  2097kB     1049kB                        bios_grub
 2      2097kB  8588886kB  8586789kB                     raid

That's better -- we see our GPT partition table with its two partitions: one that hosts the GRUB bootloader and another that contains the Linux / file system. Note that we are using kilobytes as the address unit -- that will make it easier to recreate the partitions on the new drive later.

Partitioning the new drive

Let's now start parted again on /dev/sda and switch to kilobytes as our address units:

~ # parted /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit kb                                                          
unit kb

We need to create a partition table first with the mklabel command.

(parted) mklabel gpt                                                      
mklabel gpt

We are using a gpt partitioning scheme here, but your system may use a different one -- check the available types by running help mklabel. Let us now create the first partition, the BIOS boot one by specifying that we want a primary partition with exact start and end addresses:

(parted) mkpart primary 1049 2097                                         
mkpart primary 1049 2097
(parted) set 1 bios_grub on                                               
set 1 bios_grub on

We give it roughly 1MB of space (note that I am using the same start/end addresses as the one we got from the /dev/sdb printout above. In addition we set the bios_grub flag (1 is the partition number), so that GRUB knows where to install itself.

Now to create the data partition:

(parted) mkpart primary 2097 8588886                                      
mkpart primary 2097 8588886
(parted) set 2 raid on                                                    
set 2 raid on

Again, we use the same start/end addresses as the ones for /dev/sdb2 as we want an identical partition. We also set the raid flag on to specify this partition is a part of a Linux RAID array.

Let's check our work now:

(parted) print                                                            
print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 8589935kB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name     Flags
 1      1049kB  2097kB     1049kB                  primary  bios_grub
 2      2097kB  8588886kB  8586789kB               primary  raid

And we are done. Good job!

Starting the RAID array in degraded mode

So far, so good. Let's see in what state our RAID array is. We check /proc/mdstat first:

~ # cat /proc/mdstat
Personalities : 
md127 : inactive sdb2[1](S)
      8381440 blocks super 1.2

unused devices: <none>

Yes, We have an inactive md127 array (I'd appreciate a md0 or md1 simple name, but apparently the Ubuntu installer thinks otherwise.) that contains just the sdb2 partition. Let's try getting details via the mdadm helper tool:

~ # mdadm --detail /dev/md127
mdadm: md device /dev/md127 does not appear to be active.

Okay, I get it -- the array is inactive. Let's activate it.

~ # mdadm --manage --run /dev/md127
mdadm: started /dev/md127

Mini rant: I love the terminology soup here. One does not simply "activate" an inactive RAID array. You have to "run" it. Oh, and to deactivate it, you need the "stop" command.

Let us look at the state of /dev/md127 now:

~ # mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Thu Oct 25 15:24:34 2012
     Raid Level : raid1
     Array Size : 8381376 (7.99 GiB 8.58 GB)
  Used Dev Size : 8381376 (7.99 GiB 8.58 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Thu Oct 25 16:18:54 2012
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : ubuntu-gpt:0  (local to host ubuntu-gpt)
           UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc
         Events : 64

    Number   Major   Minor   RaidDevice State
       3       8       18        0      active sync   /dev/sdb2
       1       0        0        1      removed

It's alive! Yes, it is in a degraded state, running off only /dev/sdb2, but it's mountable and all.

Rebuilding the array

We have a perfectly-partitioned new drive now, and we are not afraid to use it. We add its /dev/sda2 partition to the array like this:

~ # mdadm --manage --add /dev/md127 /dev/sda2
mdadm: added /dev/sda2

The kernel will automatically start syncing data from /dev/sdb2 to /dev/sda2, so that both partitions contain exact copies of the data. You check its progress with mdadm's --detail command:

~ # mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Thu Oct 25 15:24:34 2012
     Raid Level : raid1
     Array Size : 8381376 (7.99 GiB 8.58 GB)
  Used Dev Size : 8381376 (7.99 GiB 8.58 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Oct 25 16:28:44 2012
          State : clean, degraded, recovering 
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 1% complete

           Name : ubuntu-gpt:0  (local to host ubuntu-gpt)
           UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc
         Events : 68

    Number   Major   Minor   RaidDevice State
       3       8       18        0      active sync   /dev/sdb2
       2       8        2        1      spare rebuilding   /dev/sda2

It may be a good idea to wait for the rebuild to finish and the array to get promoted to clean status.

Repairing the bootloader

So, you think you have fixed your broken hard drive problem? Well, almost -- you have just fixed your RAID array. You still need to make that drive bootable. We do that by using the existing GRUB configuration and just reinstalling the bootloader code.

To access the existing GRUB configuration, we need to mount the partition inside our RAID array. We'll mount it under /mnt/rescue:

~ # mkdir /mnt/rescue
~ # mount /dev/md127 /mnt/rescue
~ # ls -l /mnt/rescue
drwxr-xr-x    2 root     root          4096 Oct 24 14:57 bin
drwxr-xr-x    3 root     root          4096 Oct 24 15:01 boot
drwxr-xr-x   12 root     root          3900 Oct 25 16:27 dev
...

As you can see, we can access the data in our file system. We now need to run grub-install as if it's run off the original filesystem. To do that we need two things:

  • Chroot to /mnt/rescue so grub-install can locate its config files in the usual locations.
  • Expose our devices from /dev/ as /mnt/rescue/dev so that GRUB can access them.

We first forward /dev to /mnt/rescue/dev using a little-known feature of Linux's mount command, bind:

~ # mount -o bind /dev /mnt/rescue/dev

We can then use chroot to run a /bin/bash shell below /mnt/rescue:

~ # chroot /mnt/rescue /bin/bash

We are now rooted in the right area, and we can finally reinstall GRUB to our new drive:

root@ubuntu-gpt:/# grub-install /dev/sda
Installation finished. No error reported.

GRUB is smart enough to detect the partitioning scheme and will install itself either to the master boot record for MBR partitions or to the BIOS boot partition for GPT schemes.

We are done here. Let us reboot the system and login as normal.

Checking our work

After logging in to our system, we inspect the state of the RAID array:

root@ubuntu-gpt:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid1 sdb2[3] sda2[2]
      8381376 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Using mdadm:

root@ubuntu-gpt:~# mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Thu Oct 25 18:24:34 2012
     Raid Level : raid1
     Array Size : 8381376 (7.99 GiB 8.58 GB)
  Used Dev Size : 8381376 (7.99 GiB 8.58 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Thu Oct 25 19:37:45 2012
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : ubuntu-gpt:0  (local to host ubuntu-gpt)
           UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc
         Events : 93

    Number   Major   Minor   RaidDevice State
       3       8       18        0      active sync   /dev/sdb2
       2       8        2        1      active sync   /dev/sda2

Everything is humming along just fine. Mission accomplished.