Even with the best RAID setups, the sad truth is that hard drives fail. A lot. One minute your server is happily spinning its platters, and the next everything is gone. You had a backup in place, right?
Situations like that have turned RAID setups into the defacto standard for servers and probably all machines of some importance. While a mirrored RAID setup is no substitute for a good backup (Sadly, it can only protect you from physical failures, not from malicious or foolish actions.) it can save you a lot of work. Nobody likes reinstalling operating systems, reconfiguring databases and servers and restoring data from backups when s/he can just replace a faulty drive, wait for the data to get synchronized and call it a day.
The problem any noob sysadmin faces is that RAID arrays are pretty easy to set up. People are happily using them all over the place, while they have never restored a degraded RAID array before. I know I am guilty of that too. The usual thinking is "Hey, it can't be too hard." It really isn't, but it is not the best idea to learn about it when disaster strikes. It is very easy to set up a virtual machine with a two-disk mirrored RAID 1 array and experiment there until you get it right. In this blog post, I will use the test VM I set up in our Ubuntu RAID article, but the principles and tools are generic and will work on any distribution.
Booting a machine with broken hard drives
Suppose you have a machine with two mirrored hard drives (RAID 1) and one of them dies. You shut it down, and replace the drive with a healthy one. Now you have a new unpartitioned hard drive and an old one that has all your data. If you got lucky, and it's the second drive that died, you might still be able to boot your machine -- the first drive is healthy and bootable after all. If your first drive died, then you will most likely have to use your rescue media to boot a system that has all the needed tools to recover your array.
In my case I just use the Ubuntu Server install CD. I choose the "Rescue a broken system" option and drop into a shell. Again, it should be perfectly possible to use any Linux rescue distribution, like Knoppix, Grml Live, etc, but the simple Ubuntu install CD does the job just fine.
Getting a feel of the broken system
Let us first look around our system and make sure things are really as we expect them to be. You can't be too careful when you partition hard disks.
For starters, let's see which of the hard drives is the broken one. I'll use GNU parted to view and edit partitions since my machine uses a GPT partitioning scheme. If you are using a plain MBR partition table, you can happily use fdisk. I'd even recommend the simple GUI cfdisk offers. Anyway, what I like about parted is that it seems to work with most partitioning schemes out there. Here it is checking the
~ # parted /dev/sda GNU Parted 2.3 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) print print Error: /dev/sda: unrecognised disk label (parted) quit quit
Oops! "Unrecognized disk label" is parted's way of saying that drive hasn't been partitioned yet. It's probably our newly-replaced drive. Let's check the other one:
~ # parted /dev/sdb GNU Parted 2.3 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) unit kb unit kb (parted) print print Model: ATA VBOX HARDDISK (scsi) Disk /dev/sdb: 8589935kB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 2097kB 1049kB bios_grub 2 2097kB 8588886kB 8586789kB raid
That's better -- we see our GPT partition table with its two partitions: one that hosts the GRUB bootloader and another that contains the Linux
/ file system. Note that we are using kilobytes as the address unit -- that will make it easier to recreate the partitions on the new drive later.
Partitioning the new drive
Let's now start parted again on
/dev/sda and switch to kilobytes as our address units:
~ # parted /dev/sda GNU Parted 2.3 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) unit kb unit kb
We need to create a partition table first with the
(parted) mklabel gpt mklabel gpt
We are using a
gpt partitioning scheme here, but your system may use a different one -- check the available types by running
help mklabel. Let us now create the first partition, the BIOS boot one by specifying that we want a primary partition with exact start and end addresses:
(parted) mkpart primary 1049 2097 mkpart primary 1049 2097 (parted) set 1 bios_grub on set 1 bios_grub on
We give it roughly 1MB of space (note that I am using the same start/end addresses as the one we got from the
/dev/sdb printout above. In addition we set the
bios_grub flag (
1 is the partition number), so that GRUB knows where to install itself.
Now to create the data partition:
(parted) mkpart primary 2097 8588886 mkpart primary 2097 8588886 (parted) set 2 raid on set 2 raid on
Again, we use the same start/end addresses as the ones for
/dev/sdb2 as we want an identical partition. We also set the
raid flag on to specify this partition is a part of a Linux RAID array.
Let's check our work now:
(parted) print print Model: ATA VBOX HARDDISK (scsi) Disk /dev/sda: 8589935kB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1049kB 2097kB 1049kB primary bios_grub 2 2097kB 8588886kB 8586789kB primary raid
And we are done. Good job!
Starting the RAID array in degraded mode
So far, so good. Let's see in what state our RAID array is. We check
~ # cat /proc/mdstat Personalities : md127 : inactive sdb2(S) 8381440 blocks super 1.2 unused devices: <none>
Yes, We have an inactive
md127 array (I'd appreciate a
md1 simple name, but apparently the Ubuntu installer thinks otherwise.) that contains just the
sdb2 partition. Let's try getting details via the
mdadm helper tool:
~ # mdadm --detail /dev/md127 mdadm: md device /dev/md127 does not appear to be active.
Okay, I get it -- the array is inactive. Let's activate it.
~ # mdadm --manage --run /dev/md127 mdadm: started /dev/md127
Mini rant: I love the terminology soup here. One does not simply "activate" an inactive RAID array. You have to "run" it. Oh, and to deactivate it, you need the "stop" command.
Let us look at the state of
~ # mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Thu Oct 25 15:24:34 2012 Raid Level : raid1 Array Size : 8381376 (7.99 GiB 8.58 GB) Used Dev Size : 8381376 (7.99 GiB 8.58 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Update Time : Thu Oct 25 16:18:54 2012 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Name : ubuntu-gpt:0 (local to host ubuntu-gpt) UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc Events : 64 Number Major Minor RaidDevice State 3 8 18 0 active sync /dev/sdb2 1 0 0 1 removed
It's alive! Yes, it is in a degraded state, running off only
/dev/sdb2, but it's mountable and all.
Rebuilding the array
We have a perfectly-partitioned new drive now, and we are not afraid to use it. We add its
/dev/sda2 partition to the array like this:
~ # mdadm --manage --add /dev/md127 /dev/sda2 mdadm: added /dev/sda2
The kernel will automatically start syncing data from
/dev/sda2, so that both partitions contain exact copies of the data. You check its progress with
~ # mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Thu Oct 25 15:24:34 2012 Raid Level : raid1 Array Size : 8381376 (7.99 GiB 8.58 GB) Used Dev Size : 8381376 (7.99 GiB 8.58 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Oct 25 16:28:44 2012 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 1% complete Name : ubuntu-gpt:0 (local to host ubuntu-gpt) UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc Events : 68 Number Major Minor RaidDevice State 3 8 18 0 active sync /dev/sdb2 2 8 2 1 spare rebuilding /dev/sda2
It may be a good idea to wait for the rebuild to finish and the array to get promoted to clean status.
Repairing the bootloader
So, you think you have fixed your broken hard drive problem? Well, almost -- you have just fixed your RAID array. You still need to make that drive bootable. We do that by using the existing GRUB configuration and just reinstalling the bootloader code.
To access the existing GRUB configuration, we need to mount the partition inside our RAID array. We'll mount it under
~ # mkdir /mnt/rescue ~ # mount /dev/md127 /mnt/rescue ~ # ls -l /mnt/rescue drwxr-xr-x 2 root root 4096 Oct 24 14:57 bin drwxr-xr-x 3 root root 4096 Oct 24 15:01 boot drwxr-xr-x 12 root root 3900 Oct 25 16:27 dev ...
As you can see, we can access the data in our file system. We now need to run
grub-install as if it's run off the original filesystem. To do that we need two things:
- Chroot to
grub-installcan locate its config files in the usual locations.
- Expose our devices from
/mnt/rescue/devso that GRUB can access them.
We first forward
/mnt/rescue/dev using a little-known feature of Linux's
~ # mount -o bind /dev /mnt/rescue/dev
We can then use
chroot to run a
/bin/bash shell below
~ # chroot /mnt/rescue /bin/bash
We are now rooted in the right area, and we can finally reinstall GRUB to our new drive:
root@ubuntu-gpt:/# grub-install /dev/sda Installation finished. No error reported.
GRUB is smart enough to detect the partitioning scheme and will install itself either to the master boot record for MBR partitions or to the BIOS boot partition for GPT schemes.
We are done here. Let us reboot the system and login as normal.
Checking our work
After logging in to our system, we inspect the state of the RAID array:
root@ubuntu-gpt:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md127 : active raid1 sdb2 sda2 8381376 blocks super 1.2 [2/2] [UU] unused devices: <none>
root@ubuntu-gpt:~# mdadm --detail /dev/md127 /dev/md127: Version : 1.2 Creation Time : Thu Oct 25 18:24:34 2012 Raid Level : raid1 Array Size : 8381376 (7.99 GiB 8.58 GB) Used Dev Size : 8381376 (7.99 GiB 8.58 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Oct 25 19:37:45 2012 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Name : ubuntu-gpt:0 (local to host ubuntu-gpt) UUID : f2c2e89d:143ea70b:6b6878e9:ebfaecbc Events : 93 Number Major Minor RaidDevice State 3 8 18 0 active sync /dev/sdb2 2 8 2 1 active sync /dev/sda2
Everything is humming along just fine. Mission accomplished.