Stripes and mirrors • 1 Nov 2008
Note: This article is about GEOM software RAID only.
Stripes and mirrors are forms of RAID, which lets you combine multiple harddisks into one logical volume with some interesting properties.
Stripes combine harddisks by concatenating and interleaving them, for example if you have two disks of 160 GB each, they can be combined to form a stripe of 320 GB. This optimizes for size, but also for speed, since data access can be divided among multiple harddisks. The big disadvantage is that when one disk fails, all the data on the stripe is lost, with no chance of recovery.
Mirrors combine harddisks by cloning them. Two harddisks of 160G will give you a mirror of 160GB. Mirrors optimize for fault tolerance. If one disk breaks, the data is still intact on the other disk. Running applications don't even notice that a harddisk failed. I use mirrors on all my server systems, and it has already saved me from a few harddisk failures.
Recovery from a broken mirror is easy: you replace the failed harddisk with a new one (the same size, or bigger), and then add this new disk to the mirrorset, and the operating system will copy the entire surface from the old disk to the new one. This can take quite some time, but it happens in the background, so the system can be running as usual, albeit a little more loaded due to the copying.
Also, a big advantage of mirrors is that you can mount an individual disk from a set into another system and still access it as a complete volume, should that ever be necessary. At least, it's possible. This is something you can't do with stripes.
Stripes are called RAID0, and mirrors RAID1. Stripes and mirrors can also be combined, for example as follows: You create two mirror sets of two disks each. Then you create a stripe of these two mirrorsets. This way, you get safety and speed. This is called RAID10.
There also exist RAID schemes where multiple disks are striped together and yet redundancy is available.
This happens in RAID5. The redundancy is done by interleaving the data with parity sectors.
In this way it's possible to combine four drives of 250GB into one volume of 750GB and still have resilience against a drive failure.
Mirroring these four disks in pairs would give 500GB, and a stripe 1000GB, so RAID5 is in the middle as a kind of compromise.
However, I find RAID5 too risky for redundancy purposes in servers, since any individual harddisk in such a set cannot be accessed on its own. And harddisks come in such huge sizes nowadays, that the argument that RAID5 provides more space than mirroring with the same amount of disks is less valid than it used to be 10 years ago.
Adding two harddisks as a stripe
To salvage pictures from a badly damaged Windows system, I wanted to make an image copy of a 320GB harddisk, but I had no other harddisk lying around big enough to hold this. However, I did find two 250GB harddisks. By combining them into a stripe, I created a volume which could hold 500GB.
My system boots from /dev/ad0. I plugged in the two harddisks, rebooted, and saw in /dev that they were recognized as ad6 and ad8. With the following commands I prepared them for use:
# gstripe label gs0 /dev/ad6 /dev/ad8 # newfs /dev/stripe/gs0
Then I plugged in the disk from the Windows system, rebooted, and noticed that FreeBSD saw it as /dev/ad2.
# mount /dev/stripe/gs0 /mnt # dd bs=1m if=/dev/ad2 of=/mnt/salvage.img
Converting an existing FreeBSD installation to a mirror
Suppose your disks are ad4 and ad6, and you have FreeBSD installed on ad4, and this is also the disk from which the system boots. To convert this system to use ad4 and ad6 as a mirror set, use these steps:
# sysctl kern.geom.debugflags=16 # gmirror label gm0 /dev/ad4
The sysctl is necessary because otherwise FreeBSD won't allow you to label the boot volume.
To be sure the kernel loads the gmirror driver during startup, put the following line in /boot/loader.conf:
Change /etc/fstab so that ad4 is changed in gm0, and /mirror is put behind /dev. The effect of this is that the name /dev/ad4s1a changes into /dev/mirror/gm0s1a
Now reboot the system. Log in as root.
Add the ad6 disk to the mirror set:
# gmirror insert gm0 /dev/ad6
The mirror starts synchronizing. You can keep an eye on the progress by giving the gmirror status command from time to time. When the synchronisation is complete, the output looks like this:
# gmirror status Name Status Components mirror/gm0 COMPLETE ad4 ad6
Recovering from harddisk failure
Emulating a harddisk failure is pretty easy: just yank the power cable from a harddisk ;-) When you do this, the gmirror status command will show the mirror set as degraded. In the example below, I pulled the power cable from ad6. The status command shows the set is 'degraded', and a list of disks that are still functioning:
# gmirror status Name Status Components mirror/gm0 DEGRADED ad4
To recover, replace the failed harddisk with a good one, the same size or bigger. I simply reconnected the power cable, rebooted FreeBSD so it saw /dev/ad6 again, and gave the following commands:
# gmirror forget gm0 # gmirror insert gm0 /dev/ad6 ...wait a bit... # gmirror status Name Status Components mirror/gm0 DEGRADED ad4 ad6 (2%)
If you have the luxury of hot-swappable harddisks, rebooting isn't even necessary.