RAID On Linux

So you’ve decommissioned an old Windows machine and you want to use it in your proper Linux system. You dd from /dev/zero all over it and are ready to format it. But then you find that there’s a weird and tenacious entry in lsblk or /dev/mapper/ belonging to it. How to get Linux to stop worrying about it?

Problem solved.

Set up partitions to use the partition type "fd" which is "Linux raid autodetect".

Here I am making a boot partition, a swap partition and 2 identically sized data partitions. These can be made into RAID volumes with each other or across other physical drives. I leave the swap as swap because I don’t think it’s entirely useful to have swap space RAID protected.

Once you have the partition table the way you want it on one physical drive, you can clone it to another by doing something like this:

You need to have the Linux RAID enabled in the kernel. From an install disk you might need to:

(Or whatever raid level you’re after). To get this module configure here:

You might also need this if it’s not already there:

Now set up some md devices (md=multi disk, I think):

Time to actually setup the raid devices:

Here’s an example of a serious RAID setup:

Here’s one with a hot spare:

Note that if the RAID volumes were already set up, then just make the devices with mknod and then instead of "creating" the RAID components, just re"assemble" them like this:

Note that if the RAID volumes were already set up and there are redundant disks, it might happen that the RAID volume starts without its redundant mirror. For example, if you find:

And you know that it should be being mirrored with /dev/sdb1, then do this:

And then look for this:

This seems to finish right away, but actually takes a long time to get established properly. Presumably this can be done with a drive full of data and this process copies everything.

You can check up on a running array with this.

Note that a better way to get more information about what is going on with a RAID array is through mdadm:

Or

For information about the components of a RAID set use examine.

To stop a RAID array basically means to have the system stop worrying about it:

This is useful if you start an array but something goes wrong and you need to reuse the disks that are part of the aborted or inactive or unwanted array.

Since this is Linux, you can get an absurd amount of information about the status of your RAID volume. Check out the virtual /sys/block/md1/... directory tree and explore very fine details of the setup. Read about what all that good stuff is in the kernel’s md.txt documentation.

Once you’re comfortable with checking on the status of the RAID, a good thing to do is to actively test it for behavior under stress. This is a good guide to testing RAID setups.

For temporary purposes, you can setup a /etc/mdadm.conf file like this:

but for a more permanent installation (once the OS is installed), this is a good format for this file:

Don’t forget that your kernel will want a root=/dev/md3 parameter.

So imagine that you have some error like this:

The second partition on the first disk is dead and not being used. Install a new disk which will then show unused partitions for the blank drive.

The partition table has to be recreated. Make sure you get this right!!

From this you can see that hdd is the blank drive - no doubt about it. So don’t copy sdd’s table to sdc; that would be bad. Note also that the drives are identical size. This is good. In fact, it might be smart to partition to begin with with a few percent not used to account for the difference between various brands in a nominal size.

Rebuild the partition table using the same technique used at installation:

Double check:

Set up swap space on the other volume and go ahead and use it:

Sometimes you might run one drive for a while and then introduce it’s matching RAID mirror and the mirror will spontaneously associate with an md device. Here’s how to clear that before adding it back to the real md set.

If I add hdc1 and it shows an md0 that shouldn’t exist, like this:

Mark it as failed:

Or if it’s running and you don’t want it to be:

And then:

This actually didn’t seem to work! I just fixed the mdadm.conf and rebooted. Maybe I needed to:

Sometimes none of this works! You have to use dd. You can overwrite the whole thing but try this first.

Do the small easy one first:

Now check to see that it’s working:

Check again to see it complete:

Do the rest:

These things go sequentially, so don’t panic if one waits for the other to finish. If you issue these commands, they’ll work eventually.

You can keep an eye on it:

That might look like:

Go ahead and put the bootloader in the MBR on the new drive using the procedure outlined above.

Maybe you’ve decommissioned a pair of mirrored RAID drives and you’re thinking of reusing them but you want to see what was on them to make sure it’s nothing important. Here is how to simply mount this drive on a different Linux box. You can check to see that the drive is plugged in and recognized somewhere with:

There it is, sdb. But if you try:

You get:

It doesn’t work like that. You need to make sure all your kernel modules are happy for RAID (or have them compiled in works too):

The next thing to have ready is mdadm. This can be installed with apt install mdadm but I found that it installed a lot of serious cruft that I didn’t want (email stuff, MySQL, etc). I found that to just get a quick easy mdadm this was effective.

If that’s good and mdadm is installed then you can do:

And you’re looking at the contents.

Recently I’ve been having a lot of trouble with RAID volumes showing up with names like /dev/md127. This seems especially problematic with making RAID volumes which are for a Gentoo system made with sysresccd. I make the volumes something sensible like dev/md1 and 1. it doesn’t boot into the system and 2. when analyzed in a subsequent boot of sysresccd, they show up with the strange device names.

This could have something to do with:

The exact problem seems to be described here as:

Fun fact: I had given up on my previous setup and was going to redo the entire RAID. One of the changes I planned to make was to not use partitions and try to just make a RAID volume from a couple of complete drives (/dev/sd[ab]). What shocked me was that when I created the array as such, it started mirroring the data that was on one of the drives which was left over from a set that I’d previously installed an OS on using /dev/sda1 and /dev/sdb1. The fact that this was possible and automatic is either extremely wonderful or unnerving.

A mitigation?

Sadly, I believe that sysresccd has been responsible for some of the problems I’ve been experiencing. It seems that it aggressively tries to automatically assemble RAID volumes and that’d be fine except that it seems to corrupt them in certain cases.

Knoppix

First find the problem. Use a command much like this:

This shows that drive 9 is showing signs of flakiness and though it may work great, it may not and should be replaced. It is likely that if there is a problem drive, it will be isolated on it’s own RAID unit as the controller will grab the good drive from the hot spare and start using it.

Now extract the bad drive. This can be very confusing. I have proven twice this month that the labels that are on hb are good. There are no labels on puzzlebox. It’s very important to pull the right drive, especially when things are bad, and this can be confusing, so be careful.

This is as expected and now the hot spare unit (u1) goes away.

Put a new drive in. Much easier than step #2 - don’t forget to bring a Phillips screwdriver to the machine room, by the way.

So now drive 9 is back in and it seem ok, but it’s not part of any unit. It’s basically doing nothing useful.

Assign the new drive to a hot spare group. This is actually the important and non-obvious part of the operation.

Now the controller knows that this is a hot spare. Everything looks good again.

Immediately go buy a replacement! For hb, there’s one in my office that should not be used for anything but hb and there’s one in the machine room locker which the full sys admin team can get at in my absence. They should both be there!

Bonus step: send the dead drive in to the manufacturer if it’s still under warranty. Can’t have too many of these things lying around.

I usually put this utility at /root/cli64. To get help do this.

Show status of current raid environment. This is good for diagnostic checks.

Note that drive info also may work, but I’ve found that it sometimes does not so disk info should be considered canonical.

If there is a failure you probably need to pull that failed disk out. It is important to get the correct number. The proper number for finding the physical disk is in the column labeled "Slot#".

There is a password protection. I generally have no need for it and will set it to "0000". This command must be issued in the session that requires password clearance. Just do this before proceeding.

However, I did confirm that the password does not need to be set to do a volume check. To check raid consistency in a volume (and possibly to trigger any deep I/O errors) try this.

Note that the number seems to be the # when you do vsf info. So "1" for "vs0", and "3" for "vs2". Using vsf info will show you the progress of a running check.

To set auto activation of an incomplete RAID. 1 is on, 0 is off. Unfortunately, this doesn’t always work.

To see what is going on with a "raid set":

If you pull the bad drive and replace it, you might get a "Free" status. This doesn’t help anything. Either your previous Hot Spares are hard at work now becoming primary working drives or you will be activating them to do so. Either way, you want that "Free" drive to be a Hot Spare (assuming you weren’t dumb enough to set up a system without a hot spare). To do this you need the following command with the number of the drive. The number is important to get right and confusing. The disk info command has a column labeled "CLI> #", the first column. Use this number to specify which drive to turn into a hot spare. This is different (probably) from which bay to pull a physical drive out of for a failure. For example, this is to turn the drive in physical by 17 into a hot spare.

Sometimes there is a failure and the raid set just sits there. I think this tends to happen when the drive fails during a reboot (which is a time slightly more prone to failure). Here is a sequence where I check the raid set which contained a "Failed" that was replaced and turned to "Free". That is not ok since you actually want the drive’s status to be "rs2" or whatever raid set is correct and the rsf info’s "Raid Set State" to be "Rebuilding".

How’s that rebuild going? Check with:

If there are problems, check the log:

Areca support likes to see what you’re running:

Download manual and MegaCLI program at:

http://www.avagotech.com/support/download-search

Put "MegaCLI" in the "keyword" field of the search (nothing else).

Look for the one at the top called MegaCLI 5.5 T2. I think the rest are not useful or necessary. This creates a file called 8-07-14_MegaCLI.zip which contains binaries for many platforms. The Linux one is really an RPM (good job with the double compression).

The RPM installs the following files.

If trying to use this on a (64bit!) SysrescCD (or some non Red Hat system) use

This will put the correct files in place in /opt.

Or to just extract the files you need directly with no RPM installation do something like this.

This general summary from Cisco shows a lot of good usage tips for MegaCLI including the replacing of a drive.

https://supportforums.cisco.com/document/62901/megacli-common-commands-and-procedures

The disks in example.edu appear to be these.

http://www.amazon.com/RE4-Enterprise-Hard-Drive-WD2003FYYS/dp/B002XW44QY/

They are Dell branded Western Digital 2TB, 3.5", 7200 RPM, SATA II, 64MB Cache

Specifically -

The drive bay labels are consistent with this.

When the drives are good (#7 has just been replaced) it looks like this.

A failure looks like this.

The correct status looks like this.

An even better check.

Here’s a full script to run from cron. Run with no args it’s silent unless there are problems. Run with any kind of args, it prints the full report. Run the no arg version every 15 min or so and the arg version every day to get a positive assertion that the check is working.

Yes, this script is uncool because there should be a hot spare making for 2 different states when no drives have failed. But hey, I didn’t set that up. Just keep it in mind and adjust the script as necessary.