mdadm

From Wikipedia, the free encyclopedia - View original article

mdadm
Original author(s)Neil Brown
Developer(s)Neil Brown
Initial release2001
Stable release3.3[1] / September 3, 2013; 3 months ago (2013-09-03)
Development statusActive
Written inC
Operating systemLinux
Available inEnglish
TypeDisk utility
LicenseGNU General Public License
Websitehttp://neil.brown.name/blog/mdadm
 
Jump to: navigation, search
mdadm
Original author(s)Neil Brown
Developer(s)Neil Brown
Initial release2001
Stable release3.3[1] / September 3, 2013; 3 months ago (2013-09-03)
Development statusActive
Written inC
Operating systemLinux
Available inEnglish
TypeDisk utility
LicenseGNU General Public License
Websitehttp://neil.brown.name/blog/mdadm

mdadm is a Linux utility used to manage software RAID devices.

The name is derived from the md (multiple device) device nodes it administers or manages, and it replaced a previous utility mdctl. The original name was "Mirror Disk", but was changed as the functionality increased.

It is free software licensed under version 2 or later of the GNU General Public License - maintained and copyrighted to Neil Brown of SUSE.

Functionality[edit]

Physical devices[edit]

Linux software RAID configurations can include anything presented to the Linux kernel as a block device. This includes whole hard drives (for example, /dev/sda), and their partitions (for example, /dev/sda1).

RAID configurations[edit]

Note that RAID 10 is distinct from RAID 0+1, which consists of a top-level RAID 1 mirror composed of high-performance RAID 0 stripes directly across the physical hard disks. A single-drive failure in a RAID 10 configuration results in one of the lower-level mirrors entering degraded mode, but the top-level stripe performing normally (except for the performance hit). A single-drive failure in a RAID 0+1 configuration results in one of the lower-level stripes completely failing, and the top-level mirror entering degraded mode. Which of the two setups is preferable depends on the details of the application in question, such as whether or not spare disks are available, and how they should be spun up.

Non-RAID configurations[edit]

Types of MD devices[edit]

The original (standard) form was /dev/md<n> where <n> is a number between 0 and 99. More recent kernels have supported the use of names such as /dev/md/Home. Under kernel 2.4 and earlier these two were the only options. Both of them are non-partitionable.

From kernel 2.6 a new type of MD device was introduced, a partitionable array. The device names were modified by changing md to md_d. The partitions were identified by adding p<m>; thus /dev/md/md_d2p3 for example.

From kernel 2.6.28 non-partitionable arrays can be partitioned, the partitions being referred to in the same way as for partitionable arrays — /dev/md/md1p2, for example.

Booting[edit]

Since support for MD is found in the kernel, there is an issue with using it before the kernel is running. Specifically it will not be present if the boot loader is either (e)LiLo or GRUB legacy. It may not be present for GRUB 2. In order to circumvent this problem a /boot filesystem must be used either without md support, or else with RAID1. In the latter case the system will boot by treating the RAID1 device as a normal filesystem, and once the system is running it can be remounted as md and the second disk added to it. This will result in a catch-up, but /boot filesystems are usually small.

With more recent bootloaders it is possible to load the MD support as a kernel module through the initramfs mechanism, this approach allows you to let the /boot filesystem be inside any RAID system without the need of a complex manual configuration.

External metadata[edit]

Besides its own formats for RAID volumes metadata, Linux software RAID also supports external metadata formats, since version 2.6.27 of the Linux kernel and version 3.0 of the mdadm userspace utility. This allows Linux to use various firmware- or driver-based RAID volumes, also known as "fake RAID".[2]

As of October 2013, there are two supported formats of the external metadata:

Quick reference[edit]

Creating an array[edit]

 mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/sda1 /dev/sdb1 

Create a RAID 1 (mirror) array from two partitions. If the partitions differ in size, the array is the size of the smaller partition. You can create a RAID 1 array with more than two devices. This gives you multiple copies. Whilst there is little extra safety in this, it makes sense when you are creating a RAID 5 array for most of your disk space and using RAID 1 only for a small /boot partition. Using the same partitioning for all member drives keeps things simple.

 mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sda2 /dev/sdb2 /dev/sdc2 

Create a RAID 5 volume from three partitions. If the partitions used in your RAID array are not the same size, mdadm will use the size of the smallest from each partition. If you receive an error, such as: "mdadm: RUN_ARRAY failed: Invalid argument", make sure your kernel supports (either via a module or by being directly compiled in) the raid mode you are trying to use. (Most modern kernels do.)

It is possible to create a degraded mirror, with one half missing by replacing a drive name with "missing":

 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 missing 

The other half mirror is added to the set thus:

 mdadm --manage /dev/md1 --add /dev/sda1 

This is useful when you are adding a disk to a computer which currently isn't mirrored. The new drive is...

The computer is then booted off the secondary drive (or a rescue disk), the now idle original disk can be repartitioned if required (no need to format), and then the primary drive submirrors are added.

Note that the partition types should be changed to 0xFD with fdisk to indicate that they are mirrored devices.

Recording the array[edit]

 mdadm --detail /dev/md0 

View the status of the multi disk array md0.

 mdadm -Es | grep md >> /etc/mdadm/mdadm.conf 

This adds md0 to the configuration file so that it is recognised next time you boot.

You may wish to keep a copy of /proc/mdstat on another machine or as a paper copy. The information will allow you to restart the array manually if mdadm fails to do so.

Growing an array by adding devices[edit]

 mdadm --add /dev/md1 /dev/sdd1 mdadm --grow /dev/md1 --raid-devices=4 

This adds the new device to the array then grows the array to use its space.

In some configurations you may not be able to grow the array until you have removed the internal bitmap. You can add the bitmap back again after the array has been grown.

 mdadm --grow /dev/md1 -b none mdadm --grow /dev/md1 -b internal 

Growing an array by upgrading devices[edit]

An array may be upgraded by replacing the devices one by one, either as a planned upgrade or ad hoc as a result of replacing failed devices.

 mdadm /dev/md1 --fail /dev/sda1 # replace the first drive with the new, larger one then partition it mdadm --add /dev/md1 /dev/sda1 

Allow the new drive to resync. If replacing all the devices repeat the above for each device, allowing the array to resync between repetitions. Finally, grow the array to use the maximum space available and then grow the filesystem(s) on the RAID array to use the new space.

 mdadm --grow /dev/md1 --size=max 

Deleting an array[edit]

 mdadm --stop /dev/md0  # to halt the array mdadm --remove /dev/md0  # to remove the array mdadm --zero-superblock /dev/sd[abc]1  # delete the superblock from all drives in the array # edit /etc/mdadm/mdadm.conf to delete any rows related to deleted array 

Convert an existing partition to RAID 5[edit]

Assume that the existing data is on /dev/sda1:

 mdadm --create /dev/md1 --level=5 --raid-devices=3 missing /dev/sdb2 /dev/sdc2 mdadm -Es >>/etc/mdadm/mdadm.conf update-initramfs -u dd if=/dev/sda1 of=/dev/md1 # replace /dev/sdaX with /dev/md1pX in your boot loader's menu # reboot into /dev/md1 # format /dev/sdaY by marking the partition as autoraid mdadm --add /dev/md1 /dev/sda1 # update your boot loader 

Notes:

Mdmpd[edit]

Mdmpd is a daemon used for monitoring MD multipath devices, developed by Red Hat as part of the mdadm package.[4] The program is used to monitor multipath (RAID) devices, and is usually started at boot time as a service, and afterwards running as a daemon.

Enterprise storage requirements often include the desire to have more than one way to talk to a single disk drive so that in the event of some failure to talk to a disk drive via one controller, the system can automatically switch to another controller and keep going. This is called multipath disk access. The linux kernel implements multipath disk access via the software RAID stack known as the md (Multiple Devices) driver. The kernel portion of the md multipath driver only handles routing I/O requests to the proper device and handling failures on the active path. It does not try to find out if a path that has previously failed might be working again. That's what this daemon does. Upon startup, the daemon will fork and place itself in the background. Then it reads the current state of the md raid arrays, saves that state, and then waits for the kernel to tell it something interesting has happened. It then wakes up, checks to see if any paths on a multipath device have failed, and if they have then it starts to poll the failed path once every 15 seconds until it starts working again. Once it starts working again, the daemon will then add the path back into the multipath md device it was originally part of as a new spare path.

If one is using the /proc filesystem, /proc/mdstat lists all active md devices with information about them. Mdmpd requires this to find arrays to monitor paths on and to get notification of interesting events.

Known problems[edit]

A common error when creating RAID devices is that the dmraid driver has taken control of all the devices that are to be used in the new RAID device. Error messages like this will occur:

 mdadm: Cannot open /dev/sdb1: Device or resource busy 

Typically, the solution to this problem involves adding the "nodmraid" kernel parameter to the boot loader config. Another way this error can present itself is if the device mapper has its way with the drives. Issue "dmsetup table" to see if the drive in question is listed. Issuing "dmsetup remove <drive id>" will remove the drive from device mapper and the error message from above will go away as well.

RAID already running[edit]

First check that the device is not already used in another array:

 cat /proc/mdstat 

Probably you will have to stop the array with:

 mdadm --stop /dev/md<xxx> 

Check the /etc/mdadm/mdadm.conf file (and restart the system if possible):

 vi /etc/mdadm/mdadm.conf 

Then you should be able to delete the superblock of this device:

 mdadm --zero-superblock /dev/sd<x>N 

Now the device should be no longer marked as busy.

Sometimes dmraid "owns" the devices and will not let them go. There is a solution.[5]

Tweaking the kernel[edit]

To solve this problem, you need to build a new initrd without the dmraid driver. The following command does this on a system running the 2.6.18-8.1.6.el5 kernel:

 mkinitrd --omit-dmraid /boot/NO_DMRAID_initrd-2.6.18-8.1.6.el5.img 2.6.18-8.1.6.el5 

After this, the system has to be rebooted with the new initrd. Edit your /boot/grub/grub.conf to achieve this.

Alternatively if you have a self customized and compiled kernel from a Linux distribution like Gentoo (the default option in Gentoo) which does not use initrd then check kernel .config file in /usr/src/linux for the line

 # CONFIG_BLK_DEV_DM is not configured 

If the above line is set as follows:

 CONFIG_BLK_DEV_DM=y 

then you might have to disable that option, recompile the kernel, put it in /boot and finally edit grub config file in /boot/grub. PLEASE be careful NOT to disable

 CONFIG_BLK_DEV_MD=y 

(Note the MD instead of DM) which is essential for raid to work at all!

If both methods have not helped you then booting from live CD probably will (the below example is for starting a degraded RAID-1 mirror array and adding a spare HDD to it and syncing. Creating a new one should not be more difficult because the underlying problem was a "Device or resource busy" error):

 modprobe raid1 mknod /dev/md1 b 9 1 mknod /dev/md3 b 9 3 mdadm --assemble /dev/md1 /dev/hda1 mdadm --assemble /dev/md3 /dev/hda3 mdadm --add /dev/md1 /dev/hdb1 mdadm --add /dev/md3 /dev/hdb3 

It might be easier to try and automatically assemble the devices:

 mdadm --assemble --scan 

Remember to change the corresponding md* and hd* values with the corresponding ones from your system. You can monitor the sync progress using:

 cat /proc/mdstat 

When the sync is done you can reboot the computer as usual.

Zeroing the superblock[edit]

Another way to prevent the kernel autostarting the raid is to remove all the previous raid-related information from the disks before proceeding with the creation, for example:

 mdadm --stop /dev/md0 mdadm --zero-superblock /dev/sd[abcd]1 

And now the usual create, for example:

 mdadm --create /dev/md0 --level=5 --raid-devices=4 --spare-devices=0 /dev/sd[abcd]1 

Recovering from a loss of RAID superblock[edit]

There are superblocks on the drives themselves and on the RAID (apparently). If you are unable to recover a RAID set damaged due to power outage or hardware failure, try the following procedure in order to re-create the superblocks and make your data accessible again.

Get a list of the devices in the raid in question:

 mdadm --detail /dev/md<x> 

Output of the command above looks like this:

 /dev/md127:         Version : 1.2   Creation Time : Sun Aug 21 23:35:28 2011      Raid Level : raid6      Array Size : 7814047744 (7452.06 GiB 8001.58 GB)   Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)    Raid Devices : 6   Total Devices : 4     Persistence : Superblock is persistent     Update Time : Sun Jan  1 11:43:17 2012           State : clean, degraded   Active Devices : 4 Working Devices : 4  Failed Devices : 0   Spare Devices : 0          Layout : left-symmetric      Chunk Size : 512K            Name : clop:1  (local to host clop)            UUID : 7ee1e93a:1b011f80:04503b8d:c5dd1e23          Events : 62     Number   Major   Minor   RaidDevice State        0       8       33        0      active sync   /dev/sdc1        1       8       49        1      active sync   /dev/sdd1        2       8       81        2      active sync   /dev/sdf1        3       8       65        3      active sync   /dev/sde1        4       0        0        4      removed        5       0        0        5      removed 

RaidDevice order (sdc1, sdd1, sdf1, sde1) and Chunk Size value are critical.

Record all your raid member parameters:

 mdadm --examine /dev/sd[abcde...]1 | egrep 'dev|Update|Role|State|Chunk Size' 

Look carefully at the Update time values. If you have RAID members attached to the motherboard and others attached to a RAID card, and the card actually failed while leaving enough members to keep the RAID set operational, you want to make a note of that. Look at the Array State and Update Time values, for example (output is separated with blank lines for better readability):

 /dev/sdc1:    Update Time : Wed Jun 15 00:32:35 2011   Array State : AAAA.. ('A' == active, '.' == missing)  /dev/sdd1:    Update Time : Thu Jun 16 21:49:27 2011   Array State : .AAA.. ('A' == active, '.' == missing)  /dev/sde1:    Update Time : Thu Jun 16 21:49:27 2011   Array State : .AAA.. ('A' == active, '.' == missing)  /dev/sdf1:    Update Time : Thu Jun 16 21:49:27 2011   Array State : .AAA.. ('A' == active, '.' == missing)  /dev/sdk1:    Update Time : Tue Jun 14 07:09:34 2011   Array State : ....AA ('A' == active, '.' == missing)  /dev/sdl1:    Update Time : Tue Jun 14 07:09:34 2011   Array State : ....AA ('A' == active, '.' == missing) 

Devices sdc1, sdd1, sde1 and sdf1 are the last members in the array and they will rebuild correctly. On the other side, sdk1 and sdl1 are droppeed from the array (in this example, due to a RAID card failure).

Also note the RAID member, starting with 0, as the RAID set needs to be rebuilt in the same order. Chunk Size value is also important.

Wipe the drives' superblocks:

 mdadm --stop /dev/md0    # halt the array mdadm --remove /dev/md0  # remove the array mdadm --zero-superblock /dev/sd[cdefkl]1 

Reassemble the RAID set:

 mdadm --create /dev/md1 --chunk=4096 --level=6 --raid-devices=6 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sde1 missing missing 

Parameter missing tells the create command to rebuild the RAID set in a degraded state. Devices sdk1 and sdl1 can be added later.

Edit /etc/mdadm.conf and add an ARRAY line with the appropriate UUID value. In order to do that, first find the UUID for your RAID set:

 mdadm -D /dev/md<x> 

And then make the changes:

 vi /etc/mdadm.conf 

Example of the configuration file is below — note there must be no # in front of the active line you are adding:

 #ARRAY /dev/md0 UUID=3aaa0122:29827cfa:5331ad66:ca767371 #ARRAY /dev/md1 super-minor=1 #ARRAY /dev/md2 devices=/dev/hda1,/dev/hdb1 ARRAY /dev/md1 UUID=7ee1e93a:1b011f80:04503b8d:c5dd1e23 

Save your edited /etc/mdadm.conf.

Last, mark the array as possibly dirty with:

 mdadm --assemble /dev/md1 --update=resync 

Monitor the rebuild with:

 watch -n 1 cat /proc/mdstat 

All your data should be recovered and accessible!

Increasing RAID ReSync Performance[edit]

In order to increase the resync speed, we can use a bitmap, which mdadm will use to mark which areas may be out-of-sync. Add the bitmap with the grow option like below:

 mdadm -G /dev/md2 --bitmap=internal 
Note: mdadm – v2.6.9 – 10 March 2009 on Centos 5.5 requires this to be run on a stable "clean" array. If the array is rebuilding the following error will be displayed:
 mdadm: failed to set internal bitmap. 
And the following line is added to the log file:
 md: couldn't update array info. -16 

then verify that the bitmap was added to the md2 device using

 cat /proc/mdstat 

you can also adjust Linux kernel limits by editing files /proc/sys/dev/raid/speed_limit_min and /proc/sys/dev/raid/speed_limit_max.

You can also edit this with the sysctl utility

 sysctl -w dev.raid.speed_limit_min=50000 

References[edit]

  1. ^ Brown, Neil (2013-09-03). "ANNOUNCE: mdadm 3.3 - A tool for managing Soft RAID under Linux". 
  2. ^ a b "External Metadata". RAID Setup. kernel.org. 2013-10-05. Retrieved 2014-01-01. 
  3. ^ "DDF Fake RAID". RAID Setup. kernel.org. 2013-09-12. Retrieved 2014-01-01. 
  4. ^ "Updated mdadm package includes multi-path device enhancements". RHEA-2003:397-06. Redhat. 2004-01-16. 
  5. ^ Jonik (2010-01-29). "How to build initrd without dmraid-driver on Ubuntu 9.10?". 

External links[edit]