1. Introduction
1.1. Credits
This document is based on the following documents, and a few chapters are verbatim from these.
- An introduction to Vinum on FreeBSD by Rocky
- Chapter 13 of the FreeBSD handbook: The Vinum Volume Manager, originally written by Greg Lehey.
- Vinum HOWTO for two mirrored disks, by mic at netbase dot org.
- Bootstrapping Vinum by Robert A. Van Valzah
- Chapter 12 of O'Reilly's The Complete FreeBSD: The Vinum Volume Manager
- Replacing a failed Vinum drive.
man vinum[4|8]- The FreeBSD-questions mailing list
1.2. Concepts
I found myself one day with a bunch of unused hard drives which I wanted to put to good use. As my file server never has too much disk space, I started to think about ways I could put my disks to use in this server. RAID was one of the first things to pop into my mind. Being a poor student, I however did not have the funds to buy a hardware RAID adapter for my server, and that's how a stumbled upon a software RAID solution for FreeBSD (and NetBSD and OpenBSD!), called 'Vinum'. My objective here is to create a functional RAID-1 (mirroring) setup using Vinum and FreeBSD.
I am by no means a guru of any of the softwares mentioned here, and therefore suggestions and comments are always welcome.
You should at least have basic knowledge of the FreeBSD installation and how partitions work in FreeBSD. Go read up on them first. Also, read (and don't just skim through) the documentations provided with the various softwares noted here. This how-to is not a replacement for those documents, but more of a complementation and quick reference. Setting up this for the first time will take a while.
Throughout this document, you will see either hash symbols (#) or dollar signs ($) in front of the commands. These indicate whether the command should be run as a regular user or with superuser privileges (i.e. root). Do not include these in the commands! And by the way, copy-pasting commands is not such a great idea. I also provide my configuration files here for reference only. Please do not just copy them over to your server without fully understanding what they do. Additionally, please do not use the authors of this documents as a technical support, which we are most certainly not. Instead, you should consult the respective softwares' manual pages, documentations, forums and mailing lists.
1.5. Disclaimer
Use the information in this document at your own risk. I disavow any potential liability for the contents of this document. Use of the concepts, examples, and/or other content of this document is entirely at your own risk.
All copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.
Naming of particular products or brands should not be seen as endorsements.
You are strongly recommended to make a backup of your system before major installation and should make backups at regular intervals.
2. Theory
2.1. Terminology
This chapter has been put together using quotes from chapter 13 of the FreeBSD handbook: The Vinum Volume Manager, originally written by Greg Lehey and Bootstrapping Vinum by Robert A. Van Valzah.
Discussion of storage management can get very tricky simply because of the terminology involved. I will refer to a physical disk drive as a "spindle".
Vinum implements a four-level hierarchy of objects:
- The most visible object is the virtual disk, called a volume. Volumes have essentially the same properties as a UNIX disk drive, though there are some minor differences. They have no size limitations.
- Volumes are composed of plexes, each of which represents the total address space of a volume. This level in the hierarchy thus provides redundancy. Think of plexes as individual disks in a mirrored array, each containing the same data.
- Vinum exists within the UNIX disk storage framework, so it would be possible to use UNIX partitions as the building block for multi-disk plexes, but in fact this turns out to be too inflexible: UNIX disks can have only a limited number of partitions. Instead, Vinum subdivides a single UNIX partition (the drive) into contiguous areas called subdisks, which it uses as building blocks for plexes.
- Subdisks reside on Vinum drives, currently UNIX partitions. Vinum drives can contain any number of subdisks. With the exception of a small area at the beginning of the drive, which is used for storing configuration and state information, the entire drive is available for data storage.
Plexes can include multiple subdisks spread over all drives in the Vinum configuration, so the size of an individual drive does not limit the size of a plex, and thus of a volume.
Robert A. Van Valzah calls the process of getting Vinum initially installed and operating to the point where you have met your resilience or performance goals "bootstrapping." I will use this term a bit more loosely; I will refer to 'bootstrapping' as the process of getting FreeBSD installed, so that even the root partition (/) is available to Vinum. There are basically two different bootstrapping methods you can use;
- The first is basically the same method as described in chapter 12 of O'Reilly's The Complete FreeBSD: The Vinum Volume Manager, pages 236-240. In this method, BSD-type partitions are created using bsdlabel/disklabel, and then simply converted into equal sized Vinum partitions. The disadvantage of this approach is the fact that the number of BSD-type partitions is limited.
- The second method does not use BSD-type partitions for dividing the disk space, but 'Vinum subdisks'. The advantage of this approach is, that the number of Vinum subdisks are not limited.
My purpose here is to document a Vinum bootstrapping method that I have found works well for me. My method also uses bits of Robert A. Van Valzah's bootstrapping method, most resembling the manual bootstrapping phases. My method has the disadvantage of requiring a bit more work, but does not require Perl.
2.2. Vinum
This chapter has been put together using quotes from chapter 13 of the FreeBSD handbook: The Vinum Volume Manager, originally written by Greg Lehey and Bootstrapping Vinum by Robert A. Van Valzah.
Vinum is a Volume Manager, a virtual disk driver that addresses these three issues:
- Disks can be too small.
- Disks can be too slow.
- Disks can be too unreliable.
From a user viewpoint, Vinum looks almost exactly the same as a disk, but in addition to the disks there is a maintenance program.
Plexes can include multiple subdisks spread over all drives in the Vinum configuration, so the size of an individual drive does not limit the size of a plex, and thus of a volume.
RAID-1 a.k.a. RAID level 1 refers to mirroring. Mirroring maintains two or more copies of the data on different physical hardware (in my case 'spindles'). Any write to the volume writes to both locations; a read can be satisfied from either, so if one spindle fails, the data is still available on the other spindle. This has two problems:
- The price. It requires twice as much disk storage as a non-redundant solution. However, many disk drive manufactures now sell 7,200 RPM disk drives with quite low seek times and high transfer rates through ATA-100 interfaces, all at very attractive prices. You can indeed get started with Vinum very simply. A minimum system can be as simple as an old CPU (even a 486 is fine) and a pair of drives that are 500 MB or more. They need not be the same size or even use the same interface (i.e., it is fine to mix ATAPI and SCSI).
- The performance impact. Writes must be performed to both spindles, so they take up twice the bandwidth of a non-mirrored volume. Reads do not suffer from a performance penalty: you only need to read from one of the disks, so in some cases, they can even be faster.
While there are other RAID levels for Vinum (0, 4 and 5), I will not go into them in this document.
2.3. Blocks and bytes
FreeBSD's disklabel and bsdlabel use blocks as measurement of disk space. One block is 512 bytes. Thus, if you needed to convert a number of blocks to megabytes, you could do something like:
4096 blocks / 2 = 2048 bytes
^ each block is 512 bytes
2048 bytes / 1024 bytes = 2MB
Here, I calculated, that 4096 blocks is equal to 2 megabytes. Converting megabytes to blocks could be done, for example, like this:
1024 bytes * 256 megabytes = 262144 bytes
First convert the megabytes to bytes (remember; one megabyte is 1024 bytes, and NOT 1000 bytes, like harddrive manufacturers are claiming). Then, multiply by two (as each block is 512 bytes):
262144 bytes * 2 = 524288 blocks
Here, I calculated, that 256MB is equal to 524288 blocks.
3. Installing FreeBSD
3.1. Installing FreeBSD
It is possible to add Vinum to a server configuration after it is already in production use, but this is much harder than designing for it from the start. Ironically, Vinum is not supported by /stand/sysinstall and therefore you cannot install /usr right onto a Vinum volume.
I'm starting off with a completely new installation of FreeBSD 5.2-RELEASE. I have also successfully installed the following setup on FreeBSD 4.9-RELEASE and 5.1-RELEASE. In my testing environment, I have two hard drives, ad0 (primary IDE master) and ad2 (secondary IDE master). ad0 is an old 6Gb Seagate, which I will be calling "YouCrazy", a.k.a. the root spindle. ad2 is a bit newer 8Gb Seagate, which I will be calling "UpWindow", a.k.a. the rootback spindle. I will setup RAID-1 (mirroring) on these two spindles, even with a mirrored root volume. Mirroring makes the drives identical.
Choose 1 from the boot menu (Boot FreeBSD default).
Choose "Standard" installation from the sysinstall Main Menu.
Select ad0 and press enter.
Press on the "A" button on your keyboard in order to use the entire disk.
Press "Q" to exit.
Select "BootMgr" from the list. Choosing BootMgr for both the drives will allow us to also boot from ad2, in case ad0 should be damaged/unbootable.
Repeat the above for ad2.
Now you will enter the Disklabel Editor.
I used this layout for ad0s1 ("YouCrazy", a.k.a. the root spindle):
Part Mount Size
ad0s1b swap 512m
ad0s1a / 1024m
ad0s1d /home 1024m
ad0s1e /tmp 512m
ad0s1f /var the rest (in my case 6424401 blocks,
i.e. about 3136 megabytes)
and this for ad2s1 ("UpWindow", a.k.a. the rootback spindle):
Part Mount Size
ad2s1b swap 512m
ad2s1d /rootback 1024m
ad2s1e /home2 1024m
ad2s1f /tmp2 512m
ad1s2g /var2 6424401 blocks
ad2s1h /hope the rest (in my case 4125744 blocks,
i.e. about 2014 megabytes)
First, I created swap partitions on both drives. It's important that you create these first, so that they get offset 0, as we will shrink them by 281 sectors each later on.
After that, I created a partition of 1024 megabytes on ad0 with the mount point / (this will be the 'a' partition). Then, I created an equally large filesystem on ad2s1, with the dummy mount point /rootback. As you may have noticed, Disklabel will name this the 'd' partition. Don't worry about that for now, we'll correct that later on.
Next, I created equal sized partitions on both slices for /home, /tmp and /var.
As my ad2 is bigger than my ad0, I created a partition with the mount point /hope using the remaining space in ad2. I will use /hope for storage of backup files, though you could use it for anything else as well. Remember kids, RAID is not a substitute for backups!
Press "Q" to exit the Disklabel Editor.
Choose "Minimal" for now. You can add software later.
Wait for the install to complete. You will be asked a few questions, but it's safe to answer "No" to all for now, in case you don't want to put time and/or effort in to these.
Choose "X Exit Install" from the sysinstall Main Menu.
3.2. Loader configuration
Your computer will be rebooted after the install has completed. Choose F1 from the BootMgr and then choose number 4 (Boot FreeBSD in single user mode). Issue the command:
# mount -u /
This will mount the root filesystem read/write instead of read-only.
Vinum must be available in the kernel at boot-time. The first option would be to compile Vinum statically into the kernel, so it is available all the time, but this is usually not desirable: in this case, you will not be able to stop Vinum. There is another option as well, to have /boot/loader load the Vinum kernel module early, before starting the kernel. This can be accomplished by putting the line:
vinum_load="YES"
into the file /boot/loader.conf.
Vinum must also be initialized early since it needs to supply the volume for the root filesystem. By default, the Vinum kernel part is not looking for drives that might contain Vinum volume information until the administrator (or one of the startup scripts) issues a 'vinum start' command.
Note: The following paragraphs are outlining the steps needed for FreeBSD 5.1-RELEASE and above. The setup required for FreeBSD 4.x differs (see the Vinum chapter in the FreeBSD handbook for more info on that).
By placing the line:
vinum.autostart="YES"
into /boot/loader.conf, Vinum is instructed to automatically scan all drives for Vinum information as part of the kernel startup.
Note that it is not necessary to instruct the kernel where to look for the root filesystem. /boot/loader looks up the name of the root device in /etc/fstab, and passes this information on to the kernel. When it comes to mount the root filesystem, the kernel figures out from the device name provided which driver to ask to translate this into the internal device ID (major/minor number).
Once you are done, save the file and exit the editor.
4. Setting up ad0s1
4.1. bsdlabel/disklabel
Next, we need to edit the disklabels. Issue the command (bsdlabel is a newer version of the traditional disklabel):
# bsdlabel -e ad0s1
The output of the command should be similar to:
# /dev/ad0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2097152 1048576 4.2BSD 0 0 0 b: 1048576 0 swap c: 12715857 0 unused 0 0 # "raw" part, don't edit d: 2097152 3145728 4.2BSD 0 0 0 e: 1048576 5242880 4.2BSD 0 0 0 f: 6424401 6291456 4.2BSD 0 0 0
Traditionally, the 'd' partition has been reserved, but has lost it's meaning since then. FreeBSD 5.1's disklabel seems to create a normal 'd' partition on the first disk.
Shorten the length of the swap partition by 281 sectors, or it will overlap the root partition and cause extreme data corruption when swap gets full, which might not happen until months later. In my case, I simply subtracted 281 from the original size of the 'b' partition, thus resulting in a value of 1048295. Then, I also changed the offset of the swap partition from 0 to 281.
Next, we'll create a new 'h' partition for Vinum. We need to subtract 16 sectors from the total size of the disk. The total size of the disk is the size of the 'c' partition. So, in my case, I subtracted 16 from 12715857 and thus got a value of 12715841. The offset of our new 'h' partition should be 16 sectors, because otherwise it will cut into the bootstrap. The type should be 'vinum'. Note that 'vinum' should be written in lowercase letters.
After the changes, your ad0s1 partition table should look like this (changes have been bolded):
# /dev/ad0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2097152 1048576 4.2BSD 0 0 0 b: **1048295 281** swap c: 12715857 0 unused 0 0 # "raw" part, don't edit d: 2097152 3145728 4.2BSD 0 0 0 e: 1048576 5242880 4.2BSD 0 0 0 f: 6424401 6291456 4.2BSD 0 0 0 **h: 12715841 16 vinum**
Save the file and exit the editor, the partition table for ad0 is now in order!
4.2. Vinum configuration for ad0s1
Now we will create a config file for Vinum. Where you put it and what you name it doesn't really matter. I prefer to name mine /etc/vinum.conf. In this file, we will explain to Vinum what we want it to do with the Vinum partitions.
First, we need to tell Vinum about our partition. The syntax for this is "drive <name> device <interface>". In my case, I chose to name the drives 'YouCrazy' and 'UpWindow', so the first two lines of my vinum.conf look like this:
drive YouCrazy device /dev/ad0s1h
The FreeBSD kernel would refer to the master spindle on the primary and secondary ATA controllers as /dev/ad0 and /dev/ad2 respectively. Vinum, however, also needs to have a name for each spindle that will stay the same name regardless of how it is attached to the CPU (i.e., if the drive moves, the Vinum name moves with the drive).
What you name your drives is up to you, but just remember the names, we will need them later. However, do not name your drives after the disk device on which they are located. The purpose of having drive names is to be device independent. For example, if you have two drives a and b, and they are located on devices /dev/da1s1h and /dev/da2s1h respectively, you can remove the drives, swap their locations and restart Vinum. Vinum will still correctly locate the drives. If you had called the drives da1 and da2, you would then see something confusing like this:
2 drives: D da2 State: up /dev/da1s1h A: 3582/4094 MB (87%) D da1 State: up /dev/da1s2h A: 3582/4094 MB (87%)
This is clearly not helpful. Once you have picked names for your spindles, it might be a good idea to label them with a permanent marker. If you have hot-swappable hardware, write the names on the sleds in which the spindles are mounted. This will significantly reduce the likelihood of error when you are moving spindles around later as part of failure recovery or routine system management procedures.
Anyway, back to the configuration. We should tell Vinum that we want a mirrored volume. The keyword here is volume. I named my first volume 'root' and I want it to be accessible at all times, so my third line in vinum.conf will be:
volume root
Now we should add a subdisk to this volume. This is done like this:
plex org concat
sd len 2097152s driveoffset 1048560s drive YouCrazy
So what did we do here? Well, 'plex org concat' tells Vinum that we want to create a plex, and that we want it concatenated (concatenate = to join or merge two or several parts into a single part). 'sd len 2097152s' tells Vinum to create a subdisk (sd) to this plex of the size 2097152 blocks (1024 megabytes). 'driveoffset 1048560s' tells Vinum where the subdisk should start. Note that because of the bootstrap, you must subtract 16 sectors (the length of the bootstrap) from each offset in your bsdlabel in order to get a valid driveoffset for Vinum. Finally, 'drive YouCrazy' tells Vinum which drive the subdisk resides on.
Now that we've got a volume for /, we want to add volumes for our swap, /home, /tmp and /var as well. Basically, we'll just repeat what we did above for the volume root.
volume swap
plex org concat
sd len 1048295s driveoffset 265s drive YouCrazy
volume home
plex org concat
sd len 2097152s driveoffset 3145712s drive YouCrazy
volume tmp
plex org concat
sd len 1048576s driveoffset 5242864s drive YouCrazy
volume var
plex org concat
sd len 6424401s driveoffset 6291440s drive YouCrazy
Some guides also use length instead of len. It doesn't really matter which one you use, as they're the same thing.
Now you should have a file containing something like this:
drive YouCrazy device /dev/ad0s1h
volume root
plex org concat
sd len 2097152s driveoffset 1048560s drive YouCrazy
volume swap
plex org concat
sd len 1048295s driveoffset 265s drive YouCrazy
volume home
plex org concat
sd len 2097152s driveoffset 3145712s drive YouCrazy
volume tmp
plex org concat
sd len 1048576s driveoffset 5242864s drive YouCrazy
volume var
plex org concat
sd len 6424401s driveoffset 6291440s drive YouCrazy
Save the file and exit the editor. It's also a good idea to have a copy of the configuration file on a floppy, CD-ROM or similar media, as it will prove useful if your drive crashes.
4.3. Starting Vinum
Now it's time to start Vinum and see if it likes the config. Simply type vinum. Your prompt should now change from whatever it was before to 'vinum ->', as Vinum has it's own shell. Type list to see the current config. Since this probably is your first time running Vinum, you should get the following output:
vinum -> list 0 drives: 0 volumes: 0 plexes: 0 subdisks:
If you didn't, it means you already have some sort of Vinum configuration. If you are certain you don't use the current Vinum configuration, you can erase it by typing resetconfig. This command can be used anytime you want to clear the current config and start over.
To load our config file, type create -f /etc/vinum.conf. This tells Vinum to create what we specified in /etc/vinum.conf. You should get a output similar to this:
vinum -> create -f /etc/vinum.conf vinum: drive YouCrazy is up viunm: root.p0.s0 is up viunm: root.p0 is up vinum: root is up vinum: swap.p0.s0 is up vinum: swap.p0 is up vinum: swap is up vinum: home.p0.s0 is up vinum: home.p0 is up vinum: home is up vinum: tmp.p0.s0 is up vinum: tmp.p0 is up vinum: tmp is up vinum: var.p0.s0 is up vinum: var.p0 is up vinum: var is up 1 drives: D YouCrazy State: up /dev/ad0s1h A: 0/6208 MB (0%) 5 volumes: V root State: up Plexes: 1 Size: 1024 MB V swap State: up Plexes: 1 Size: 511 MB V home State: up Plexes: 1 Size: 1024 MB V tmp State: up Plexes: 1 Size: 512 MB V var State: up Plexes: 1 Size: 3136 MB 5 plexes: P root.p0 C State: up Subdisks: 1 Size: 1024 MB P swap.p0 C State: up Subdisks: 1 Size: 511 MB P home.p0 C State: up Subdisks: 1 Size: 1024 MB P tmp.p0 C State: up Subdisks: 1 Size: 512 MB P var.p0 C State: up Subdisks: 1 Size: 3136 MB 5 subdisks: S root.p0.s0 State: up D: YouCrazy Size: 1024 MB S swap.p0.s0 State: up D: YouCrazy Size: 511 MB S home.p0.s0 State: up D: YouCrazy Size: 1024 MB S tmp.p0.s0 State: up D: YouCrazy Size: 512 MB S var.p0.s0 State: up D: YouCrazy Size: 3136 MB
Check the 'state' column. It should read 'State: up' all the way. If it says otherwise, something is wrong. For example, you might get output such as this if you've miscalculated something:
16: sd len 6424401s driveoffset 6291440s drive YouCrazy** 16 No space for on YouCrazy: No space left on device
If something is wrong, flush the config using 'resetconfig', check your config file and try again. If you've come this far without errors, press ^D (Ctrl-D) to exit Vinum's shell.
4.4. Filesystem setup
Vinum should have created new devices under /dev/vinum when it parsed the configuration file we created. To check that all is correct, run fsck against the new devices, like this:
# fsck -n -t ufs /dev/vinum/root
The -n argument to fsck tells it not to correct any errors it may find. The last part of the command represents the name of the volume. The output of that command should be something like this:
** /dev/vinum/root (NO WRITE) ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 6769 files, 75162 used, 431325 free (9925 frags, 52675 blocks, 2.0% fragmentation)
If there are any errors, they will probably be because you have miscalculated size or offset. You'll see something like this:
# fsck -n -t ufs /dev/vinum/root
** /dev/vinum/root (NO WRITE)
Cannot find file system superblock
/dev/vinum/root: CANNOT FIGURE OUT FILE SYSTEM PARTITION
Run fsck against /dev/vinum/home, /dev/vinum/tmp and /dev/vinum/var, making
sure they are fine, too. Note that fsck does not need to be run against /dev/vinum/swap, as fsck does not support it.
You need to do this in single-user mode because the volumes are shadowing file systems, and it's normal for open file systems to fail fsck, since some of the state is in buffer cache.
You must now edit /etc/fstab to reflect the new Vinum drives. For example, my
/etc/fstab originally contained this:
# Device Mountpoint FStype Options Dump Pass# /dev/ad0s1b none swap sw 0 0 /dev/ad2s1b none swap sw 0 0 /dev/ad0s1a / ufs rw 1 1 /dev/ad0s1d /home ufs rw 2 2 /dev/ad2s1e /home2 ufs rw 2 2 /dev/ad2s1h /hope ufs rw 2 2 /dev/ad2s1d /rootback ufs rw 2 2 /dev/ad0s1e /tmp ufs rw 2 2 /dev/ad2s1f /tmp2 ufs rw 2 2 /dev/ad0s1f /var ufs rw 2 2 /dev/ad2s1g /var2 ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
After modifications, I had a file such as this:
# Device Mountpoint FStype Options Dump Pass# /dev/vinum/swap none swap sw 0 0 /dev/vinum/root / ufs rw 1 1 /dev/vinum/home /home ufs rw 2 2 /dev/ad2s1g /hope ufs rw 2 2 /dev/vinum/tmp /tmp ufs rw 2 2 /dev/vinum/var /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
Notice also, how I have changed /hope's device from /dev/ad2s1h to /dev/ad2s1g, as we renamed that earlier. Once you've edited the file, save it, exit the editor and reboot to check that all is still okay.
After your computer has rebooted, you could issue a mount command to check that all is still well. The output of the 'mount' command should be similar to this:
/dev/vinum/root on / (ufs, local) devfs on /dev (devfs, local) /dev/vinum/home on /home (ufs, local, soft-updates) /dev/ad2s1g on /hope (ufs, local, soft-updates) /dev/vinum/tmp on /tmp (ufs, local, soft-updates) /dev/vinum/var on /var (ufs, local, soft-updates)
However, we have still not configured ad2s1 ('UpWindow') for mirroring. Let's do that now.
5. Setting up ad2s1
5.1. bsdlabel/disklabel
Boot into single user mode, in case you aren't in it already. Note that you MUST reboot before you start to edit ad2! Mount the root filesystem read-write using the command:
# mount -u /
Again, start bsdlabel, but this time for ad2s1:
# bsdlabel -e ad2s1
The output of the command should be similar to:
# /dev/ad2s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] b: 1048576 0 swap c: 16841601 0 unused 0 0 # "raw" part, don't edit d: 2097152 1048576 4.2BSD 0 0 0 e: 2097152 3145728 4.2BSD 0 0 0 f: 1048576 5242880 4.2BSD 0 0 0 g: 6424401 6291456 4.2BSD 0 0 0 h: 4125744 12715857 4.2BSD 0 0 0
Note how there is no 'a' partition, and that our 'h' partition is currently occupied by /hope. In order for us to be able to boot off ad2 in case ad0 should be damaged/unbootable, we need an 'a' partition. Also, it's good to keep the mirrored spindles as identical as possible, in order to avoid confusion.
First, change 'd' to 'a'. Simply erase the 'd' and replace it with an 'a'. Now that 'd' is available, change 'e' to 'd'. Likewise, change 'f' to 'e', 'g' to 'f' and finally 'h' to 'g'. Now we have the 'h' partition available for our new Vinum partition. Again, shorten the length of the swap partition by 281 sectors, in my case 1048576 - 281 = 1048295. Set it's offset to 281.
Next, we need to calculate the size for the new 'h' partition for Vinum. However, as I do not plan on mirroring the /hope partition (now the 'g' partition on ad2s1), I needn't include it in the size of the Vinum partition. So, in my case, I'll simply create a Vinum partition exactly the same size as on ad0s1, i.e. 12715841 sectors.
After the changes, your ad2s1 partition table should look like this (changes have been bolded):
# /dev/ad2s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] **a**: 2097152 1048576 4.2BSD 0 0 0 b: **1048295 281** swap c: 16841601 0 unused 0 0 # "raw" part, don't edit **d**: 2097152 3145728 4.2BSD 0 0 0 **e**: 1048576 5242880 4.2BSD 0 0 0 **f**: 6424401 6291456 4.2BSD 0 0 0 **g**: 4125744 12715857 4.2BSD 0 0 0 **h: 12715841 16 vinum**
Save the file and exit the editor.
5.2. Vinum configuration for ad2s1
Let's create a new configuration file, for example, /etc/vinum-upwindow.conf. No not under any circumstances edit /etc/vinum.conf, the original configuration file. If you re-run the create command for objects that already exist, strange things will happen (I tried it once, resulting in an endless loop of kernel panics and reboots). Vinum already knows about the objects, and the start command should find them. In this new file, we need the following contents:
drive UpWindow device /dev/ad2s1h
volume root
plex org concat
sd len 2097152s driveoffset 1048560s drive UpWindow
volume swap
plex org concat
sd len 1048295s driveoffset 265s drive UpWindow
volume home
plex org concat
sd len 2097152s driveoffset 3145712s drive UpWindow
volume tmp
plex org concat
sd len 1048576s driveoffset 5242864s drive UpWindow
volume var
plex org concat
sd len 6424401s driveoffset 6291440s drive UpWindow
Notice now I have simply used the original configuration file, and changed the device from /dev/ad0s1 to /dev/ad2s1, and the name 'YouCrazy' to 'UpWindow'. No other changes are necessary, as the drives are identical.
Save the file and exit the editor. It's also a good idea to have a copy of the configuration file on a floppy, CD-ROM or similar media, as it will prove useful if your drive crashes.
5.3. Starting Vinum
Now its time to start Vinum again and see if it likes the new config. Simply type vinum. Your prompt should now change from whatever it was before to 'vinum ->', as Vinum has it's own shell.
To load our config file, type create -f /etc/vinum-upwindow.conf. This tells Vinum to create what we specified in /etc/vinum-upwindow.conf. You should get a output similar to this:
vinum -> create -f /etc/vinum-upwindow.conf vinum: drive UpWindow is up vinum: root.p1 is faulty vinum: swap.p1 is faulty vinum: home.p1 is faulty vinum: tmp.p1 is faulty vinum: var.p1 is faulty 2 drives: D YouCrazy State: up /dev/ad0s1h A: 0/6208 MB (0%) D UpWindow State: up /dev/ad2s1h A: 0/6208 MB (0%) 10 volumes: V root State: up Plexes: 2 Size: 1024 MB V swap State: up Plexes: 2 Size: 511 MB V home State: up Plexes: 2 Size: 1024 MB V tmp State: up Plexes: 2 Size: 512 MB V var State: up Plexes: 2 Size: 3136 MB 10 plexes: P root.p0 C State: up Subdisks: 1 Size: 1024 MB P swap.p0 C State: up Subdisks: 1 Size: 511 MB P home.p0 C State: up Subdisks: 1 Size: 1024 MB P tmp.p0 C State: up Subdisks: 1 Size: 512 MB P var.p0 C State: up Subdisks: 1 Size: 3136 MB P root.p1 C State: faulty Subdisks: 1 Size: 1024 MB P swap.p1 C State: faulty Subdisks: 1 Size: 511 MB P home.p1 C State: faulty Subdisks: 1 Size: 1024 MB P tmp.p1 C State: faulty Subdisks: 1 Size: 512 MB P var.p1 C State: faulty Subdisks: 1 Size: 3136 MB 10 subdisks: S root.p0.s0 State: up D: YouCrazy Size: 1024 MB S swap.p0.s0 State: up D: YouCrazy Size: 511 MB S home.p0.s0 State: up D: YouCrazy Size: 1024 MB S tmp.p0.s0 State: up D: YouCrazy Size: 512 MB S var.p0.s0 State: up D: YouCrazy Size: 3136 MB S root.p1.s0 State: stale D: UpWindow Size: 1024 MB S swap.p1.s0 State: stale D: UpWindow Size: 511 MB S home.p1.s0 State: stale D: UpWindow Size: 1024 MB S tmp.p1.s0 State: stale D: UpWindow Size: 512 MB S var.p1.s0 State: stale D: UpWindow Size: 3136 MB
Don't worry about this, that's normal! You see, we only have data on one spindle so far (ad0, 'YouCrazy'). We need to copy over the data to spindle number two (ad2, 'UpWindow'). Luckily, Vinum has a command for this, called start. Simply start all of the plexes that are marked 'stale'. For example, I would execute start root.p1.s0 in order to start the copying (Vinum calls this 'reviving') of data for my 'root' volume. The output of the command should be similar to this:
vinum -> start root.p1.s0 Reviving root.p1.s0 in the background vinum -> vinum[57]: reviving root.p1.s0 Nov 21 15:14:39 vinum[57]: reviving root.p1.s0
If you issue a list command while the reviving process is running, you should see something similar to this:
S root.p1.s0 State: R 31% D: UpWindow Size: 1024 MB
This tells us, that the reviving process has revived 31% of the subdisk so far. Once the reviving process is done, the 'State' column of the subdisk should change to up, and you should get the following message in your console:
vinum: root.p1.s0 is up by force vinum: root.p1 is up vinum: root.p1.s0 is up vinum[57]: root.p1.s0 is up Nov 21 15:26:15 vinum[57]: root.p1.s0 is up
You can start several reviving processes at once, but this will have a dramatic impact on the disk performance for as long as the process continues.
Reboot again (to multi-user, if you wish) to check that all is fine. Start vinum again, and issue a list command. The command should list 2 drives, 10 volumes, 10 plexes and 10 subdisks, all with the state up. In my case, the output looked like this:
2 drives: D UpWindow State: up /dev/ad2s1h A: 0/6208 MB (0%) D YouCrazy State: up /dev/ad0s1h A: 0/6208 MB (0%) 5 volumes: V root State: up Plexes: 2 Size: 1024 MB V swap State: up Plexes: 2 Size: 511 MB V home State: up Plexes: 2 Size: 1024 MB V tmp State: up Plexes: 2 Size: 512 MB V var State: up Plexes: 2 Size: 3136 MB 10 plexes: P root.p0 C State: up Subdisks: 1 Size: 1024 MB P swap.p0 C State: up Subdisks: 1 Size: 511 MB P home.p0 C State: up Subdisks: 1 Size: 1024 MB P tmp.p0 C State: up Subdisks: 1 Size: 512 MB P var.p0 C State: up Subdisks: 1 Size: 3136 MB P root.p1 C State: up Subdisks: 1 Size: 1024 MB P swap.p1 C State: up Subdisks: 1 Size: 511 MB P home.p1 C State: up Subdisks: 1 Size: 1024 MB P tmp.p1 C State: up Subdisks: 1 Size: 512 MB P var.p1 C State: up Subdisks: 1 Size: 3136 MB 10 subdisks: S root.p0.s0 State: up D: YouCrazy Size: 1024 MB S swap.p0.s0 State: up D: YouCrazy Size: 511 MB S home.p0.s0 State: up D: YouCrazy Size: 1024 MB S tmp.p0.s0 State: up D: YouCrazy Size: 512 MB S var.p0.s0 State: up D: YouCrazy Size: 3136 MB S root.p1.s0 State: up D: UpWindow Size: 1024 MB S swap.p1.s0 State: up D: UpWindow Size: 511 MB S home.p1.s0 State: up D: UpWindow Size: 1024 MB S tmp.p1.s0 State: up D: UpWindow Size: 512 MB S var.p1.s0 State: up D: UpWindow Size: 3136 MB
If all is good, feel free to delete the dummy mount points:
# rm -r /home2 /tmp2 /var2 /rootback
Congratulations, you are now the proud owner of a RAID-1 setup!
6. Failure scenarios
This chapter is based in large extent on Bootstrapping Vinum by Robert A. Van Valzah and Replacing a failed Vinum drive.
This section contains descriptions of various failure scenarios. For each scenario, there is a subsection on how to configure your server for degraded mode operation, how to recover from the failure, how to exit degraded mode, and how to simulate the failure.
6.1. Drive ad2 fails
This section deals with the total failure of /dev/ad2. We assume here that your server is up and running multi-user in degraded mode on just /dev/ad0 and that you have a new spindle standing by.
Simulation: You can simulate a total ad2 failure by wiping out the partition table for the drive. I've also found a small DOS program called hardcure, which wipes the 0-track of the hard drive, quite effective.
Step 1: Assuming, that the hardware needs to be replaced, unplug ad2 ('UpWindow'). Now, aren't you glad you created the drives with some labels earlier, didn't you? ;-) In case you aren't running hot-swap hardware, you need to do a shutdown, install the new drive on ad2 and then boot. If you have hot-plugged the drive, it must have the same ID, the Vinum drive must be on the same partition, and it must have the same size. If you have had to stop the system to replace the drive, the old drive will not be associated with a device name, and you can put it anywhere. If you want to have this freedom with a hot-pluggable drive, you must stop Vinum and restart it.
Step 2: Use BootMgr to load the kernel from /dev/ad0s1a, just as you normally would.
Step 3: Boot into single-user mode by selecting number 4 from the list. Note that Vinum will produce some error messages when booting, similar to 'vinum: root.p1.s0 is crashed', 'vinum: root.p1 is faulty' and 'vinum: incompatible sector sizes. root.p1 has 0, root 512. Ignored.' This is normal in this situation.
Step 4: Now you need to create a BIOS partition (slice) on the new spindle.
- Launch
/stand/sysinstallas root. - Select 'Custom'.
- Select 'Partition'.
- Select
ad2 - Create a FreeBSD (type 165) slice large enough to hold everything mentioned above.
- Write the changes.
- Yes, you are absolutely sure.
- Select BootMgr.
- Quit Partitioning.
-Exit
/stand/sysinstall
Step 5: Create disk label partitioning based on current /dev/ad0s1 partitioning. Execute the following commands:
# mount -u /
# bsdlabel ad0s1 > /ad0s1
# bsdlabel -e ad2s1
This will drop you into your favorite editor.
- Copy the lines for the partitions you want mirrores from
/ad0s1to thead2s1disklabel, including the 'h' a.k.a. the Vinum partition. - Create any additional partitions with what space you might have left on the new drive.
- Save the file and exit the editor.
Step 5: Create a new configuration file for Vinum, for example, /etc/new-vinum.conf. Inside it, place a single line containing the following:
drive UpWindow device /dev/ad2s1h
Save the file and exit the editor.
Step 6: Load the new configuration file into Vinum:
# vinum
vinum -> create -f /etc/new-vinum.conf
The output should be similar to this:
vinum: drive UpWindow is up vinum: swap.p1.s0 is reborn vinum: swap.p1 is flaky vinum: home.p1.s0 is reborn vinum: home.p1 is flaky vinum: var.p1.s0 is reborn vinum: var.p1 is flaky vinum: swap.p1 is faulty vinum: home.p1 is faulty vinum: var.p1 is faulty 2 drives: D YouCrazy State: up /dev/ad0s1h A: 0/6208 MB (0%) D UpWindow State: up /dev/ad2s1h A: 0/6208 MB (0%) 5 volumes: V root State: up Plexes: 2 Size: 1024 MB V swap State: up Plexes: 2 Size: 511 MB V home State: up Plexes: 2 Size: 1024 MB V tmp State: up Plexes: 2 Size: 512 MB V var State: up Plexes: 2 Size: 3136 MB 10 plexes: P root.p0 C State: up Subdisks: 1 Size: 1024 MB P swap.p0 C State: up Subdisks: 1 Size: 511 MB P home.p0 C State: up Subdisks: 1 Size: 1024 MB P tmp.p0 C State: up Subdisks: 1 Size: 512 MB P var.p0 C State: up Subdisks: 1 Size: 3136 MB P root.p1 C State: faulty Subdisks: 1 Size: 1024 MB P swap.p1 C State: faulty Subdisks: 1 Size: 511 MB P home.p1 C State: faulty Subdisks: 1 Size: 1024 MB P tmp.p1 C State: faulty Subdisks: 1 Size: 512 MB P var.p1 C State: faulty Subdisks: 1 Size: 3136 MB 10 subdisks: S root.p0.s0 State: up D: YouCrazy Size: 1024 MB S swap.p0.s0 State: up D: YouCrazy Size: 511 MB S home.p0.s0 State: up D: YouCrazy Size: 1024 MB S tmp.p0.s0 State: up D: YouCrazy Size: 512 MB S var.p0.s0 State: up D: YouCrazy Size: 3136 MB S root.p1.s0 State: stale D: UpWindow Size: 1024 MB S swap.p1.s0 State: stale D: UpWindow Size: 511 MB S home.p1.s0 State: stale D: UpWindow Size: 1024 MB S tmp.p1.s0 State: stale D: UpWindow Size: 512 MB S var.p1.s0 State: stale D: UpWindow Size: 3136 MB
Step 7: Start the plexes that are down. The -w argument to start instructs Vinum to wait for the process to finish before returning the user to the prompt.
vinum -> start -w root.p1.s0
vinum -> start -w swap.p1.s0
vinum -> start -w home.p1.s0
vinum -> start -w tmp.p1.s0
vinum -> start -w var.p1.s0
Your Vinum configuration has now recovered. It is now safe to remove the temporary /ad0s1 file. Exit/reboot to multi-user mode.
6.2. Drive ad0 fails
This section deals with the total failure of /dev/ad0. We assume here that your server is up and running multi-user in degraded mode on just /dev/ad2 and that you have a new spindle standing by.
Simulation: You can simulate a total ad0 failure by wiping out the partition table for the drive. I've also found a small DOS program called hardcure, which wipes the 0-track of the hard drive, quite effective.
Step 1: Assuming, that the hardware needs to be replaced, unplug ad0 ('YouCrazy'). Now, aren't you glad you created the drives with some labels earlier, didn't you? ;-) In case you aren't running hot-swap hardware, you need to do a shutdown, install the new drive on ad0 and then boot. If you have hot-plugged the drive, it must have the same ID, the Vinum drive must be on the same partition, and it must have the same size. If you have had to stop the system to replace the drive, the old drive will not be associated with a device name, and you can put it anywhere. If you want to have this freedom with a hot-pluggable drive, you must stop Vinum and restart it.
Step 2: Use BootMgr to load the kernel from /dev/ad2s1a. This is done by pressing F5 to select 'Drive 1' while the BootMgr is awaiting for keypresses. You will now be presented with another menu, with two choices, 'F1 FreeBSD' and 'F5 Drive 0'. Press F1 to select 'FreeBSD'. FreeBSD should now boot from ad2s1a. After that, do steps 3-7 as described in the previous section 6.1. Just mentally replace ad0 with ad2 and vice versa.