One of the challenges that comes with managing Xen domU instances is dealing with virtual machines that run out of disk space. If the initial setup was not done carefully, you may find that resizing the virtual machine is a bit more involved than it sounds, especially if the domU is backed by a disk image or single physical partition.
In this post, I’ll demonstrate two different methods for adding more disk space to a Xen virtual machine. Unless otherwise indicated, all systems described here are running fully up-to-date versions of CentOS 5.6 (Final), and all Xen domU instances are using file based disk images.
When a file-backed virtual machine runs out of space, one way to gain additional space is to add a second file-backed virtual disk. Note that you could also add a physical disk, and the steps would largely be the same, but that is left as an exercise for the reader.
Create a new virtual disk
First, create a sparse file to use as the new disk image. In this case, we will create a new 10G image file. Note that creating a sparse file in this way will be very fast, but it will only actually use 1M of space on disk (as reported by
df), so keep in mind that the actual usage can increase up to the total size of the file you are creating (here, 10G).
# cd /var/lib/xen/images # dd if=/dev/zero of=new_disk.img oflag=direct bs=1M seek=10239 count=1
Add the Disk to the Virtual Machine
Once you have the new disk image created, you will need to add it to the virtual machine. First, modify the
disk entry of the config file so that the disk is recognized when the virtual machine is restarted. In this case, the second entry is the one being added:
disk = [ "tap:aio:/var/lib/xen/images/original.img,xvda,w", "tap:aio:/var/lib/xen/images/new_disk.img,xvdb,w" ]
Note: Be sure that you use a device name that is not already being used by the domain. In this case, only
xvda was in use so using
xvdb for the new disk device name is fine.
At this point, you can restart the virtual machine to have the disk recognized, but you can also dynamically add the new storage to the domU without restarting by using the
xm utility and the
block-attach sub-command (substitute the name of your domain in the following command):
# xm block-attach MyDomain tap:aio:/var/lib/xen/images/new_disk.img xvdb w
You can use the
block-list sub-command to confirm that the device was added to the running virtual machine; prior to running
xm block-attach, only one device would have been listed here:
# xm block-list MyDomain Vdev BE handle state evt-ch ring-ref BE-path 51712 0 0 4 6 8 /local/domain/0/backend/tap/3/51712 51728 0 0 4 8 12 /local/domain/0/backend/tap/3/51728
Once the domU knows about the new disk, you can configure it for use. Log into the virtual machine and use
fdisk to create a new partition table. Here we create a single partition using the default type of
# fdisk /dev/xvdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. The number of cylinders for this disk is set to 1305. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1305, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305): Using default value 1305 Command (m for help): p Disk /dev/xvdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/xvdb1 1 1305 10482381 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. #
Once the partition table is initialized, there are two paths you can take to add the new space to the virtual machine:
In general, my preference is to add the space to an LVM group, but there are valid reasons for both methods so each is be presented here for completeness.
Mount the new partition traditionally
This is the best option if you need to create isolated space for a specific purpose, or if you do not have LVM available to you on the virtual machine. In either case, the first step is to create a filesystem on the new partition. Here, the standard ext3 format is used:
# mkfs.ext3 /dev/xvdb1
Next, a directory must be created to serve as the mount point for the new partition. I like to create mount points with permissions
000 to prevent data from accidentally being written to the directories when the partition is not mounted; any files living in the mount point will be hidden when the partition gets mounted, and making it non-writable, is a good way to help prevent that.
# mkdir -m 000 /extraspace
Create an entry for the new mount point in
/etc/fstab (so that it is mounted on boot) and mount the partition. Make changes the the following mount options as needed for your system:
# echo "/dev/xvdb1 /extraspace ext3 defaults 1 3" >> /etc/fstab # mount /extraspace
df to confirm that the new partition is mounted and available for use:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 45G 40G 2.4G 95% / /dev/xvda1 99M 22M 73M 23% /boot tmpfs 1.0G 0 1.0G 0% /dev/shm /dev/xvdb1 9.9G 151M 9.2G 2% /extraspace
Incorporate the new space into an LVM group
As an alternative to mounting the new disk in the filesystem, you can also add it as a physical volume to an existing LVM volume group. This is the best option if you need to increase the size of an existing disk transparently without disrupting existing operations.
If you choose to add the disk this way, start by following the procedure for creating the new partition table using
fdisk as described previously, but be sure to use the
t command to change the partition type to
8e (Linux LVM). The additional output of fdisk for this portion of the process will resemble the following:
Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): p Disk /dev/xvdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/xvdb1 1 1305 10482381 8e Linux LVM Command (m for help):
Once the partition has been initialized, use the LVM tools to create a new physical volume with the
xvdb1 partition that was just created. Then add the physical volume to the volumegroup of your choice, and extend your logical volume to use all available space.
In this example, I am using volume group
VolGroup00 and logical volume
LogVol00; these are the defaults for a standard CentOS LVM configuration.
# pvcreate /dev/xvdb1 Physical volume "/dev/xvdb1" successfully created # vgextend VolGroup00 /dev/xvdb1 Volume group "VolGroup00" successfully extended # lvextend -l+100%FREE -r VolGroup00/LogVol00 Extending logical volume LogVol00 to 55.88 GB Logical volume LogVol00 successfully resized resize2fs 1.39 (29-May-2006) Filesystem at /dev/mapper/VolGroup00-LogVol00 is mounted on /; on-line resizing required Performing an on-line resize of /dev/mapper/VolGroup00-LogVol00 to 14647296 (4k) blocks. The filesystem on /dev/mapper/VolGroup00-LogVol00 is now 14647296 blocks long.
Note: LVM will automatically re-size the filesystem on the logical volume, which is nice.
Finally, compare the file system disk space usage with the output of the previous
df command to confirm that the additional space is indeed available:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 55G 40G 12G 78% / /dev/xvda1 99M 22M 73M 23% /boot tmpfs 1.0G 0 1.0G 0% /dev/shm
In the end, adding additional space to a Xen virtual machine is not that difficult. Please let me know if you have found this tutorial helpful, or if you have had similar experiences modifying disks on a Xen domU.
Stay tuned for my next article on migrating a Xen virtual machine from a file based container to an LVM-backed block device.