Adding LUNs to Xsan Volumes fails

Under certain conditions, Xsan volumes may not expand when you introduce new LUNs to an existing storage pool. Here's some background information and how to deal with the issue.

Bandwidth Expansion versus Storage Expansion

Adding LUNs to an existing storage pool will increase both the capacity of a volume as well as the amount of data you can write to a volume per unit of time, so adding LUNs to an existing storage pool known as "bandwidth expansion." (Note that this is true within reasonable limits. If you attempt to make a volume arbitrarily large, you will at some point encounter a diminishing return, and ultimately performance issues as you approach the limits of your equipment.)

Alternately, adding LUNs to a new storage pool will increase the capacity of a volume, but it will not increase bandwidth. Therefore, adding LUNs to an new storage pool is referred to as "storage expansion."

This article pertains to bandwidth expansion.

Issue description

Xsan 1.1 and earlier: After adding LUNs to an existing storage pool, Xsan volumes will no longer mount at Xsan clients.

Xsan 1.2 and later: After adding LUNs to an existing storage pool, changes made to the volume configuration will revert.

Solution

In each of the following steps, replace the phrase <VolumeName> with the name of your Xsan Volume, and replace the phrase <PoolName> with the name of the storage pool you want to expand. Also, after using the command sudo, authenticate if prompted.

Preparation:

If using multiple Metadata Controllers:

Determine which metadata controller (MDC) is controlling the volume by visiting the <VolumeName> Overview in Xsan Admin. Note which MDC is hosting the volume by referencing the "Hosted By" value. Perform the following steps on the computer that is hosting the volume.
If the MDC is running Xsan 1.1 or earlier, ensure that the Xsan volume will start up and mount. If the volume will not start or mount, then stop here and contact AppleCare for support. If the volume will start and mount, make a backup of the config file by issuing the following command in the Terminal:
```
$ sudo cp /Library/Filesystems/Xsan/config/<VolumeName>.cfg /Library/Filesystems/Xsan/config/<VolumeName>.cfg.bak
```
Assess your site's needs. If you only wish to increase the volume's capacity, just add the LUNs to a new storage pool. If your site requires an increase in a volume's bandwidth, you will need to take the following steps:
1. Unmount and stop the volume.
2. Verify that there are no inconsistencies in the filesystem by following the steps described here.
3. In Xsan Admin > Setup > LUNs, label the new LUNs.
4. In Xsan Admin > Setup > Storage, drag the new LUNs from the "Available LUNs" drawer to the preexisting storage pool.
5. Click Save.
6. If the expansion failed, proceed to step 4.
Verify that the issue is occurring by reading the cvupdatefs log located in /var/run. To do so, type the following in the Terminal application located in /Applications/Utilities:
```
$ sudo cat /var/run/xsan_cvupdatefs_<VolumeName>.log
```
(Note that rebooting your MDC will result in this file being deleted.)

If an Xsan volume is affected by this issue, you will see the following entry at the end of the file:
```
<snip>

Bitmap fragmentation threshold exceeded. Aborting. Invalid argument

*Fatal*: Failed to expand stripe group

Check configuration and try again

<snip>
```
What this means:

The addition of the LUNs to a pre-existing storage pool would have fragmented the storage pool so that performance would have been unacceptably slow.

Preparation for Workaround

If your MDC is running Xsan 1.2 and you're affected by the issue, skip to step 3.
If the MDC is running Xsan 1.1 of earlier, restore the backup of the volume config file by issuing the following commands in the Terminal:
```
$ sudo rm /Library/Filesystems/Xsan/config/<VolumeName>.cfg
$ sudo cp /Library/Filesystems/Xsan/config/<VolumeName>.cfg.bak /Library/Filesystems/Xsan/config/<VolumeName>.cfg
```
Verify that the volume will start and mount. If it will not, stop here and contact AppleCare Support. If it will start and mount, continue to step 3.
Run cvfsck again to verify that there are no inconsistencies in the filesystem by following the steps described here.

Workaround

All data that resides on the storage pool will need to be temporarily backed-up to another volume in order to correct this issue.
1. Start and mount the Xsan volume only on the Metadata controller.
2. In the Terminal, type:
```
sudo cvfsck -x <VolumeName> | awk -F, '{print $5","$6}' | egrep -v "^none,"
```
  This command should produce a list of all folders at the root of the Xsan volume which have one or more files located on the Storage Pool that is listed before the folder name. For instance, in the following example the folder Affinity1 has one of more of it's files on the Storage Pool called "MyPool."
```
MyPool,Affinity1
```
  A Note about "cvfsck -x": This command only returns the first eight characters of a Storage Pool's name. For example, a storage pool with the name "VideoPool" would be changed to "VideoPoo."
  
  If you have two or more pools with names containing the same first 8 characters, cvfsck will change the name of the first of these Storage Pools listed in Xsan Admin > Setup > Storage to eight characters. For example, a pool named DataPoolMoe would be converted to "DataPool."
  
  Next, cvfsck will convert each subsequent pool encountered in the config settings located in Xsan Admin with the same first 8 characters to have a trailing numeric character. For instance, if there were two other storage Pools named DataPoolLarry and DataPoolCurly, cvfsck would convert these to DataPoo1 and DataPoo2. To determine which converted name corresponds to which pool, visit Xsan Admin > Setup > Storage and note in which order these storage pools are listed from top to bottom and correspond these to the numeric values.
After you have determined where you will temporarily back up the data, mount the temporary backup volume on the MDC.
Run the following command to back up each folder that resides on the storage pool you wish to expand:
```
$ sudo tar -czpf /Volumes/<TempVol>/<FolderName>.tgz -C/Volumes/<VolumeName>/ <FolderName>
```
...replacing <TempVol> with the name of the volume to which the data will be temporarily backed up, <FolderName> with the name of the folder which is to be backed up, and <VolumeName> with the name of the Xsan volume.
Verify that all files that reside on the storage pool have been backed up.
Verify that all backed up folders can be extracted and that the contents are identical to the originals.
Delete the folders that have been backed up from the Xsan volume.
Add the new LUN(s) to the existing storage pool as directed above.
Verify that the LUNs have successfully been incorporated into the storage pool.
Run cvfsck again to verify that there are no inconsistencies in the filesystem by following the steps described here.
Start and mount the Xsan volume.
Restore the backed-up files from the temporary backup location to the newly expanded filesystem:
```
$ sudo tar -xzf /Volumes/<TempVol>/<FolderName>.tgz-C /Volumes/<VolumeName>/
```
Verify that all restored folders and their contents are identical to the originals.
Delete the tar files from the temporary location.