The following error is encountered after a successful mirroring of a previously failed, now replaced rootmirror disk:

VxVM vxbootsetup ERROR V-5-2-3599 No free partition found on device c0t1d0s2 for oracle_vol

The following are the sequence of events:
1.)  rootmirror disk fails due to hardware errors

2.)   Customer decides to perform a physical replacement of the failed rootmirrordisk
        i.)     vxdiskadm option '4 -Remove a disk for replacement'
       ii.)    Remove the disk from the OS
       iii.)   Replace the disk physically with a new LUN

3.)   After physical replacement and the OS identifying the new device correctly we need to add the disk back into the Volume Manager Configuration:
       i.)     Ensure a good OSlabel has been put on the newdisk
       ii.)    vxdiskadm option '5 -Replace a failed or removed disk'

Once the "vxdiskadm option 5" is initiated, the mirroring starts for all the volumes in the rootdg (bootdg) diskgroup and completes the mirroring process successfully but exits with an Error message:
--------------------------------------------------------------------------------------------------------------------------------
UseFMR for plex re sync? [y,n,q,?] (default: n)
VxVM vxbootsetup ERRORV-5-2-3599 No free partition found on device c0t1d0s2 for oracle_vol.
VxVM  INFO V-5-2-282 Replacement of disk rootmirror ingroup rootdg with disk device c0t1d0 completed successfully.
--------------------------------------------------------------------------------------------------------------------------------

The message encountered above explains that the actual mirroring process was successful but it was not possible to create the underlying native Solaris Operating System slice for the corresponding Volume oracle_vol.

The vxdiskadm option 5 will identify that the replacement of the disk is for a bootdisk. Hence after adding the newly replaced disk back into the boot disk group as the rootmirror disk, the re-sync of all the plexes will be initiated on the rootmirror disk which will produce a disk that can be used as an alternate bootdisk. After the re-sync of the rootmirror disk, this process of vxdiskadm(option 5) will also include the execution of the "vxbootsetup" utility, which configures a boot disk by writing a boot track at the beginning of the disk and by creating physical disk partitions in the UNIX VTOC that match the mirrors of the rootdisk volumes. The "vxbootsetup" will also include non-system volumes by default.

In our case, this part of the vxbootsetup is exactly what fails causing the above errors due to a limitation imposed by the Solaris Operating System, wherein we have a limit of the number of underlying slices that any one physical disk can hold.

A Solaris disk can have a maximum of 8 slices from 0 to 7. Slice 2 is reserved by the Operating System as the backup slice referencing the entire disk. A further two free slices will be required for Volume Manager Private and Public region slices for a sliced format disk, thereby only leaving 5 other free slices for the remaining corresponding Volumes.

If there are more than 5 Volumes including OS-related system volumes/file systems for rootvol, swapvol, var, usr, home, opt or any other nonOS-related data volumes/file systems, then any volumes in excess of total 5Volumes in the rootdg (bootdg) disk group will not be able to have an underlying native operating slice on the boot device.

What would be the impact of not having an underlying slice referencing a corresponding volume in the encapsulated boot disk group?

The VxVM Volumes are a logical representation of data blocks on a physical disk. In the case of an encapsulated boot disk, each Volume references an underlying operating system-based native slice that refers to specific blocks of data. During the encapsulation process, the Volume Manager creates individual logical volumes for each of the existing native slices on the Solaris Disk that reference the same underlying blocks. There may be some additional free space and free slices that may not have been used and are available for future use. However, there is no such limit for the number of Logical Volumes created by Veritas Volume Manager on a disk as long as it has available free block space on the disk. Hence customers may be able to create more logical Volumes and file systems on those volumes, such that the configuration would end up having more Logical Volumes than the total maximum number of 8 underlying native slices that are possible on any one Solaris disk. Under normal circumstances and regular system functionality, there would be no operational impact to access the data on any of the Volumes. However, the problems may arise under a situation where we are compelled to unencapsulated the boot disk so as to enable to boot from native OS slices instead of the encapsulated volume manager disk from volumes. This is where, after the unencapsulation, there will be no guaranteed way of recovering the data on the volumes that do not have a reference to an underlying OS native slice.

There could be certain situations where an unencapsulation may be necessary or required:

1.)   A planned outage to upgrade the StorageFoundation and/or the Operating System, or
2.)   An unforeseen situation that renders the Operating System unbootable from Volume Manager encapsulated Volumes (The only option available then is to manually unencapsulate the boot disk to enable booting up of the operating system from native slices instead of encapsulated volumes), or
3.)   Any other conditions that may require the unencapsulation of the boot disk

Veritas does not support or guarantee any recovery of such data volumes in an encapsulated operating system boot disk group, which may not have reference to a corresponding underlying operating system native slice, for the reasons explained above. Hence as per best recommended practices, customers are recommended that the Operating System boot disk should not include any non-operating system-related data volumes. Instead, we should use distinct disk/s in a separate data disk group to contain any non-operating system-related volumes/file systems.