Removing a VMFS extent
Until vSphere 4, the only way to add space to an existing VMFS was to add an extent. This means creating a new partition, most of the time on a new LUN, and extend the VMFS there (vSphere 4 is now able to resize a partition on a grown LUN). This is somehow equivalent to adding a physical volume in a volume group under LVM. But contrary to LVM, once you added an extent to a VMFS, it is impossible to remove it.
Well, until now, it was.
I just pushed my lvm branch of vmfs-tools (get a git snapshot tarball), which includes a new tool named vmfs-lvm, allowing to just do that. For the moment, the tool is not cluster-safe, which means you'd better run it on an offline VMFS (i.e. make sure no server is using it). Data should not be at risk because the tool checks the removed extent doesn't contain any data, but it also assumes the filesystem is in a consistent state beforehand.
The command line to remove an extent looks like the following:
# vmfs-lvm extent0 extent1 ... extentn remove
This will remove the last extent.
Update: There was a bug when setting some values at volume level. The git snapshot link above has been updated accordingly.
2009-12-02 16:40:55+0900
You can leave a response, or trackback from your own site.
2011-04-12 10:38:09+0900
Oh, I was really looking for something like this. But…a little help, please?
# ls -lhgo /vmfs/devices/disks/mpx.vmhba0:C0:T*:L0*
-rw——- 1 69G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0
-rw——- 1 1.1G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:1
-rw——- 1 110M Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:2
-rw——- 1 68G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:3
-rw——- 1 68G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:5
-rw——- 1 69G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T1:L0
-rw——- 1 69G Apr 12 01:32 /vmfs/devices/disks/mpx.vmhba0:C0:T1:L0:1
I’m trying to remove mpx.vmhba0:C0:T1:L0:1
# ./vmfs-lvm /vmfs/devices/disks/mpx.vmhba0:C0:T1:L0:1 remove
VMFS: R/W support is experimental. Use at your own risk
VMFS: Missing extents
Unable to open filesystem
Or, according to your example above:
# ./vmfs-lvm /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:[1235] /vmfs/devices/disks/mpx.vmhba0:C0:T1:L0:1 remove
VMFS: R/W support is experimental. Use at your own risk
VMFS VolInfo: invalid magic number 0x00000000
VMFS: Unable to read volume information
Trying to find partitions
Unable to open device/file “/vmfs/devices/disks/mpx.vmhba0:C0:T0:L0:1”.
Unable to open filesystem
Any ideas?
Thanks,
C.
2011-04-12 15:20:46+0900
Seeing the sizes of your partitions, you probably want to use C0:T0:L0:3 and C0:T0:L0:5 only. I’d advise checking that debugvmfs works with the given list of extents.
Please make sure no server is using the VMFS.
2011-04-16 03:21:14+0900
Hm, I still don’t fully understand. But maybe I have a wrong assumption of what vmfs-lvm will do. I have sda & sdb, ESX is installed on sda (and a 7GB sdc, dunno where that comes from)
and sdb has only one partition (VMware VMFS). I added sdb to the datastore, now I want to remove it again – with vmfs-lvm.
C0:T0:L0:3 and C0:T0:L0:5 is sda3 & sda5, and sda3 is just an “extended partition”, holding only sda5:
Device Start End Blocks Id System
/dev/sda1 1 140 1124518+ 83 Linux
/dev/sda2 141 154 112455 fc VMware VMKCORE
/dev/sda3 155 8924 70445025 5 Extended
/dev/sda5 155 8924 70444993+ fb VMware VMFS
So, I don’t really want to remove C0:T0:L0:3 or C0:T0:L0:5 but sdc1 aka C0:T1:L0:1.
debugvmfs can read C0:T0:L0:5 (and even C0:T0:L0:3), but not C0:T1:L0:1 (VMFS: Unable to read FS information).
So, I guess I’m out of luck here.
Thanks for replying!
C.
2011-04-16 11:57:36+0900
so, you need to pass C0:T0:L0:5 and C0:T1:L0:1 to debugvmfs to verify, and to vmfs-lvm once you’re sure they are the ones.
2011-04-21 19:45:15+0900
Well, “debugvmfs /vmfs/devices/disks/mpx.vmhba0\:C0\:T0\:L0:5 ls” gives me a file list and indeed, it’s the VMFS I want to keep.
However, “C0:T1:L0:1” gives me:
VMFS: Unable to read FS information
and “C0:T1:L0” says:
VMFS VolInfo: invalid magic number 0x00000000
VMFS: Unable to read volume information
And vmfs-lvm won’t accept it as well:
—————————-
# vmfs-lvm /vmfs/devices/disks/mpx.vmhba0\:C0\:T1\:L0\:1 remove
VMFS: R/W support is experimental. Use at your own risk
VMFS: Missing extents
Unable to open filesystem
# vmfs-lvm /vmfs/devices/disks/mpx.vmhba0\:C0\:T1\:L0 remove
VMFS: R/W support is experimental. Use at your own risk
VMFS VolInfo: invalid magic number 0x00000000
VMFS: Unable to read volume information
Trying to find partitions
VMFS: Missing extents
Unable to open filesystem
—————————-
Thanks for your replies, glandium. But please, don’t let me bother you any further. I appreciate your efforts but I don’t want to turn this into a support session. Unfortunately I cannot grant you SSH access to the system so that you could play around a bit…
Thanks again,
C.
2011-04-22 10:25:57+0900
You need to pass *both* disks to debugvmfs and vmfs-lvm.
2011-04-24 02:38:51+0900
Ah, right. I’ve even done this before (see above), but specified C0:T0:L0:[1235], which was wrong perhaps.
I’ve migrated all the VMs (including their data) to another ESX, so the datastore should be empty and only the esxconsole* data is on this datastore.
$ vmfs-lvm ../mpx.vmhba0\:C0\:T0\:L0\:5 ../mpx.vmhba0\:C0\:T1\:L0\:1 remove
VMFS: R/W support is experimental. Use at your own risk
Extents removal is experimental ! Use at your own risk !
…but “debugvmfs … show_volume” still shows:
– Size : 135.25 GB
– Blocks : 543
– Num. Extents : 2
and list’s both disks. “fsck.vmfs” reports 1512 “lost blocks”, so that might be why vmfs-lvm won’t touch it before these lost blocks are cleaned up. I’ll try run this again in single user mode…
2011-07-27 12:01:49+0900
I compiled vmfs-lvm and debugvmfs to run on ESXi 4.1, and used vmfs-lvm to remove an extent that I mistakenly added to a datastore. There did not appear to be any errors, and debugvmfs no longer lists the removed device in the “physical drive” entries of “show_volume”, and furthermore, it reports the size of the volume as that from before the addition of the errant extent.
Unfortunately, ESXi’s GUI interface does not show any change in the size of the volume after removing the extent, but it did, however, remove the extent from the “Properties” dialog for the datastore in the “Storage” section of the “Summary” tab for the host.
Did I miss a step here? Should I be doing something to get the datastore to shrink back down to its original size after removing the extent?
Thanks so much for your hard work on this project!
2011-07-27 12:37:14+0900
Minor correction to the above — it appears that the GUI does in fact list the removed extent device in the “Properties” dialog for the datastore (not sure if this was after a restart or if I was just plain wrong). In any case, as mentioned above, the size reported by this dialog still does show the larger size (i.e., including the removed extent), whereas debugvmfs’ “show_volume” re[prts the volume as if the extent were in fact removed.
2011-07-27 13:55:35+0900
spf1000: did you try to rescan the volumes from the GUI ?
2011-07-27 19:07:58+0900
Yes, but to no avail. A restart didn’t help either.
Running fsck.vmfs on the volume complains of 835 orphaned inodes (yikes!). My fingers are crossed that this doesn’t reflect actual filesystem damage.
Of course, shrinking the volume might have been the ideal originally , but at this point, I’m certainly willing to add the extent back somehow if it meant getting the filesystem into a reliable state.
I would be forever in your debt if you had any advice that could help in this situation.
2012-02-28 23:40:37+0900
Given that the underpinnings of VMFS has some resemblance to LVM; is there any hope of one day seeing an equivalent tool to pvmove; i.e. being able to shuffle used PEs from a given PV out to another PV, to enable the removal of specific extents?