Error Message

Feb 11 02:41:38 systemA vxio: [ID 750076 kern.warning] WARNING: VxVM vxio V-5-0-176 Highwater space usage on cache object cacheobj reached.
Feb 11 04:27:16 systemA vxio: [ID 512288 kern.warning] WARNING: VxVM vxio V-5-0-38 Cache object cacheobj is full.

Email sent by vxcache to root:
Message 27:
From root@sydfire03.sydlab.veritas.com Sat Feb 25 16:35:12 2012
Date: Sat, 25 Feb 2012 16:35:11 +1100 (EST)
From: Super-User <root@sydfire03.sydlab.veritas.com>
To: root@sydfire03.sydlab.veritas.com
Subject: ERROR: Volume Manager failed autogrow on host sydfire03
Grow operation failed for cache-volume acachevol associated to cache-object acacheobj in disk-group adg
 

If vxcached fails to reduce the cache object usage to below the high watermark, "needs_grow" flag will be left marked on the cache object.   When "needs_grow" flag is marked on a cache object, vxnotify will not send any "grow" messages to vxcached and vxcached will no longer be able to monitor the cache object.

Please check the email sent by vxcached to the root user periodically.   If vxcached fails to reduce the cache object usage to below the high watermark, manual intervention is required to fix the issue and restore the normal functioning of vxcached.

Please grow the cacheobj manually by using the vxcache command, e.g.

# vxcache -g "diskgroup" growcacheby "cacheobj" "length_change"
or
# vxcache -g "diskgroup" growcacheto "cacheobj" "length_final"

You can also reduce the cache object usage level by removing existing Space Optimized Snapshot, e.g.

# vxedit -g "diskgroup" -rf rm "space-optimized-snapshot"

Check that the "needs_grow" is now removed.

# vxprint -g adg -l acacheobj
Cache:    acacheobj
info:     region_size=128
          highwatermark=1
          autogrow=on
          autogrowby=81920
          max_autogrow=1024000
state:    state=ACTIVE kernel=ENABLED
assoc:    cache_volume=acachevol cache_volume_len=1730560
          sub-disks=acacheobj-03,snapvol01-S01
flags:    closed             <<<<<< make sure needs_grow flag is removed
 

Applies To

Veritas Volume Manager (VxVM) with Space Optimized Snapshots

A cache object is used by the Veritas Volume Manager (VxVM) Space Optimized Snapshot to cache the snapshot data when the data in source volume is modified.   The cache object can be set to autogrow and the vxcached (VxVM Cache Daemon) will monitor the usage level of the cache object and grow the cache object as required.   Please refer to the vxcached(1M) and vxcache(1M) manual pages for further details on the setup and configuration of the autogrow feature. vxcached monitors the cache object by running a vxnotify process to receive the "grow" notification from VxVM.   If the vxcached fails to grow the cache object, the cache object will be marked with "needs_grow" flag.   When the "needs_grow" flag is marked, vxnotify will stop sending further "grow" messages and vxcached will not be to detect any grow request.   If the usage level of the cache object continues to raise, the cache object can overflows eventually. The following is a typical message from vxnotify when the usage level of a cache object reaches the high watermark. # vxnotify -C -w 15
...
grow on cachevolume acachevol rid 0.1032 for cache acacheobj rid 0.1042 dg adg dgid 1320101897.53.sydfire03
... At the same time, VxVM will mark the cache object with the "needs_grow" flag. # vxprint -g adg -l acacheobj
Cache:    acacheobj
info:     region_size=128
          highwatermark=1
          autogrow=on
          autogrowby=81920
          max_autogrow=1024000
state:    state=ACTIVE kernel=ENABLED
assoc:    cache_volume=acachevol cache_volume_len=1730560
          sub-disks=acacheobj-03,snapvol01-S01
flags:    closed needs_grow             <<<<<< needs_grow
guid: {c1ceafca-5e95-11e1-bb91-0003baa59c37}
mediatype: hdd
  Under normal circumstances vxcached will be able to either grow the cache object or remove existing Space Optimized Snapshots to reduce the cache object usage to below the high watermark.    Once the cache object is grown or an existing Space Optimized Snapshot is removed, VxVM will remove the "needs_grow" flag.   After the "needs_grow" flag is removed, vxnotify will send further "grow" messages to vxcached if the high watermark is reached. If vxcached fails to grow the cache object (e.g. when the max_autogrow cache size is reached and there is no Space Optimized Snapshot than can be removed), vxcached will  send the following email to root user and leave the cache object marked with "needs_grow".   The "needs_grow" flag will prevent vxnotify from sending further "grow" message.    --------------- A typical message sent by vxcached with autogrow failed ---------------
Message 27:
From root@sydfire03.sydlab.veritas.com Sat Feb 25 16:35:12 2012
Date: Sat, 25 Feb 2012 16:35:11 +1100 (EST)
From: Super-User <root@sydfire03.sydlab.veritas.com>
To: root@sydfire03.sydlab.veritas.com
Subject: ERROR: Volume Manager failed autogrow on host sydfire03 Grow operation failed for cache-volume acachevol associated to cache-object acacheobj in disk-group adg
---------------- End messages ---------------------------------------------------------------- After receiving the above email, the system administrator must either grow the cache object manually or remove existing Space Optimized Snapshot manually to clear the "needs_grow" flag and restore the normal functioning of vxcached.