1LVMCACHE(7) LVMCACHE(7)
2
6 lvmcache — LVM caching
7
10 lvm(8) includes two kinds of caching that can be used to improve the
11 performance of a Logical Volume (LV). Typically, a smaller, faster
12 device is used to improve i/o performance of a larger, slower LV. To do
13 this, a separate LV is created from the faster device, and then the
14 original LV is converted to start using the fast LV.
15 The two kinds of caching are:
17 · A read and write hot-spot cache, using the dm-cache kernel module.
20 This cache is slow moving, and adjusts the cache content over time so
21 that the most used parts of the LV are kept on the faster device.
22 Both reads and writes use the cache. LVM refers to this using the LV
23 type cache.
24 · A streaming write cache, using the dm-writecache kernel module. This
27 cache is intended to be used with SSD or PMEM devices to speed up all
28 writes to an LV. Reads do not use this cache. LVM refers to this
29 using the LV type writecache.
30
33 Both kinds of caching use similar lvm commands:
34 1. Identify main LV that needs caching
36 A main LV exists on slower devices.
38 $ lvcreate -n main -L Size vg /dev/slow
40 2. Identify fast LV to use as the cache
42 A fast LV exists on faster devices. This LV will be used to hold the
44 cache.
45 $ lvcreate -n fast -L Size vg /dev/fast
47 $ lvs -a
49 LV Attr Type Devices
50 fast -wi------- linear /dev/fast
51 main -wi------- linear /dev/slow
52 3. Start caching the main LV
54 To start caching the main LV using the fast LV, convert the main LV to
56 the desired caching type, and specify the fast LV to use:
57 using dm-cache:
59 $ lvconvert --type cache --cachevol fast vg/main
61 using dm-writecache:
63 $ lvconvert --type writecache --cachevol fast vg/main
65 using dm-cache (with cachepool):
67 $ lvconvert --type cache --cachepool fast vg/main
69 4. Display LVs
71 Once the fast LV has been attached to the main LV, lvm reports the main
73 LV type as either cache or writecache depending on the type used.
74 While attached, the fast LV is hidden, and renamed with a _cvol or
75 _cpool suffix. It is displayed by lvs -a. The _corig or _wcorig LV
76 represents the original LV without the cache.
77 using dm-cache:
79 $ lvs -a
81 LV Pool Type Devices
82 main [fast_cvol] cache main_corig(0)
83 [fast_cvol] linear /dev/fast
84 [main_corig] linear /dev/slow
85 using dm-writecache:
87 $ lvs -a
89 LV Pool Type Devices
90 main [fast_cvol] writecache main_wcorig(0)
91 [fast_cvol] linear /dev/fast
92 [main_wcorig] linear /dev/slow
93 using dm-cache (with cachepool):
95 $ lvs -a
97 LV Pool Type Devices
98 main [fast_cpool] cache main_corig(0)
99 [fast_cpool] cache-pool fast_pool_cdata(0)
100 [fast_cpool_cdata] linear /dev/fast
101 [fast_cpool_cmeta] linear /dev/fast
102 [main_corig] linear /dev/slow
103 5. Use the main LV
105 Use the LV until the cache is no longer wanted, or needs to be changed.
107 6. Stop caching
109 To stop caching the main LV, separate the fast LV from the main LV.
111 This changes the type of the main LV back to what it was before the
112 cache was attached.
113 $ lvconvert --splitcache vg/main
115 $ lvs -a
117 LV VG Attr Type Devices
118 fast vg -wi------- linear /dev/fast
119 main vg -wi------- linear /dev/slow
120
124 option args
128 --cachevol LV
131 Pass this option a standard LV. With a cachevol, cache data and meta‐
133 data are contained within the single LV. This is used with dm-write‐
134 cache or dm-cache.
135 --cachepool CachePoolLV|LV
137 Pass this option a cache pool object. With a cache pool, lvm places
139 cache data and cache metadata on different LVs. The two LVs together
140 are called a cache pool. This permits specific placement of data and
141 metadata. A cache pool is represented as a special type of LV that
142 cannot be used directly. (If a standard LV is passed to this option,
143 lvm will first convert it to a cache pool by combining it with another
144 LV to use for metadata.) This can be used with dm-cache.
145 dm-cache block size
150 A cache pool will have a logical block size of 4096 bytes if it is cre‐
153 ated on a device with a logical block size of 4096 bytes.
154 If a main LV has logical block size 512 (with an existing xfs file sys‐
156 tem using that size), then it cannot use a cache pool with a 4096 logi‐
157 cal block size. If the cache pool is attached, the main LV will likely
158 fail to mount.
159 To avoid this problem, use a mkfs option to specify a 4096 block size
161 for the file system, or attach the cache pool before running mkfs.
162 dm-writecache block size
165 The dm-writecache block size can be 4096 bytes (the default), or 512
168 bytes. The default 4096 has better performance and should be used
169 except when 512 is necessary for compatibility. The dm-writecache
170 block size is specified with --cachesettings block_size=4096|512 when
171 caching is started.
172 When a file system like xfs already exists on the main LV prior to
174 caching, and the file system is using a block size of 512, then the
175 writecache block size should be set to 512. (The file system will
176 likely fail to mount if writecache block size of 4096 is used in this
177 case.)
178 Check the xfs sector size while the fs is mounted:
180 $ xfs_info /dev/vg/main
182 Look for sectsz=512 or sectsz=4096
183 The writecache block size should be chosen to match the xfs sectsz
185 value.
186 It is also possible to specify a sector size of 4096 to mkfs.xfs when
188 creating the file system. In this case the writecache block size of
189 4096 can be used.
190 dm-writecache settings
193 Tunable parameters can be passed to the dm-writecache kernel module
196 using the --cachesettings option when caching is started, e.g.
197 $ lvconvert --type writecache --cachevol fast \
199 --cachesettings 'high_watermark=N writeback_jobs=N' vg/main
200 Tunable options are:
202 · high_watermark = <count>
205 Start writeback when the number of used blocks reach this watermark
207 · low_watermark = <count>
210 Stop writeback when the number of used blocks drops below this water‐
212 mark
213 · writeback_jobs = <count>
216 Limit the number of blocks that are in flight during writeback. Set‐
218 ting this value reduces writeback throughput, but it may improve
219 latency of read requests.
220 · autocommit_blocks = <count>
223 When the application writes this amount of blocks without issuing the
225 FLUSH request, the blocks are automatically commited.
226 · autocommit_time = <milliseconds>
229 The data is automatically commited if this time passes and no FLUSH
231 request is received.
232 · fua = 0|1
235 Use the FUA flag when writing data from persistent memory back to the
237 underlying device. Applicable only to persistent memory.
238 · nofua = 0|1
241 Don't use the FUA flag when writing back data and send the FLUSH
243 request afterwards. Some underlying devices perform better with fua,
244 some with nofua. Testing is necessary to determine which. Applica‐
245 ble only to persistent memory.
246 dm-cache with separate data and metadata LVs
250 When using dm-cache, the cache metadata and cache data can be stored on
253 separate LVs. To do this, a "cache pool" is created, which is a spe‐
254 cial LV that references two sub LVs, one for data and one for metadata.
255 To create a cache pool from two separate LVs:
257 $ lvcreate -n fast -L DataSize vg /dev/fast1
259 $ lvcreate -n fastmeta -L MetadataSize vg /dev/fast2
260 $ lvconvert --type cache-pool --poolmetadata fastmeta vg/fast
261 Then use the cache pool LV to start caching the main LV:
263 $ lvconvert --type cache --cachepool fast vg/main
265 A variation of the same procedure automatically creates a cache pool
267 when caching is started. To do this, use a standard LV as the
268 --cachepool (this will hold cache data), and use another standard LV as
269 the --poolmetadata (this will hold cache metadata). LVM will create a
270 cache pool LV from the two specified LVs, and use the cache pool to
271 start caching the main LV.
272 $ lvcreate -n fast -L DataSize vg /dev/fast1
274 $ lvcreate -n fastmeta -L MetadataSize vg /dev/fast2
275 $ lvconvert --type cache --cachepool fast --poolmetadata fastmeta vg/main
276 dm-cache cache modes
279 The default dm-cache cache mode is "writethrough". Writethrough
282 ensures that any data written will be stored both in the cache and on
283 the origin LV. The loss of a device associated with the cache in this
284 case would not mean the loss of any data.
285 A second cache mode is "writeback". Writeback delays writing data
287 blocks from the cache back to the origin LV. This mode will increase
288 performance, but the loss of a cache device can result in lost data.
289 With the --cachemode option, the cache mode can be set when caching is
291 started, or changed on an LV that is already cached. The current cache
292 mode can be displayed with the cache_mode reporting option:
293 lvs -o+cache_mode VG/LV
295 lvm.conf(5) allocation/cache_mode
297 defines the default cache mode.
298 $ lvconvert --type cache --cachevol fast \
300 --cachemode writethrough vg/main
301 dm-cache chunk size
304 The size of data blocks managed by dm-cache can be specified with the
307 --chunksize option when caching is started. The default unit is KiB.
308 The value must be a multiple of 32KiB between 32KiB and 1GiB.
309 Using a chunk size that is too large can result in wasteful use of the
311 cache, in which small reads and writes cause large sections of an LV to
312 be stored in the cache. However, choosing a chunk size that is too
313 small can result in more overhead trying to manage the numerous chunks
314 that become mapped into the cache. Overhead can include both excessive
315 CPU time searching for chunks, and excessive memory tracking chunks.
316 Command to display the chunk size:
318 lvs -o+chunksize VG/LV
319 lvm.conf(5) cache_pool_chunk_size
321 controls the default chunk size.
322 The default value is shown by:
324 lvmconfig --type default allocation/cache_pool_chunk_size
325 dm-cache cache policy
329 The dm-cache subsystem has additional per-LV parameters: the cache pol‐
332 icy to use, and possibly tunable parameters for the cache policy.
333 Three policies are currently available: "smq" is the default policy,
334 "mq" is an older implementation, and "cleaner" is used to force the
335 cache to write back (flush) all cached writes to the origin LV.
336 The older "mq" policy has a number of tunable parameters. The defaults
338 are chosen to be suitable for the majority of systems, but in special
339 circumstances, changing the settings can improve performance.
340 With the --cachepolicy and --cachesettings options, the cache policy
342 and settings can be set when caching is started, or changed on an
343 existing cached LV (both options can be used together). The current
344 cache policy and settings can be displayed with the cache_policy and
345 cache_settings reporting options:
346 lvs -o+cache_policy,cache_settings VG/LV
348 Change the cache policy and settings of an existing LV.
350 $ lvchange --cachepolicy mq --cachesettings \
352 'migration_threshold=2048 random_threshold=4' vg/main
353 lvm.conf(5) allocation/cache_policy
355 defines the default cache policy.
356 lvm.conf(5) allocation/cache_settings
358 defines the default cache settings.
359 dm-cache spare metadata LV
362 See lvmthin(7) for a description of the "pool metadata spare" LV. The
365 same concept is used for cache pools.
366 dm-cache metadata formats
369 There are two disk formats for dm-cache metadata. The metadata format
372 can be specified with --cachemetadataformat when caching is started,
373 and cannot be changed. Format 2 has better performance; it is more
374 compact, and stores dirty bits in a separate btree, which improves the
375 speed of shutting down the cache. With auto, lvm selects the best
376 option provided by the current dm-cache kernel module.
377 mirrored cache device
380 The fast LV holding the cache can be created as a raid1 mirror so that
383 it can tolerate a device failure. (When using dm-cache with separate
384 data and metadata LVs, each of the sub-LVs can use raid1.)
385 $ lvcreate -n main -L Size vg /dev/slow
387 $ lvcreate --type raid1 -m 1 -n fast -L Size vg /dev/fast1 /dev/fast2
388 $ lvconvert --type cache --cachevol fast vg/main
389 dm-cache command shortcut
392 A single command can be used to create a cache pool and attach that new
395 cache pool to a main LV:
396 $ lvcreate --type cache --name Name --size Size VG/LV [PV]
398 In this command, the specified LV already exists, and is the main LV to
400 be cached. The command creates a new cache pool with the given name
401 and size, using the optionally specified PV (typically an ssd). Then
402 it attaches the new cache pool to the existing main LV to begin
403 caching.
404 (Note: ensure that the specified main LV is a standard LV. If a cache
406 pool LV is mistakenly specified, then the command does something dif‐
407 ferent.)
408 (Note: the type option is interpreted differently by this command than
410 by normal lvcreate commands in which --type specifies the type of the
411 newly created LV. In this case, an LV with type cache-pool is being
412 created, and the existing main LV is being converted to type cache.)
413
417 lvm.conf(5), lvchange(8), lvcreate(8), lvdisplay(8), lvextend(8), lvre‐
418 move(8), lvrename(8), lvresize(8), lvs(8), vgchange(8), vgmerge(8),
419 vgreduce(8), vgsplit(8)
420Red Hat, Inc LVM TOOLS 2.03.06(2) (2019-10-23) LVMCACHE(7)