1LVMCACHE(7) LVMCACHE(7)
2
3
4
6 lvmcache — LVM caching
7
8
10 lvm(8) includes two kinds of caching that can be used to improve the
11 performance of a Logical Volume (LV). Typically, a smaller, faster
12 device is used to improve i/o performance of a larger, slower LV. To do
13 this, a separate LV is created from the faster device, and then the
14 original LV is converted to start using the fast LV.
15
16 The two kinds of caching are:
17
18
19 · A read and write hot-spot cache, using the dm-cache kernel module.
20 This cache is slow moving, and adjusts the cache content over time so
21 that the most used parts of the LV are kept on the faster device.
22 Both reads and writes use the cache. LVM refers to this using the LV
23 type cache.
24
25
26 · A streaming write cache, using the dm-writecache kernel module. This
27 cache is intended to be used with SSD or PMEM devices to speed up all
28 writes to an LV. Reads do not use this cache. LVM refers to this
29 using the LV type writecache.
30
31
33 Both kinds of caching use similar lvm commands:
34
35 1. Identify main LV that needs caching
36
37 A main LV exists on slower devices.
38
39 $ lvcreate -n main -L Size vg /dev/slow
40
41 2. Identify fast LV to use as the cache
42
43 A fast LV exists on faster devices. This LV will be used to hold the
44 cache.
45
46 $ lvcreate -n fast -L Size vg /dev/fast
47
48 $ lvs vg -o+devices
49 LV VG Attr LSize Devices
50 fast vg -wi------- xx.00m /dev/fast(0)
51 main vg -wi------- yyy.00m /dev/slow(0)
52
53 3. Start caching the main LV
54
55 To start caching the main LV using the fast LV, convert the main LV to
56 the desired caching type, and specify the fast LV to use:
57
58 using dm-cache:
59
60 $ lvconvert --type cache --cachepool fast vg/main
61
62 or dm-writecache:
63
64 $ lvconvert --type writecache --cachepool fast vg/main
65
66 4. Display LVs
67
68 Once the fast LV has been attached to the main LV, lvm reports the main
69 LV type as either cache or writecache depending on the type used.
70 While attached, the fast LV is hidden, and only displayed when lvs is
71 given -a. The _corig or _wcorig LV represents the original LV without
72 the cache.
73
74 using dm-cache:
75
76 $ lvs -a -o name,vgname,lvattr,origin,segtype,devices vg
77 LV VG Attr Origin Type Devices
78 [fast] vg Cwi-aoC--- linear /dev/fast(xx)
79 main vg Cwi-a-C--- [main_corig] cache main_corig(0)
80 [main_corig] vg owi-aoC--- linear /dev/slow(0)
81
82 or dm-writecache:
83
84 $ lvs -a -o name,vgname,lvattr,origin,segtype,devices vg
85 LV VG Attr Origin Type Devices
86 [fast] vg -wi-ao---- linear /dev/fast(xx)
87 main vg Cwi-a----- [main_wcorig] writecache main_wcorig(0)
88 [main_wcorig] vg -wi-ao---- linear /dev/slow(0)
89
90 5. Use the main LV
91
92 Use the LV until the cache is no longer wanted, or needs to be changed.
93
94 6. Stop caching
95
96 To stop caching the main LV, separate the fast LV from the main LV.
97 This changes the type of the main LV back to what it was before the
98 cache was attached.
99
100 $ lvconvert --splitcache vg/main
101
102
103
105
106
107
108 dm-writecache block size
109
110
111 The dm-writecache block size can be 4096 bytes (the default), or 512
112 bytes. The default 4096 has better performance and should be used
113 except when 512 is necessary for compatibility. The dm-writecache
114 block size is specified with --writecacheblocksize 4096b|512b when
115 caching is started.
116
117 When a file system like xfs already exists on the main LV prior to
118 caching, and the file system is using a block size of 512, then the
119 writecache block size should be set to 512. (The file system will
120 likely fail to mount if writecache block size of 4096 is used in this
121 case.)
122
123 Check the xfs sector size while the fs is mounted:
124
125 $ xfs_info /dev/vg/main
126 Look for sectsz=512 or sectsz=4096
127
128 The writecache block size should be chosen to match the xfs sectsz
129 value.
130
131 It is also possible to specify a sector size of 4096 to mkfs.xfs when
132 creating the file system. In this case the writecache block size of
133 4096 can be used.
134
135
136 dm-writecache settings
137
138
139 Tunable parameters can be passed to the dm-writecache kernel module
140 using the --cachesettings option when caching is started, e.g.
141
142 $ lvconvert --type writecache --cachepool fast \
143 --cachesettings 'high_watermark=N writeback_jobs=N' vg/main
144
145 Tunable options are:
146
147
148 · high_watermark = <count>
149
150 Start writeback when the number of used blocks reach this watermark
151
152
153 · low_watermark = <count>
154
155 Stop writeback when the number of used blocks drops below this water‐
156 mark
157
158
159 · writeback_jobs = <count>
160
161 Limit the number of blocks that are in flight during writeback. Set‐
162 ting this value reduces writeback throughput, but it may improve
163 latency of read requests.
164
165
166 · autocommit_blocks = <count>
167
168 When the application writes this amount of blocks without issuing the
169 FLUSH request, the blocks are automatically commited.
170
171
172 · autocommit_time = <milliseconds>
173
174 The data is automatically commited if this time passes and no FLUSH
175 request is received.
176
177
178 · fua = 0|1
179
180 Use the FUA flag when writing data from persistent memory back to the
181 underlying device. Applicable only to persistent memory.
182
183
184 · nofua = 0|1
185
186 Don't use the FUA flag when writing back data and send the FLUSH
187 request afterwards. Some underlying devices perform better with fua,
188 some with nofua. Testing is necessary to determine which. Applica‐
189 ble only to persistent memory.
190
191
192
193 dm-cache with separate data and metadata LVs
194
195
196 When using dm-cache, the cache metadata and cache data can be stored on
197 separate LVs. To do this, a "cache-pool LV" is created, which is a
198 special LV that references two sub LVs, one for data and one for meta‐
199 data.
200
201 To create a cache-pool LV from two separate LVs:
202
203 $ lvcreate -n fastpool -L DataSize vg /dev/fast1
204 $ lvcreate -n fastpoolmeta -L MetadataSize vg /dev/fast2
205 $ lvconvert --type cache-pool --poolmetadata fastpoolmeta vg/fastpool
206
207 Then use the cache-pool LV to start caching the main LV:
208
209 $ lvconvert --type cache --cachepool fastpool vg/main
210
211 A variation of the same procedure automatically creates a cache-pool
212 when caching is started. To do this, use a standard LV as the
213 --cachepool (this will hold cache data), and use another standard LV as
214 the --poolmetadata (this will hold cache metadata). LVM will create a
215 cache-pool LV from the two specified LVs, and use the cache-pool to
216 start caching the main LV.
217
218 $ lvcreate -n fastpool -L DataSize vg /dev/fast1
219 $ lvcreate -n fastpoolmeta -L MetadataSize vg /dev/fast2
220 $ lvconvert --type cache --cachepool fastpool \
221 --poolmetadata fastpoolmeta vg/main
222
223
224 dm-cache cache modes
225
226
227 The default dm-cache cache mode is "writethrough". Writethrough
228 ensures that any data written will be stored both in the cache and on
229 the origin LV. The loss of a device associated with the cache in this
230 case would not mean the loss of any data.
231
232 A second cache mode is "writeback". Writeback delays writing data
233 blocks from the cache back to the origin LV. This mode will increase
234 performance, but the loss of a cache device can result in lost data.
235
236 With the --cachemode option, the cache mode can be set when caching is
237 started, or changed on an LV that is already cached. The current cache
238 mode can be displayed with the cache_mode reporting option:
239
240 lvs -o+cache_mode VG/LV
241
242 lvm.conf(5) allocation/cache_mode
243 defines the default cache mode.
244
245 $ lvconvert --type cache --cachepool fast \
246 --cachemode writethrough vg/main
247
248
249 dm-cache chunk size
250
251
252 The size of data blocks managed by dm-cache can be specified with the
253 --chunksize option when caching is started. The default unit is KiB.
254 The value must be a multiple of 32KiB between 32KiB and 1GiB.
255
256 Using a chunk size that is too large can result in wasteful use of the
257 cache, in which small reads and writes cause large sections of an LV to
258 be stored in the cache. However, choosing a chunk size that is too
259 small can result in more overhead trying to manage the numerous chunks
260 that become mapped into the cache. Overhead can include both excessive
261 CPU time searching for chunks, and excessive memory tracking chunks.
262
263 Command to display the chunk size:
264 lvs -o+chunksize VG/LV
265
266 lvm.conf(5) cache_pool_chunk_size
267 controls the default chunk size.
268
269 The default value is shown by:
270 lvmconfig --type default allocation/cache_pool_chunk_size
271
272
273
274 dm-cache cache policy
275
276
277 The dm-cache subsystem has additional per-LV parameters: the cache pol‐
278 icy to use, and possibly tunable parameters for the cache policy.
279 Three policies are currently available: "smq" is the default policy,
280 "mq" is an older implementation, and "cleaner" is used to force the
281 cache to write back (flush) all cached writes to the origin LV.
282
283 The older "mq" policy has a number of tunable parameters. The defaults
284 are chosen to be suitable for the majority of systems, but in special
285 circumstances, changing the settings can improve performance.
286
287 With the --cachepolicy and --cachesettings options, the cache policy
288 and settings can be set when caching is started, or changed on an
289 existing cached LV (both options can be used together). The current
290 cache policy and settings can be displayed with the cache_policy and
291 cache_settings reporting options:
292
293 lvs -o+cache_policy,cache_settings VG/LV
294
295 Change the cache policy and settings of an existing LV.
296
297 $ lvchange --cachepolicy mq --cachesettings \
298 'migration_threshold=2048 random_threshold=4' vg/main
299
300 lvm.conf(5) allocation/cache_policy
301 defines the default cache policy.
302
303 lvm.conf(5) allocation/cache_settings
304 defines the default cache settings.
305
306
307 dm-cache spare metadata LV
308
309
310 See lvmthin(7) for a description of the "pool metadata spare" LV. The
311 same concept is used for cache pools.
312
313
314 dm-cache metadata formats
315
316
317 There are two disk formats for dm-cache metadata. The metadata format
318 can be specified with --cachemetadataformat when caching is started,
319 and cannot be changed. Format 2 has better performance; it is more
320 compact, and stores dirty bits in a separate btree, which improves the
321 speed of shutting down the cache. With auto, lvm selects the best
322 option provided by the current dm-cache kernel module.
323
324
325 mirrored cache device
326
327
328 The fast LV holding the cache can be created as a raid1 mirror so that
329 it can tolerate a device failure. (When using dm-cache with separate
330 data and metadata LVs, each of the sub-LVs can use raid1.)
331
332 $ lvcreate -n main -L Size vg /dev/slow
333 $ lvcreate --type raid1 -m 1 -n fast -L Size vg /dev/fast1 /dev/fast2
334 $ lvconvert --type cache --cachepool fast vg/main
335
336
338 lvm.conf(5), lvchange(8), lvcreate(8), lvdisplay(8), lvextend(8), lvre‐
339 move(8), lvrename(8), lvresize(8), lvs(8), vgchange(8), vgmerge(8),
340 vgreduce(8), vgsplit(8)
341
342
343
344Red Hat, Inc LVM TOOLS 2.03.02(2)-RHEL8 (2019-01-04) LVMCACHE(7)