1BTRFS(5) BTRFS BTRFS(5)
2
6 btrfs - topics about the BTRFS filesystem (mount options, supported
7 file attributes and other)
8
10 This document describes topics related to BTRFS that are not specific
11 to the tools. Currently covers:
12 1. mount options
14 2. filesystem features
16 3. checksum algorithms
18 4. compression
20 5. sysfs interface
22 6. filesystem exclusive operations
24 7. filesystem limits
26 8. bootloader support
28 9. file attributes
30 10. zoned mode
32 11. control device
34 12. filesystems with multiple block group profiles
36 13. seeding device
38 14. RAID56 status and recommended practices
40 15. storage model, hardware considerations
42
44 BTRFS SPECIFIC MOUNT OPTIONS
45 This section describes mount options specific to BTRFS. For the
46 generic mount options please refer to mount(8) manual page. The options
47 are sorted alphabetically (discarding the no prefix).
48 NOTE:
50 Most mount options apply to the whole filesystem and only options in
51 the first mounted subvolume will take effect. This is due to lack of
52 implementation and may change in the future. This means that (for
53 example) you can't set per-subvolume nodatacow, nodatasum, or com‐
54 press using mount options. This should eventually be fixed, but it
55 has proved to be difficult to implement correctly within the Linux
56 VFS framework.
57 Mount options are processed in order, only the last occurrence of an
59 option takes effect and may disable other options due to constraints
60 (see e.g. nodatacow and compress). The output of mount command shows
61 which options have been applied.
62 acl, noacl
64 (default: on)
65 Enable/disable support for POSIX Access Control Lists (ACLs).
67 See the acl(5) manual page for more information about ACLs.
68 The support for ACL is build-time configurable
70 (BTRFS_FS_POSIX_ACL) and mount fails if acl is requested but the
71 feature is not compiled in.
72 autodefrag, noautodefrag
74 (since: 3.0, default: off)
75 Enable automatic file defragmentation. When enabled, small ran‐
77 dom writes into files (in a range of tens of kilobytes, cur‐
78 rently it's 64KiB) are detected and queued up for the defragmen‐
79 tation process. May not be well suited for large database work‐
80 loads.
81 The read latency may increase due to reading the adjacent blocks
83 that make up the range for defragmentation, successive write
84 will merge the blocks in the new location.
85 WARNING:
87 Defragmenting with Linux kernel versions < 3.9 or ≥ 3.14-rc2
88 as well as with Linux stable kernel versions ≥ 3.10.31, ≥
89 3.12.12 or ≥ 3.13.4 will break up the reflinks of COW data
90 (for example files copied with cp --reflink, snapshots or
91 de-duplicated data). This may cause considerable increase of
92 space usage depending on the broken up reflinks.
93 barrier, nobarrier
95 (default: on)
96 Ensure that all IO write operations make it through the device
98 cache and are stored permanently when the filesystem is at its
99 consistency checkpoint. This typically means that a flush com‐
100 mand is sent to the device that will synchronize all pending
101 data and ordinary metadata blocks, then writes the superblock
102 and issues another flush.
103 The write flushes incur a slight hit and also prevent the IO
105 block scheduler to reorder requests in a more effective way.
106 Disabling barriers gets rid of that penalty but will most cer‐
107 tainly lead to a corrupted filesystem in case of a crash or
108 power loss. The ordinary metadata blocks could be yet unwritten
109 at the time the new superblock is stored permanently, expecting
110 that the block pointers to metadata were stored permanently be‐
111 fore.
112 On a device with a volatile battery-backed write-back cache, the
114 nobarrier option will not lead to filesystem corruption as the
115 pending blocks are supposed to make it to the permanent storage.
116 check_int, check_int_data, check_int_print_mask=<value>
118 (since: 3.0, default: off)
119 These debugging options control the behavior of the integrity
121 checking module (the BTRFS_FS_CHECK_INTEGRITY config option re‐
122 quired). The main goal is to verify that all blocks from a given
123 transaction period are properly linked.
124 check_int enables the integrity checker module, which examines
126 all block write requests to ensure on-disk consistency, at a
127 large memory and CPU cost.
128 check_int_data includes extent data in the integrity checks, and
130 implies the check_int option.
131 check_int_print_mask takes a bitmask of BTRFSIC_PRINT_MASK_*
133 values as defined in fs/btrfs/check-integrity.c, to control the
134 integrity checker module behavior.
135 See comments at the top of fs/btrfs/check-integrity.c for more
137 information.
138 clear_cache
140 Force clearing and rebuilding of the free space cache if some‐
141 thing has gone wrong.
142 For free space cache v1, this only clears (and, unless
144 nospace_cache is used, rebuilds) the free space cache for block
145 groups that are modified while the filesystem is mounted with
146 that option. To actually clear an entire free space cache v1,
147 see btrfs check --clear-space-cache v1.
148 For free space cache v2, this clears the entire free space
150 cache. To do so without requiring to mounting the filesystem,
151 see btrfs check --clear-space-cache v2.
152 See also: space_cache.
154 commit=<seconds>
156 (since: 3.12, default: 30)
157 Set the interval of periodic transaction commit when data are
159 synchronized to permanent storage. Higher interval values lead
160 to larger amount of unwritten data, which has obvious conse‐
161 quences when the system crashes. The upper bound is not forced,
162 but a warning is printed if it's more than 300 seconds (5 min‐
163 utes). Use with care.
164 compress, compress=<type[:level]>, compress-force, com‐
166 press-force=<type[:level]>
167 (default: off, level support since: 5.1)
168 Control BTRFS file data compression. Type may be specified as
170 zlib, lzo, zstd or no (for no compression, used for remounting).
171 If no type is specified, zlib is used. If compress-force is
172 specified, then compression will always be attempted, but the
173 data may end up uncompressed if the compression would make them
174 larger.
175 Both zlib and zstd (since version 5.1) expose the compression
177 level as a tunable knob with higher levels trading speed and
178 memory (zstd) for higher compression ratios. This can be set by
179 appending a colon and the desired level. ZLIB accepts the range
180 [1, 9] and ZSTD accepts [1, 15]. If no level is set, both cur‐
181 rently use a default level of 3. The value 0 is an alias for the
182 default level.
183 Otherwise some simple heuristics are applied to detect an incom‐
185 pressible file. If the first blocks written to a file are not
186 compressible, the whole file is permanently marked to skip com‐
187 pression. As this is too simple, the compress-force is a work‐
188 around that will compress most of the files at the cost of some
189 wasted CPU cycles on failed attempts. Since kernel 4.15, a set
190 of heuristic algorithms have been improved by using frequency
191 sampling, repeated pattern detection and Shannon entropy calcu‐
192 lation to avoid that.
193 NOTE:
195 If compression is enabled, nodatacow and nodatasum are dis‐
196 abled.
197 datacow, nodatacow
199 (default: on)
200 Enable data copy-on-write for newly created files. Nodatacow
202 implies nodatasum, and disables compression. All files created
203 under nodatacow are also set the NOCOW file attribute (see
204 chattr(1)).
205 NOTE:
207 If nodatacow or nodatasum are enabled, compression is dis‐
208 abled.
209 Updates in-place improve performance for workloads that do fre‐
211 quent overwrites, at the cost of potential partial writes, in
212 case the write is interrupted (system crash, device failure).
213 datasum, nodatasum
215 (default: on)
216 Enable data checksumming for newly created files. Datasum im‐
218 plies datacow, i.e. the normal mode of operation. All files cre‐
219 ated under nodatasum inherit the "no checksums" property, how‐
220 ever there's no corresponding file attribute (see chattr(1)).
221 NOTE:
223 If nodatacow or nodatasum are enabled, compression is dis‐
224 abled.
225 There is a slight performance gain when checksums are turned
227 off, the corresponding metadata blocks holding the checksums do
228 not need to updated. The cost of checksumming of the blocks in
229 memory is much lower than the IO, modern CPUs feature hardware
230 support of the checksumming algorithm.
231 degraded
233 (default: off)
234 Allow mounts with fewer devices than the RAID profile con‐
236 straints require. A read-write mount (or remount) may fail when
237 there are too many devices missing, for example if a stripe mem‐
238 ber is completely missing from RAID0.
239 Since 4.14, the constraint checks have been improved and are
241 verified on the chunk level, not at the device level. This al‐
242 lows degraded mounts of filesystems with mixed RAID profiles for
243 data and metadata, even if the device number constraints would
244 not be satisfied for some of the profiles.
245 Example: metadata -- raid1, data -- single, devices -- /dev/sda,
247 /dev/sdb
248 Suppose the data are completely stored on sda, then missing sdb
250 will not prevent the mount, even if 1 missing device would nor‐
251 mally prevent (any) single profile to mount. In case some of the
252 data chunks are stored on sdb, then the constraint of sin‐
253 gle/data is not satisfied and the filesystem cannot be mounted.
254 device=<devicepath>
256 Specify a path to a device that will be scanned for BTRFS
257 filesystem during mount. This is usually done automatically by a
258 device manager (like udev) or using the btrfs device scan com‐
259 mand (e.g. run from the initial ramdisk). In cases where this is
260 not possible the device mount option can help.
261 NOTE:
263 Booting e.g. a RAID1 system may fail even if all filesystem's
264 device paths are provided as the actual device nodes may not
265 be discovered by the system at that point.
266 discard, discard=sync, discard=async, nodiscard
268 (default: async when devices support it since 6.2, async support
269 since: 5.6)
270 Enable discarding of freed file blocks. This is useful for SSD
272 devices, thinly provisioned LUNs, or virtual machine images;
273 however, every storage layer must support discard for it to
274 work.
275 In the synchronous mode (sync or without option value), lack of
277 asynchronous queued TRIM on the backing device TRIM can severely
278 degrade performance, because a synchronous TRIM operation will
279 be attempted instead. Queued TRIM requires newer than SATA revi‐
280 sion 3.1 chipsets and devices.
281 The asynchronous mode (async) gathers extents in larger chunks
283 before sending them to the devices for TRIM. The overhead and
284 performance impact should be negligible compared to the previous
285 mode and it's supposed to be the preferred mode if needed.
286 If it is not necessary to immediately discard freed blocks, then
288 the fstrim tool can be used to discard all free blocks in a
289 batch. Scheduling a TRIM during a period of low system activity
290 will prevent latent interference with the performance of other
291 operations. Also, a device may ignore the TRIM command if the
292 range is too small, so running a batch discard has a greater
293 probability of actually discarding the blocks.
294 enospc_debug, noenospc_debug
296 (default: off)
297 Enable verbose output for some ENOSPC conditions. It's safe to
299 use but can be noisy if the system reaches near-full state.
300 fatal_errors=<action>
302 (since: 3.4, default: bug)
303 Action to take when encountering a fatal error.
305 bug BUG() on a fatal error, the system will stay in the
307 crashed state and may be still partially usable, but re‐
308 boot is required for full operation
309 panic panic() on a fatal error, depending on other system con‐
311 figuration, this may be followed by a reboot. Please re‐
312 fer to the documentation of kernel boot parameters, e.g.
313 panic, oops or crashkernel.
314 flushoncommit, noflushoncommit
316 (default: off)
317 This option forces any data dirtied by a write in a prior trans‐
319 action to commit as part of the current commit, effectively a
320 full filesystem sync.
321 This makes the committed state a fully consistent view of the
323 file system from the application's perspective (i.e. it includes
324 all completed file system operations). This was previously the
325 behavior only when a snapshot was created.
326 When off, the filesystem is consistent but buffered writes may
328 last more than one transaction commit.
329 fragment=<type>
331 (depends on compile-time option CONFIG_BTRFS_DEBUG, since: 4.4,
332 default: off)
333 A debugging helper to intentionally fragment given type of block
335 groups. The type can be data, metadata or all. This mount option
336 should not be used outside of debugging environments and is not
337 recognized if the kernel config option CONFIG_BTRFS_DEBUG is not
338 enabled.
339 nologreplay
341 (default: off, even read-only)
342 The tree-log contains pending updates to the filesystem until
344 the full commit. The log is replayed on next mount, this can be
345 disabled by this option. See also treelog. Note that nologre‐
346 play is the same as norecovery.
347 WARNING:
349 Currently, the tree log is replayed even with a read-only
350 mount! To disable that behaviour, mount also with nologre‐
351 play.
352 max_inline=<bytes>
354 (default: min(2048, page size) )
355 Specify the maximum amount of space, that can be inlined in a
357 metadata b-tree leaf. The value is specified in bytes, option‐
358 ally with a K suffix (case insensitive). In practice, this
359 value is limited by the filesystem block size (named sectorsize
360 at mkfs time), and memory page size of the system. In case of
361 sectorsize limit, there's some space unavailable due to b-tree
362 leaf headers. For example, a 4KiB sectorsize, maximum size of
363 inline data is about 3900 bytes.
364 Inlining can be completely turned off by specifying 0. This will
366 increase data block slack if file sizes are much smaller than
367 block size but will reduce metadata consumption in return.
368 NOTE:
370 The default value has changed to 2048 in kernel 4.6.
371 metadata_ratio=<value>
373 (default: 0, internal logic)
374 Specifies that 1 metadata chunk should be allocated after every
376 value data chunks. Default behaviour depends on internal logic,
377 some percent of unused metadata space is attempted to be main‐
378 tained but is not always possible if there's not enough space
379 left for chunk allocation. The option could be useful to over‐
380 ride the internal logic in favor of the metadata allocation if
381 the expected workload is supposed to be metadata intense (snap‐
382 shots, reflinks, xattrs, inlined files).
383 norecovery
385 (since: 4.5, default: off)
386 Do not attempt any data recovery at mount time. This will dis‐
388 able logreplay and avoids other write operations. Note that this
389 option is the same as nologreplay.
390 NOTE:
392 The opposite option recovery used to have different meaning
393 but was changed for consistency with other filesystems, where
394 norecovery is used for skipping log replay. BTRFS does the
395 same and in general will try to avoid any write operations.
396 rescan_uuid_tree
398 (since: 3.12, default: off)
399 Force check and rebuild procedure of the UUID tree. This should
401 not normally be needed.
402 rescue (since: 5.9)
404 Modes allowing mount with damaged filesystem structures.
406 • usebackuproot (since: 5.9, replaces standalone option useback‐
408 uproot)
409 • nologreplay (since: 5.9, replaces standalone option nologre‐
411 play)
412 • ignorebadroots, ibadroots (since: 5.11)
414 • ignoredatacsums, idatacsums (since: 5.11)
416 • all (since: 5.9)
418 skip_balance
420 (since: 3.3, default: off)
421 Skip automatic resume of an interrupted balance operation. The
423 operation can later be resumed with btrfs balance resume, or the
424 paused state can be removed with btrfs balance cancel. The de‐
425 fault behaviour is to resume an interrupted balance immediately
426 after a volume is mounted.
427 space_cache, space_cache=<version>, nospace_cache
429 (nospace_cache since: 3.2, space_cache=v1 and space_cache=v2
430 since 4.5, default: space_cache=v2)
431 Options to control the free space cache. The free space cache
433 greatly improves performance when reading block group free space
434 into memory. However, managing the space cache consumes some re‐
435 sources, including a small amount of disk space.
436 There are two implementations of the free space cache. The orig‐
438 inal one, referred to as v1, used to be a safe default but has
439 been superseded by v2. The v1 space cache can be disabled at
440 mount time with nospace_cache without clearing.
441 On very large filesystems (many terabytes) and certain work‐
443 loads, the performance of the v1 space cache may degrade drasti‐
444 cally. The v2 implementation, which adds a new b-tree called the
445 free space tree, addresses this issue. Once enabled, the v2
446 space cache will always be used and cannot be disabled unless it
447 is cleared. Use clear_cache,space_cache=v1 or
448 clear_cache,nospace_cache to do so. If v2 is enabled, and v1
449 space cache will be cleared (at the first mount) and kernels
450 without v2 support will only be able to mount the filesystem in
451 read-only mode. On an unmounted filesystem the caches (both
452 versions) can be cleared by "btrfs check --clear-space-cache".
453 The btrfs-check(8) and :doc:`mkfs.btrfs commands have full v2
455 free space cache support since v4.19.
456 If a version is not explicitly specified, the default implemen‐
458 tation will be chosen, which is v2.
459 ssd, ssd_spread, nossd, nossd_spread
461 (default: SSD autodetected)
462 Options to control SSD allocation schemes. By default, BTRFS
464 will enable or disable SSD optimizations depending on status of
465 a device with respect to rotational or non-rotational type. This
466 is determined by the contents of /sys/block/DEV/queue/rota‐
467 tional). If it is 0, the ssd option is turned on. The option
468 nossd will disable the autodetection.
469 The optimizations make use of the absence of the seek penalty
471 that's inherent for the rotational devices. The blocks can be
472 typically written faster and are not offloaded to separate
473 threads.
474 NOTE:
476 Since 4.14, the block layout optimizations have been dropped.
477 This used to help with first generations of SSD devices.
478 Their FTL (flash translation layer) was not effective and the
479 optimization was supposed to improve the wear by better
480 aligning blocks. This is no longer true with modern SSD de‐
481 vices and the optimization had no real benefit. Furthermore
482 it caused increased fragmentation. The layout tuning has been
483 kept intact for the option ssd_spread.
484 The ssd_spread mount option attempts to allocate into bigger and
486 aligned chunks of unused space, and may perform better on
487 low-end SSDs. ssd_spread implies ssd, enabling all other SSD
488 heuristics as well. The option nossd will disable all SSD op‐
489 tions while nossd_spread only disables ssd_spread.
490 subvol=<path>
492 Mount subvolume from path rather than the toplevel subvolume.
493 The path is always treated as relative to the toplevel subvol‐
494 ume. This mount option overrides the default subvolume set for
495 the given filesystem.
496 subvolid=<subvolid>
498 Mount subvolume specified by a subvolid number rather than the
499 toplevel subvolume. You can use btrfs subvolume list of btrfs
500 subvolume show to see subvolume ID numbers. This mount option
501 overrides the default subvolume set for the given filesystem.
502 NOTE:
504 If both subvolid and subvol are specified, they must point at
505 the same subvolume, otherwise the mount will fail.
506 thread_pool=<number>
508 (default: min(NRCPUS + 2, 8) )
509 The number of worker threads to start. NRCPUS is number of
511 on-line CPUs detected at the time of mount. Small number leads
512 to less parallelism in processing data and metadata, higher num‐
513 bers could lead to a performance hit due to increased locking
514 contention, process scheduling, cache-line bouncing or costly
515 data transfers between local CPU memories.
516 treelog, notreelog
518 (default: on)
519 Enable the tree logging used for fsync and O_SYNC writes. The
521 tree log stores changes without the need of a full filesystem
522 sync. The log operations are flushed at sync and transaction
523 commit. If the system crashes between two such syncs, the pend‐
524 ing tree log operations are replayed during mount.
525 WARNING:
527 Currently, the tree log is replayed even with a read-only
528 mount! To disable that behaviour, also mount with nologre‐
529 play.
530 The tree log could contain new files/directories, these would
532 not exist on a mounted filesystem if the log is not replayed.
533 usebackuproot
535 (since: 4.6, default: off)
536 Enable autorecovery attempts if a bad tree root is found at
538 mount time. Currently this scans a backup list of several pre‐
539 vious tree roots and tries to use the first readable. This can
540 be used with read-only mounts as well.
541 NOTE:
543 This option has replaced recovery.
544 user_subvol_rm_allowed
546 (default: off)
547 Allow subvolumes to be deleted by their respective owner. Other‐
549 wise, only the root user can do that.
550 NOTE:
552 Historically, any user could create a snapshot even if he was
553 not owner of the source subvolume, the subvolume deletion has
554 been restricted for that reason. The subvolume creation has
555 been restricted but this mount option is still required. This
556 is a usability issue. Since 4.18, the rmdir(2) syscall can
557 delete an empty subvolume just like an ordinary directory.
558 Whether this is possible can be detected at runtime, see
559 rmdir_subvol feature in FILESYSTEM FEATURES.
560 DEPRECATED MOUNT OPTIONS
562 List of mount options that have been removed, kept for backward compat‐
563 ibility.
564 recovery
566 (since: 3.2, default: off, deprecated since: 4.5)
567 NOTE:
569 This option has been replaced by usebackuproot and should not
570 be used but will work on 4.5+ kernels.
571 inode_cache, noinode_cache
573 (removed in: 5.11, since: 3.0, default: off)
574 NOTE:
576 The functionality has been removed in 5.11, any stale data
577 created by previous use of the inode_cache option can be re‐
578 moved by btrfs check --clear-ino-cache.
579 NOTES ON GENERIC MOUNT OPTIONS
581 Some of the general mount options from mount(8) that affect BTRFS and
582 are worth mentioning.
583 noatime
585 under read intensive work-loads, specifying noatime signifi‐
586 cantly improves performance because no new access time informa‐
587 tion needs to be written. Without this option, the default is
588 relatime, which only reduces the number of inode atime updates
589 in comparison to the traditional strictatime. The worst case for
590 atime updates under relatime occurs when many files are read
591 whose atime is older than 24 h and which are freshly snapshot‐
592 ted. In that case the atime is updated and COW happens - for
593 each file - in bulk. See also https://lwn.net/Articles/499293/ -
594 Atime and btrfs: a bad combination? (LWN, 2012-05-31).
595 Note that noatime may break applications that rely on atime up‐
597 times like the venerable Mutt (unless you use maildir mail‐
598 boxes).
599
601 The basic set of filesystem features gets extended over time. The back‐
602 ward compatibility is maintained and the features are optional, need to
603 be explicitly asked for so accidental use will not create incompatibil‐
604 ities.
605 There are several classes and the respective tools to manage the fea‐
607 tures:
608 at mkfs time only
610 This is namely for core structures, like the b-tree nodesize or
611 checksum algorithm, see mkfs.btrfs(8) for more details.
612 after mkfs, on an unmounted filesystem
614 Features that may optimize internal structures or add new struc‐
615 tures to support new functionality, see btrfstune(8). The com‐
616 mand btrfs inspect-internal dump-super /dev/sdx will dump a su‐
617 perblock, you can map the value of incompat_flags to the fea‐
618 tures listed below
619 after mkfs, on a mounted filesystem
621 The features of a filesystem (with a given UUID) are listed in
622 /sys/fs/btrfs/UUID/features/, one file per feature. The status
623 is stored inside the file. The value 1 is for enabled and ac‐
624 tive, while 0 means the feature was enabled at mount time but
625 turned off afterwards.
626 Whether a particular feature can be turned on a mounted filesys‐
628 tem can be found in the directory /sys/fs/btrfs/features/, one
629 file per feature. The value 1 means the feature can be enabled.
630 List of features (see also mkfs.btrfs(8) section FILESYSTEM FEATURES):
632 big_metadata
634 (since: 3.4)
635 the filesystem uses nodesize for metadata blocks, this can be
637 bigger than the page size
638 block_group_tree
640 (since: 6.1)
641 block group item representation using a dedicated b-tree, this
643 can greatly reduce mount time for large filesystems
644 compress_lzo
646 (since: 2.6.38)
647 the lzo compression has been used on the filesystem, either as a
649 mount option or via btrfs filesystem defrag.
650 compress_zstd
652 (since: 4.14)
653 the zstd compression has been used on the filesystem, either as
655 a mount option or via btrfs filesystem defrag.
656 default_subvol
658 (since: 2.6.34)
659 the default subvolume has been set on the filesystem
661 extended_iref
663 (since: 3.7)
664 increased hardlink limit per file in a directory to 65536, older
666 kernels supported a varying number of hardlinks depending on the
667 sum of all file name sizes that can be stored into one metadata
668 block
669 free_space_tree
671 (since: 4.5)
672 free space representation using a dedicated b-tree, successor of
674 v1 space cache
675 metadata_uuid
677 (since: 5.0)
678 the main filesystem UUID is the metadata_uuid, which stores the
680 new UUID only in the superblock while all metadata blocks still
681 have the UUID set at mkfs time, see btrfstune(8) for more
682 mixed_backref
684 (since: 2.6.31)
685 the last major disk format change, improved backreferences, now
687 default
688 mixed_groups
690 (since: 2.6.37)
691 mixed data and metadata block groups, i.e. the data and metadata
693 are not separated and occupy the same block groups, this mode is
694 suitable for small volumes as there are no constraints how the
695 remaining space should be used (compared to the split mode,
696 where empty metadata space cannot be used for data and vice
697 versa)
698 on the other hand, the final layout is quite unpredictable and
700 possibly highly fragmented, which means worse performance
701 no_holes
703 (since: 3.14)
704 improved representation of file extents where holes are not ex‐
706 plicitly stored as an extent, saves a few percent of metadata if
707 sparse files are used
708 raid1c34
710 (since: 5.5)
711 extended RAID1 mode with copies on 3 or 4 devices respectively
713 RAID56 (since: 3.9)
715 the filesystem contains or contained a RAID56 profile of block
717 groups
718 rmdir_subvol
720 (since: 4.18)
721 indicate that rmdir(2) syscall can delete an empty subvolume
723 just like an ordinary directory. Note that this feature only de‐
724 pends on the kernel version.
725 skinny_metadata
727 (since: 3.10)
728 reduced-size metadata for extent references, saves a few percent
730 of metadata
731 send_stream_version
733 (since: 5.10)
734 number of the highest supported send stream version
736 supported_checksums
738 (since: 5.5)
739 list of checksum algorithms supported by the kernel module, the
741 respective modules or built-in implementing the algorithms need
742 to be present to mount the filesystem, see section CHECKSUM AL‐
743 GORITHMS.
744 supported_sectorsizes
746 (since: 5.13)
747 list of values that are accepted as sector sizes (mkfs.btrfs
749 --sectorsize) by the running kernel
750 supported_rescue_options
752 (since: 5.11)
753 list of values for the mount option rescue that are supported by
755 the running kernel, see btrfs(5)
756 zoned (since: 5.12)
758 zoned mode is allocation/write friendly to host-managed zoned
760 devices, allocation space is partitioned into fixed-size zones
761 that must be updated sequentially, see section ZONED MODE
762
764 A swapfile, when active, is a file-backed swap area. It is supported
765 since kernel 5.0. Use swapon(8) to activate it, until then (respec‐
766 tively again after deactivating it with swapoff(8)) it's just a normal
767 file (with NODATACOW set), for which the special restrictions for ac‐
768 tive swapfiles don't apply.
769 There are some limitations of the implementation in BTRFS and Linux
771 swap subsystem:
772 • filesystem - must be only single device
774 • filesystem - must have only single data profile
776 • subvolume - cannot be snapshotted if it contains any active swapfiles
778 • swapfile - must be preallocated (i.e. no holes)
780 • swapfile - must be NODATACOW (i.e. also NODATASUM, no compression)
782 The limitations come namely from the COW-based design and mapping layer
784 of blocks that allows the advanced features like relocation and
785 multi-device filesystems. However, the swap subsystem expects simpler
786 mapping and no background changes of the file block location once
787 they've been assigned to swap.
788 With active swapfiles, the following whole-filesystem operations will
790 skip swapfile extents or may fail:
791 • balance - block groups with extents of any active swapfiles are
793 skipped and reported, the rest will be processed normally
794 • resize grow - unaffected
796 • resize shrink - works as long as the extents of any active swapfiles
798 are outside of the shrunk range
799 • device add - if the new devices do not interfere with any already ac‐
801 tive swapfiles this operation will work, though no new swapfile can
802 be activated afterwards
803 • device delete - if the device has been added as above, it can be also
805 deleted
806 • device replace - ditto
808 When there are no active swapfiles and a whole-filesystem exclusive op‐
810 eration is running (e.g. balance, device delete, shrink), the swapfiles
811 cannot be temporarily activated. The operation must finish first.
812 To create and activate a swapfile run the following commands:
814 # truncate -s 0 swapfile
816 # chattr +C swapfile
817 # fallocate -l 2G swapfile
818 # chmod 0600 swapfile
819 # mkswap swapfile
820 # swapon swapfile
821 Since version 6.1 it's possible to create the swapfile in a single com‐
823 mand (except the activation):
824 # btrfs filesystem mkswapfile --size 2G swapfile
826 # swapon swapfile
827 Please note that the UUID returned by the mkswap utility identifies the
829 swap "filesystem" and because it's stored in a file, it's not generally
830 visible and usable as an identifier unlike if it was on a block device.
831 Once activated the file will appear in /proc/swaps:
833 # cat /proc/swaps
835 Filename Type Size Used Priority
836 /path/swapfile file 2097152 0 -2
837 The swapfile can be created as one-time operation or, once properly
839 created, activated on each boot by the swapon -a command (usually
840 started by the service manager). Add the following entry to /etc/fstab,
841 assuming the filesystem that provides the /path has been already
842 mounted at this point. Additional mount options relevant for the swap‐
843 file can be set too (like priority, not the BTRFS mount options).
844 /path/swapfile none swap defaults 0 0
846 From now on the subvolume with the active swapfile cannot be snapshot‐
848 ted until the swapfile is deactivated again by swapoff. Then the swap‐
849 file is a regular file and the subvolume can be snapshotted again,
850 though this would prevent another activation any swapfile that has been
851 snapshotted. New swapfiles (not snapshotted) can be created and acti‐
852 vated.
853 Otherwise, an inactive swapfile does not affect the containing subvol‐
855 ume. Activation creates a temporary in-memory status and prevents some
856 file operations, but is not stored permanently.
857
859 A swapfile can be used for hibernation but it's not straightforward.
860 Before hibernation a resume offset must be written to file
861 /sys/power/resume_offset or the kernel command line parameter re‐
862 sume_offset must be set.
863 The value is the physical offset on the device. Note that this is not
865 the same value that filefrag prints as physical offset!
866 Btrfs filesystem uses mapping between logical and physical addresses
868 but here the physical can still map to one or more device-specific
869 physical block addresses. It's the device-specific physical offset that
870 is suitable as resume offset.
871 Since version 6.1 there's a command btrfs inspect-internal map-swapfile
873 that will print the device physical offset and the adjusted value for
874 /sys/power/resume_offset. Note that the value is divided by page size,
875 i.e. it's not the offset itself.
876 # btrfs filesystem mkswapfile swapfile
878 # btrfs inspect-internal map-swapfile swapfile
879 Physical start: 811511726080
880 Resume offset: 198122980
881 For scripting and convenience the option -r will print just the offset:
883 # btrfs inspect-internal map-swapfile -r swapfile
885 198122980
886 The command map-swapfile also verifies all the requirements, i.e. no
888 holes, single device, etc.
889
891 If the swapfile activation fails please verify that you followed all
892 the steps above or check the system log (e.g. dmesg or journalctl) for
893 more information.
894 Notably, the swapon utility exits with a message that does not say what
896 failed:
897 # swapon /path/swapfile
899 swapon: /path/swapfile: swapon failed: Invalid argument
900 The specific reason is likely to be printed to the system log by the
902 btrfs module:
903 # journalctl -t kernel | grep swapfile
905 kernel: BTRFS warning (device sda): swapfile must have single data profile
906
908 Data and metadata are checksummed by default, the checksum is calcu‐
909 lated before write and verified after reading the blocks from devices.
910 The whole metadata block has a checksum stored inline in the b-tree
911 node header, each data block has a detached checksum stored in the
912 checksum tree.
913 There are several checksum algorithms supported. The default and back‐
915 ward compatible is crc32c. Since kernel 5.5 there are three more with
916 different characteristics and trade-offs regarding speed and strength.
917 The following list may help you to decide which one to select.
918 CRC32C (32bit digest)
920 default, best backward compatibility, very fast, modern CPUs
921 have instruction-level support, not collision-resistant but
922 still good error detection capabilities
923 XXHASH (64bit digest)
925 can be used as CRC32C successor, very fast, optimized for modern
926 CPUs utilizing instruction pipelining, good collision resistance
927 and error detection
928 SHA256 (256bit digest)
930 a cryptographic-strength hash, relatively slow but with possible
931 CPU instruction acceleration or specialized hardware cards, FIPS
932 certified and in wide use
933 BLAKE2b (256bit digest)
935 a cryptographic-strength hash, relatively fast with possible CPU
936 acceleration using SIMD extensions, not standardized but based
937 on BLAKE which was a SHA3 finalist, in wide use, the algorithm
938 used is BLAKE2b-256 that's optimized for 64bit platforms
939 The digest size affects overall size of data block checksums stored in
941 the filesystem. The metadata blocks have a fixed area up to 256 bits
942 (32 bytes), so there's no increase. Each data block has a separate
943 checksum stored, with additional overhead of the b-tree leaves.
944 Approximate relative performance of the algorithms, measured against
946 CRC32C using reference software implementations on a 3.5GHz intel CPU:
947 ┌────────┬─────────────┬───────┬─────────────────┐
949 │Digest │ Cycles/4KiB │ Ratio │ Implementation │
950 ├────────┼─────────────┼───────┼─────────────────┤
951 │CRC32C │ 1700 │ 1.00 │ CPU instruction │
952 ├────────┼─────────────┼───────┼─────────────────┤
953 │XXHASH │ 2500 │ 1.44 │ reference impl. │
954 ├────────┼─────────────┼───────┼─────────────────┤
955 │SHA256 │ 105000 │ 61 │ reference impl. │
956 ├────────┼─────────────┼───────┼─────────────────┤
957 │SHA256 │ 36000 │ 21 │ libgcrypt/AVX2 │
958 ├────────┼─────────────┼───────┼─────────────────┤
959 │SHA256 │ 63000 │ 37 │ libsodium/AVX2 │
960 ├────────┼─────────────┼───────┼─────────────────┤
961 │BLAKE2b │ 22000 │ 13 │ reference impl. │
962 ├────────┼─────────────┼───────┼─────────────────┤
963 │BLAKE2b │ 19000 │ 11 │ libgcrypt/AVX2 │
964 ├────────┼─────────────┼───────┼─────────────────┤
965 │BLAKE2b │ 19000 │ 11 │ libsodium/AVX2 │
966 └────────┴─────────────┴───────┴─────────────────┘
967 Many kernels are configured with SHA256 as built-in and not as a mod‐
969 ule. The accelerated versions are however provided by the modules and
970 must be loaded explicitly (modprobe sha256) before mounting the
971 filesystem to make use of them. You can check in
972 /sys/fs/btrfs/FSID/checksum which one is used. If you see
973 sha256-generic, then you may want to unmount and mount the filesystem
974 again, changing that on a mounted filesystem is not possible. Check
975 the file /proc/crypto, when the implementation is built-in, you'd find
976 name : sha256
978 driver : sha256-generic
979 module : kernel
980 priority : 100
981 ...
982 while accelerated implementation is e.g.
984 name : sha256
986 driver : sha256-avx2
987 module : sha256_ssse3
988 priority : 170
989 ...
990
992 Btrfs supports transparent file compression. There are three algorithms
993 available: ZLIB, LZO and ZSTD (since v4.14), with various levels. The
994 compression happens on the level of file extents and the algorithm is
995 selected by file property, mount option or by a defrag command. You
996 can have a single btrfs mount point that has some files that are uncom‐
997 pressed, some that are compressed with LZO, some with ZLIB, for in‐
998 stance (though you may not want it that way, it is supported).
999 Once the compression is set, all newly written data will be compressed,
1001 i.e. existing data are untouched. Data are split into smaller chunks
1002 (128KiB) before compression to make random rewrites possible without a
1003 high performance hit. Due to the increased number of extents the meta‐
1004 data consumption is higher. The chunks are compressed in parallel.
1005 The algorithms can be characterized as follows regarding the speed/ra‐
1007 tio trade-offs:
1008 ZLIB
1010 • slower, higher compression ratio
1012 • levels: 1 to 9, mapped directly, default level is 3
1014 • good backward compatibility
1016 LZO
1018 • faster compression and decompression than ZLIB, worse compres‐
1020 sion ratio, designed to be fast
1021 • no levels
1023 • good backward compatibility
1025 ZSTD
1027 • compression comparable to ZLIB with higher compression/decom‐
1029 pression speeds and different ratio
1030 • levels: 1 to 15, mapped directly (higher levels are not avail‐
1032 able)
1033 • since 4.14, levels since 5.1
1035 The differences depend on the actual data set and cannot be expressed
1037 by a single number or recommendation. Higher levels consume more CPU
1038 time and may not bring a significant improvement, lower levels are
1039 close to real time.
1040
1042 Typically the compression can be enabled on the whole filesystem, spec‐
1043 ified for the mount point. Note that the compression mount options are
1044 shared among all mounts of the same filesystem, either bind mounts or
1045 subvolume mounts. Please refer to btrfs(5) section MOUNT OPTIONS.
1046 $ mount -o compress=zstd /dev/sdx /mnt
1048 This will enable the zstd algorithm on the default level (which is 3).
1050 The level can be specified manually too like zstd:3. Higher levels com‐
1051 press better at the cost of time. This in turn may cause increased
1052 write latency, low levels are suitable for real-time compression and on
1053 reasonably fast CPU don't cause noticeable performance drops.
1054 $ btrfs filesystem defrag -czstd file
1056 The command above will start defragmentation of the whole file and ap‐
1058 ply the compression, regardless of the mount option. (Note: specifying
1059 level is not yet implemented). The compression algorithm is not persis‐
1060 tent and applies only to the defragmentation command, for any other
1061 writes other compression settings apply.
1062 Persistent settings on a per-file basis can be set in two ways:
1064 $ chattr +c file
1066 $ btrfs property set file compression zstd
1067 The first command is using legacy interface of file attributes inher‐
1069 ited from ext2 filesystem and is not flexible, so by default the zlib
1070 compression is set. The other command sets a property on the file with
1071 the given algorithm. (Note: setting level that way is not yet imple‐
1072 mented.)
1073
1075 The level support of ZLIB has been added in v4.14, LZO does not support
1076 levels (the kernel implementation provides only one), ZSTD level sup‐
1077 port has been added in v5.1.
1078 There are 9 levels of ZLIB supported (1 to 9), mapping 1:1 from the
1080 mount option to the algorithm defined level. The default is level 3,
1081 which provides the reasonably good compression ratio and is still rea‐
1082 sonably fast. The difference in compression gain of levels 7, 8 and 9
1083 is comparable but the higher levels take longer.
1084 The ZSTD support includes levels 1 to 15, a subset of full range of
1086 what ZSTD provides. Levels 1-3 are real-time, 4-8 slower with improved
1087 compression and 9-15 try even harder though the resulting size may not
1088 be significantly improved.
1089 Level 0 always maps to the default. The compression level does not af‐
1091 fect compatibility.
1092
1094 Files with already compressed data or with data that won't compress
1095 well with the CPU and memory constraints of the kernel implementations
1096 are using a simple decision logic. If the first portion of data being
1097 compressed is not smaller than the original, the compression of the
1098 file is disabled -- unless the filesystem is mounted with com‐
1099 press-force. In that case compression will always be attempted on the
1100 file only to be later discarded. This is not optimal and subject to op‐
1101 timizations and further development.
1102 If a file is identified as incompressible, a flag is set (NOCOMPRESS)
1104 and it's sticky. On that file compression won't be performed unless
1105 forced. The flag can be also set by chattr +m (since e2fsprogs 1.46.2)
1106 or by properties with value no or none. Empty value will reset it to
1107 the default that's currently applicable on the mounted filesystem.
1108 There are two ways to detect incompressible data:
1110 • actual compression attempt - data are compressed, if the result is
1112 not smaller, it's discarded, so this depends on the algorithm and
1113 level
1114 • pre-compression heuristics - a quick statistical evaluation on the
1116 data is performed and based on the result either compression is per‐
1117 formed or skipped, the NOCOMPRESS bit is not set just by the heuris‐
1118 tic, only if the compression algorithm does not make an improvement
1119 $ lsattr file
1121 ---------------------m file
1122 Using the forcing compression is not recommended, the heuristics are
1124 supposed to decide that and compression algorithms internally detect
1125 incompressible data too.
1126
1128 The heuristics aim to do a few quick statistical tests on the com‐
1129 pressed data in order to avoid probably costly compression that would
1130 turn out to be inefficient. Compression algorithms could have internal
1131 detection of incompressible data too but this leads to more overhead as
1132 the compression is done in another thread and has to write the data
1133 anyway. The heuristic is read-only and can utilize cached memory.
1134 The tests performed based on the following: data sampling, long re‐
1136 peated pattern detection, byte frequency, Shannon entropy.
1137
1139 Compression is done using the COW mechanism so it's incompatible with
1140 nodatacow. Direct IO works on compressed files but will fall back to
1141 buffered writes and leads to recompression. Currently nodatasum and
1142 compression don't work together.
1143 The compression algorithms have been added over time so the version
1145 compatibility should be also considered, together with other tools that
1146 may access the compressed data like bootloaders.
1147
1149 Btrfs has a sysfs interface to provide extra knobs.
1150 The top level path is /sys/fs/btrfs/, and the main directory layout is
1152 the following:
1153 ┌─────────────────────────────┬─────────────────────┬─────────┐
1155 │Relative Path │ Description │ Version │
1156 ├─────────────────────────────┼─────────────────────┼─────────┤
1157 │features/ │ All supported fea‐ │ 3.14+ │
1158 │ │ tures │ │
1159 ├─────────────────────────────┼─────────────────────┼─────────┤
1160 │<UUID>/ │ Mounted fs UUID │ 3.14+ │
1161 ├─────────────────────────────┼─────────────────────┼─────────┤
1162 │<UUID>/allocation/ │ Space allocation │ 3.14+ │
1163 │ │ info │ │
1164 ├─────────────────────────────┼─────────────────────┼─────────┤
1165 │<UUID>/features/ │ Features of the │ 3.14+ │
1166 │ │ filesystem │ │
1167 ├─────────────────────────────┼─────────────────────┼─────────┤
1168 │<UUID>/devices/<DE‐ │ Symlink to each │ 5.6+ │
1169 │VID>/ │ block device sysfs │ │
1170 ├─────────────────────────────┼─────────────────────┼─────────┤
1171 │<UUID>/devinfo/<DE‐ │ Btrfs specific info │ 5.6+ │
1172 │VID>/ │ for each device │ │
1173 ├─────────────────────────────┼─────────────────────┼─────────┤
1174 │<UUID>/qgroups/ │ Global qgroup info │ 5.9+ │
1175 └─────────────────────────────┴─────────────────────┴─────────┘
1176 │<UUID>/qgroups/<LEVEL>_<ID>/ │ Info for each │ 5.9+ │
1180 │ │ qgroup │ │
1181 ├─────────────────────────────┼─────────────────────┼─────────┤
1182 │<UUID>/discard/ │ Discard stats and │ 6.1+ │
1183 │ │ tunables │ │
1184 └─────────────────────────────┴─────────────────────┴─────────┘
1185 For /sys/fs/btrfs/features/ directory, each file means a supported fea‐
1187 ture for the current kernel.
1188 For /sys/fs/btrfs/<UUID>/features/ directory, each file means an en‐
1190 abled feature for the mounted filesystem.
1191 The features shares the same name in section FILESYSTEM FEATURES.
1193 Files in /sys/fs/btrfs/<UUID>/ directory are:
1195 bg_reclaim_threshold
1197 (RW, since: 5.19)
1198 Used space percentage of total device space to start auto block
1200 group claim. Mostly for zoned devices.
1201 checksum
1203 (RO, since: 5.5)
1204 The checksum used for the mounted filesystem. This includes
1206 both the checksum type (see section CHECKSUM ALGORITHMS) and the
1207 implemented driver (mostly shows if it's hardware accelerated).
1208 clone_alignment
1210 (RO, since: 3.16)
1211 The bytes alignment for clone and dedupe ioctls.
1213 commit_stats
1215 (RW, since: 6.0)
1216 The performance statistics for btrfs transaction commit. Mostly
1218 for debug purposes.
1219 Writing into this file will reset the maximum commit duration to
1221 the input value.
1222 exclusive_operation
1224 (RO, since: 5.10)
1225 Shows the running exclusive operation. Check section FILESYSTEM
1227 EXCLUSIVE OPERATIONS for details.
1228 generation
1230 (RO, since: 5.11)
1231 Show the generation of the mounted filesystem.
1233 label (RW, since: 3.14)
1235 Show the current label of the mounted filesystem.
1237 metadata_uuid
1239 (RO, since: 5.0)
1240 Shows the metadata uuid of the mounted filesystem. Check meta‐
1242 data_uuid feature for more details.
1243 nodesize
1245 (RO, since: 3.14)
1246 Show the nodesize of the mounted filesystem.
1248 quota_override
1250 (RW, since: 4.13)
1251 Shows the current quota override status. 0 means no quota over‐
1253 ride. 1 means quota override, quota can ignore the existing
1254 limit settings.
1255 read_policy
1257 (RW, since: 5.11)
1258 Shows the current balance policy for reads. Currently only
1260 "pid" (balance using pid value) is supported.
1261 sectorsize
1263 (RO, since: 3.14)
1264 Shows the sectorsize of the mounted filesystem.
1266 Files and directories in /sys/fs/btrfs/<UUID>/allocations directory
1268 are:
1269 global_rsv_reserved
1271 (RO, since: 3.14)
1272 The used bytes of the global reservation.
1274 global_rsv_size
1276 (RO, since: 3.14)
1277 The total size of the global reservation.
1279 data/, metadata/ and system/ directories
1281 (RO, since: 5.14)
1282 Space info accounting for the 3 chunk types. Mostly for debug
1284 purposes.
1285 Files in /sys/fs/btrfs/<UUID>/allocations/data,metadata,system direc‐
1287 tory are:
1288 bg_reclaim_threshold
1290 (RW, since: 5.19)
1291 Reclaimable space percentage of block group's size (excluding
1293 permanently unusable space) to reclaim the block group. Can be
1294 used on regular or zoned devices.
1295 chunk_size
1297 (RW, since: 6.0)
1298 Shows the chunk size. Can be changed for data and metadata.
1300 Cannot be set for zoned devices.
1301 Files in /sys/fs/btrfs/<UUID>/devinfo/<DEVID> directory are:
1303 error_stats:
1305 (RO, since: 5.14)
1306 Shows all the history error numbers of the device.
1308 fsid: (RO, since: 5.17)
1310 Shows the fsid which the device belongs to. It can be different
1312 than the <UUID> if it's a seed device.
1313 in_fs_metadata
1315 (RO, since: 5.6)
1316 Shows whether we have found the device. Should always be 1, as
1318 if this turns to 0, the <DEVID> directory would get removed au‐
1319 tomatically.
1320 missing
1322 (RO, since: 5.6)
1323 Shows whether the device is missing.
1325 replace_target
1327 (RO, since: 5.6)
1328 Shows whether the device is the replace target. If no dev-re‐
1330 place is running, this value should be 0.
1331 scrub_speed_max
1333 (RW, since: 5.14)
1334 Shows the scrub speed limit for this device. The unit is
1336 Bytes/s. 0 means no limit.
1337 writeable
1339 (RO, since: 5.6)
1340 Show if the device is writeable.
1342 Files in /sys/fs/btrfs/<UUID>/qgroups/ directory are:
1344 enabled
1346 (RO, since: 6.1)
1347 Shows if qgroup is enabled. Also, if qgroup is disabled, the
1349 qgroups directory would be removed automatically.
1350 inconsistent
1352 (RO, since: 6.1)
1353 Shows if the qgroup numbers are inconsistent. If 1, it's recom‐
1355 mended to do a qgroup rescan.
1356 drop_subtree_threshold
1358 (RW, since: 6.1)
1359 Shows the subtree drop threshold to automatically mark qgroup
1361 inconsistent.
1362 When dropping large subvolumes with qgroup enabled, there would
1364 be a huge load for qgroup accounting. If we have a subtree
1365 whose level is larger than or equal to this value, we will not
1366 trigger qgroup account at all, but mark qgroup inconsistent to
1367 avoid the huge workload.
1368 Default value is 8, where no subtree drop can trigger qgroup.
1370 Lower value can reduce qgroup workload, at the cost of extra
1372 qgroup rescan to re-calculate the numbers.
1373 Files in /sys/fs/btrfs/<UUID>/<LEVEL>_<ID>/ directory are:
1375 exclusive
1377 (RO, since: 5.9)
1378 Shows the exclusively owned bytes of the qgroup.
1380 limit_flags
1382 (RO, since: 5.9)
1383 Shows the numeric value of the limit flags. If 0, means no
1385 limit implied.
1386 max_exclusive
1388 (RO, since: 5.9)
1389 Shows the limits on exclusively owned bytes.
1391 max_referenced
1393 (RO, since: 5.9)
1394 Shows the limits on referenced bytes.
1396 referenced
1398 (RO, since: 5.9)
1399 Shows the referenced bytes of the qgroup.
1401 rsv_data
1403 (RO, since: 5.9)
1404 Shows the reserved bytes for data.
1406 rsv_meta_pertrans
1408 (RO, since: 5.9)
1409 Shows the reserved bytes for per transaction metadata.
1411 rsv_meta_prealloc
1413 (RO, since: 5.9)
1414 Shows the reserved bytes for preallocated metadata.
1416 Files in /sys/fs/btrfs/<UUID>/discard/ directory are:
1418 discardable_bytes
1420 (RO, since: 6.1)
1421 Shows amount of bytes that can be discarded in the async discard
1423 and nodiscard mode.
1424 discardable_extents
1426 (RO, since: 6.1)
1427 Shows number of extents to be discarded in the async discard and
1429 nodiscard mode.
1430 discard_bitmap_bytes
1432 (RO, since: 6.1)
1433 Shows amount of discarded bytes from data tracked as bitmaps.
1435 discard_extent_bytes
1437 (RO, since: 6.1)
1438 Shows amount of discarded extents from data tracked as bitmaps.
1440 discard_bytes_saved
1442 (RO, since: 6.1)
1443 Shows the amount of bytes that were reallocated without being
1445 discarded.
1446 kbps_limit
1448 (RW, since: 6.1)
1449 Tunable limit of kilobytes per second issued as discard IO in
1451 the async discard mode.
1452 iops_limit
1454 (RW, since: 6.1)
1455 Tunable limit of number of discard IO operations to be issued in
1457 the async discard mode.
1458 max_discard_size
1460 (RW, since: 6.1)
1461 Tunable limit for size of one IO discard request.
1463
1465 There are several operations that affect the whole filesystem and can‐
1466 not be run in parallel. Attempt to start one while another is running
1467 will fail (see exceptions below).
1468 Since kernel 5.10 the currently running operation can be obtained from
1470 /sys/fs/UUID/exclusive_operation with following values and operations:
1471 • balance
1473 • balance paused (since 5.17)
1475 • device add
1477 • device delete
1479 • device replace
1481 • resize
1483 • swapfile activate
1485 • none
1487 Enqueuing is supported for several btrfs subcommands so they can be
1489 started at once and then serialized.
1490 There's an exception when a paused balance allows to start a device add
1492 operation as they don't really collide and this can be used to add more
1493 space for the balance to finish.
1494
1496 maximum file name length
1497 255
1498 This limit is imposed by Linux VFS, the structures of BTRFS
1500 could store larger file names.
1501 maximum symlink target length
1503 depends on the nodesize value, for 4KiB it's 3949 bytes, for
1504 larger nodesize it's 4095 due to the system limit PATH_MAX
1505 The symlink target may not be a valid path, i.e. the path name
1507 components can exceed the limits (NAME_MAX), there's no content
1508 validation at symlink(3) creation.
1509 maximum number of inodes
1511 264 but depends on the available metadata space as the inodes
1512 are created dynamically
1513 Each subvolume is an independent namespace of inodes and thus
1515 their numbers, so the limit is per subvolume, not for the whole
1516 filesystem.
1517 inode numbers
1519 minimum number: 256 (for subvolumes), regular files and directo‐
1520 ries: 257, maximum number: (264 - 256)
1521 The inode numbers that can be assigned to user created files are
1523 from the whole 64bit space except first 256 and last 256 in that
1524 range that are reserved for internal b-tree identifiers.
1525 maximum file length
1527 inherent limit of BTRFS is 264 (16 EiB) but the practical limit
1528 of Linux VFS is 263 (8 EiB)
1529 maximum number of subvolumes
1531 the subvolume ids can go up to 248 but the number of actual sub‐
1532 volumes depends on the available metadata space
1533 The space consumed by all subvolume metadata includes bookkeep‐
1535 ing of shared extents can be large (MiB, GiB). The range is not
1536 the full 64bit range because of qgroups that use the upper 16
1537 bits for another purposes.
1538 maximum number of hardlinks of a file in a directory
1540 65536 when the extref feature is turned on during mkfs (de‐
1541 fault), roughly 100 otherwise and depends on file name length
1542 that fits into one metadata node
1543 minimum filesystem size
1545 the minimal size of each device depends on the mixed-bg feature,
1546 without that (the default) it's about 109MiB, with mixed-bg it's
1547 is 16MiB
1548
1550 GRUB2 (https://www.gnu.org/software/grub) has the most advanced support
1551 of booting from BTRFS with respect to features.
1552 U-Boot (https://www.denx.de/wiki/U-Boot/) has decent support for boot‐
1554 ing but not all BTRFS features are implemented, check the documenta‐
1555 tion.
1556 In general, the first 1MiB on each device is unused with the exception
1558 of primary superblock that is on the offset 64KiB and spans 4KiB. The
1559 rest can be freely used by bootloaders or for other system information.
1560 Note that booting from a filesystem on zoned device is not supported.
1561
1563 The btrfs filesystem supports setting file attributes or flags. Note
1564 there are old and new interfaces, with confusing names. The following
1565 list should clarify that:
1566 • attributes: chattr(1) or lsattr(1) utilities (the ioctls are
1568 FS_IOC_GETFLAGS and FS_IOC_SETFLAGS), due to the ioctl names the at‐
1569 tributes are also called flags
1570 • xflags: to distinguish from the previous, it's extended flags, with
1572 tunable bits similar to the attributes but extensible and new bits
1573 will be added in the future (the ioctls are FS_IOC_FSGETXATTR and
1574 FS_IOC_FSSETXATTR but they are not related to extended attributes
1575 that are also called xattrs), there's no standard tool to change the
1576 bits, there's support in xfs_io(8) as command xfs_io -c chattr
1577 Attributes
1579 a append only, new writes are always written at the end of the
1580 file
1581 A no atime updates
1583 c compress data, all data written after this attribute is set will
1585 be compressed. Please note that compression is also affected by
1586 the mount options or the parent directory attributes.
1587 When set on a directory, all newly created files will inherit
1589 this attribute. This attribute cannot be set with 'm' at the
1590 same time.
1591 C no copy-on-write, file data modifications are done in-place
1593 When set on a directory, all newly created files will inherit
1595 this attribute.
1596 NOTE:
1598 Due to implementation limitations, this flag can be set/unset
1599 only on empty files.
1600 d no dump, makes sense with 3rd party tools like dump(8), on BTRFS
1602 the attribute can be set/unset but no other special handling is
1603 done
1604 D synchronous directory updates, for more details search open(2)
1606 for O_SYNC and O_DSYNC
1607 i immutable, no file data and metadata changes allowed even to the
1609 root user as long as this attribute is set (obviously the excep‐
1610 tion is unsetting the attribute)
1611 m no compression, permanently turn off compression on the given
1613 file. Any compression mount options will not affect this file.
1614 (chattr support added in 1.46.2)
1615 When set on a directory, all newly created files will inherit
1617 this attribute. This attribute cannot be set with c at the same
1618 time.
1619 S synchronous updates, for more details search open(2) for O_SYNC
1621 and O_DSYNC
1622 No other attributes are supported. For the complete list please refer
1624 to the chattr(1) manual page.
1625 XFLAGS
1627 There's an overlap of letters assigned to the bits with the attributes,
1628 this list refers to what xfs_io(8) provides:
1629 i immutable, same as the attribute
1631 a append only, same as the attribute
1633 s synchronous updates, same as the attribute S
1635 A no atime updates, same as the attribute
1637 d no dump, same as the attribute
1639
1641 Since version 5.12 btrfs supports so called zoned mode. This is a spe‐
1642 cial on-disk format and allocation/write strategy that's friendly to
1643 zoned devices. In short, a device is partitioned into fixed-size zones
1644 and each zone can be updated by append-only manner, or reset. As btrfs
1645 has no fixed data structures, except the super blocks, the zoned mode
1646 only requires block placement that follows the device constraints. You
1647 can learn about the whole architecture at https://zonedstorage.io .
1648 The devices are also called SMR/ZBC/ZNS, in host-managed mode. Note
1650 that there are devices that appear as non-zoned but actually are, this
1651 is drive-managed and using zoned mode won't help.
1652 The zone size depends on the device, typical sizes are 256MiB or 1GiB.
1654 In general it must be a power of two. Emulated zoned devices like
1655 null_blk allow to set various zone sizes.
1656 Requirements, limitations
1658 • all devices must have the same zone size
1659 • maximum zone size is 8GiB
1661 • minimum zone size is 4MiB
1663 • mixing zoned and non-zoned devices is possible, the zone writes are
1665 emulated, but this is namely for testing
1666 • the super block is handled in a special way and is at different loca‐
1668 tions than on a non-zoned filesystem:
1669 • primary: 0B (and the next two zones)
1671 • secondary: 512GiB (and the next two zones)
1673 • tertiary: 4TiB (4096GiB, and the next two zones)
1675 Incompatible features
1677 The main constraint of the zoned devices is lack of in-place update of
1678 the data. This is inherently incompatible with some features:
1679 • NODATACOW - overwrite in-place, cannot create such files
1681 • fallocate - preallocating space for in-place first write
1683 • mixed-bg - unordered writes to data and metadata, fixing that means
1685 using separate data and metadata block groups
1686 • booting - the zone at offset 0 contains superblock, resetting the
1688 zone would destroy the bootloader data
1689 Initial support lacks some features but they're planned:
1691 • only single (data, metadata) and DUP (metadata) profile is supported
1693 • fstrim - due to dependency on free space cache v1
1695 Super block
1697 As said above, super block is handled in a special way. In order to be
1698 crash safe, at least one zone in a known location must contain a valid
1699 superblock. This is implemented as a ring buffer in two consecutive
1700 zones, starting from known offsets 0B, 512GiB and 4TiB.
1701 The values are different than on non-zoned devices. Each new super
1703 block is appended to the end of the zone, once it's filled, the zone is
1704 reset and writes continue to the next one. Looking up the latest super
1705 block needs to read offsets of both zones and determine the last writ‐
1706 ten version.
1707 The amount of space reserved for super block depends on the zone size.
1709 The secondary and tertiary copies are at distant offsets as the capac‐
1710 ity of the devices is expected to be large, tens of terabytes. Maximum
1711 zone size supported is 8GiB, which would mean that e.g. offset 0-16GiB
1712 would be reserved just for the super block on a hypothetical device of
1713 that zone size. This is wasteful but required to guarantee crash
1714 safety.
1715 Devices
1717 Real hardware
1718 The WD Ultrastar series 600 advertises HM-SMR, i.e. the host-managed
1719 zoned mode. There are two more: DA (device managed, no zoned informa‐
1720 tion exported to the system), HA (host aware, can be used as regular
1721 disk but zoned writes improve performance). There are not many devices
1722 available at the moment, the information about exact zoned mode is hard
1723 to find, check data sheets or community sources gathering information
1724 from real devices.
1725 Note: zoned mode won't work with DM-SMR disks.
1727 • Ultrastar® DC ZN540 NVMe ZNS SSD (product brief)
1729 Emulated: null_blk
1731 The driver null_blk provides memory backed device and is suitable for
1732 testing. There are some quirks setting up the devices. The module must
1733 be loaded with nr_devices=0 or the numbering of device nodes will be
1734 offset. The configfs must be mounted at /sys/kernel/config and the ad‐
1735 ministration of the null_blk devices is done in /sys/kernel/con‐
1736 fig/nullb. The device nodes are named like /dev/nullb0 and are numbered
1737 sequentially. NOTE: the device name may be different than the named di‐
1738 rectory in sysfs!
1739 Setup:
1741 modprobe configfs
1743 modprobe null_blk nr_devices=0
1744 Create a device mydev, assuming no other previously created devices,
1746 size is 2048MiB, zone size 256MiB. There are more tunable parameters,
1747 this is a minimal example taking defaults:
1748 cd /sys/kernel/config/nullb/
1750 mkdir mydev
1751 cd mydev
1752 echo 2048 > size
1753 echo 1 > zoned
1754 echo 1 > memory_backed
1755 echo 256 > zone_size
1756 echo 1 > power
1757 This will create a device /dev/nullb0 and the value of file index will
1759 match the ending number of the device node.
1760 Remove the device:
1762 rmdir /sys/kernel/config/nullb/mydev
1764 Then continue with mkfs.btrfs /dev/nullb0, the zoned mode is auto-de‐
1766 tected.
1767 For convenience, there's a script wrapping the basic null_blk manage‐
1769 ment operations https://github.com/kdave/nullb.git, the above commands
1770 become:
1771 nullb setup
1773 nullb create -s 2g -z 256
1774 mkfs.btrfs /dev/nullb0
1775 ...
1776 nullb rm nullb0
1777 Emulated: TCMU runner
1779 TCMU is a framework to emulate SCSI devices in userspace, providing
1780 various backends for the storage, with zoned support as well. A
1781 file-backed zoned device can provide more options for larger storage
1782 and zone size. Please follow the instructions at
1783 https://zonedstorage.io/projects/tcmu-runner/ .
1784 Compatibility, incompatibility
1786 • the feature sets an incompat bit and requires new kernel to access
1787 the filesystem (for both read and write)
1788 • superblock needs to be handled in a special way, there are still 3
1790 copies but at different offsets (0, 512GiB, 4TiB) and the 2 consecu‐
1791 tive zones are a ring buffer of the superblocks, finding the latest
1792 one needs reading it from the write pointer or do a full scan of the
1793 zones
1794 • mixing zoned and non zoned devices is possible (zones are emulated)
1796 but is recommended only for testing
1797 • mixing zoned devices with different zone sizes is not possible
1799 • zone sizes must be power of two, zone sizes of real devices are e.g.
1801 256MiB or 1GiB, larger size is expected, maximum zone size supported
1802 by btrfs is 8GiB
1803 Status, stability, reporting bugs
1805 The zoned mode has been released in 5.12 and there are still some rough
1806 edges and corner cases one can hit during testing. Please report bugs
1807 to https://github.com/naota/linux/issues/ .
1808 References
1810 • https://zonedstorage.io
1811 • https://zonedstorage.io/projects/libzbc/ -- libzbc is library and
1813 set of tools to directly manipulate devices with ZBC/ZAC support
1814 • https://zonedstorage.io/projects/libzbd/ -- libzbd uses the kernel
1816 provided zoned block device interface based on the ioctl() system
1817 calls
1818 • https://hddscan.com/blog/2020/hdd-wd-smr.html -- some details about
1820 exact device types
1821 • https://lwn.net/Articles/853308/ -- Btrfs on zoned block devices
1823 • https://www.usenix.org/conference/vault20/presentation/bjorling --
1825 Zone Append: A New Way of Writing to Zoned Storage
1826
1828 There's a character special device /dev/btrfs-control with major and
1829 minor numbers 10 and 234 (the device can be found under the misc cate‐
1830 gory).
1831 $ ls -l /dev/btrfs-control
1833 crw------- 1 root root 10, 234 Jan 1 12:00 /dev/btrfs-control
1834 The device accepts some ioctl calls that can perform following actions
1836 on the filesystem module:
1837 • scan devices for btrfs filesystem (i.e. to let multi-device filesys‐
1839 tems mount automatically) and register them with the kernel module
1840 • similar to scan, but also wait until the device scanning process is
1842 finished for a given filesystem
1843 • get the supported features (can be also found under
1845 /sys/fs/btrfs/features)
1846 The device is created when btrfs is initialized, either as a module or
1848 a built-in functionality and makes sense only in connection with that.
1849 Running e.g. mkfs without the module loaded will not register the de‐
1850 vice and will probably warn about that.
1851 In rare cases when the module is loaded but the device is not present
1853 (most likely accidentally deleted), it's possible to recreate it by
1854 # mknod --mode=600 /dev/btrfs-control c 10 234
1856 or (since 5.11) by a convenience command
1858 # btrfs rescue create-control-device
1860 The control device is not strictly required but the device scanning
1862 will not work and a workaround would need to be used to mount a
1863 multi-device filesystem. The mount option device can trigger the de‐
1864 vice scanning during mount, see also btrfs device scan.
1865
1867 It is possible that a btrfs filesystem contains multiple block group
1868 profiles of the same type. This could happen when a profile conversion
1869 using balance filters is interrupted (see btrfs-balance(8)). Some
1870 btrfs commands perform a test to detect this kind of condition and
1871 print a warning like this:
1872 WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
1874 WARNING: Data: single, raid1
1875 WARNING: Metadata: single, raid1
1876 The corresponding output of btrfs filesystem df might look like:
1878 WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
1880 WARNING: Data: single, raid1
1881 WARNING: Metadata: single, raid1
1882 Data, RAID1: total=832.00MiB, used=0.00B
1883 Data, single: total=1.63GiB, used=0.00B
1884 System, single: total=4.00MiB, used=16.00KiB
1885 Metadata, single: total=8.00MiB, used=112.00KiB
1886 Metadata, RAID1: total=64.00MiB, used=32.00KiB
1887 GlobalReserve, single: total=16.25MiB, used=0.00B
1888 There's more than one line for type Data and Metadata, while the pro‐
1890 files are single and RAID1.
1891 This state of the filesystem OK but most likely needs the user/adminis‐
1893 trator to take an action and finish the interrupted tasks. This cannot
1894 be easily done automatically, also the user knows the expected final
1895 profiles.
1896 In the example above, the filesystem started as a single device and
1898 single block group profile. Then another device was added, followed by
1899 balance with convert=raid1 but for some reason hasn't finished.
1900 Restarting the balance with convert=raid1 will continue and end up with
1901 filesystem with all block group profiles RAID1.
1902 NOTE:
1904 If you're familiar with balance filters, you can use con‐
1905 vert=raid1,profiles=single,soft, which will take only the uncon‐
1906 verted single profiles and convert them to raid1. This may speed up
1907 the conversion as it would not try to rewrite the already convert
1908 raid1 profiles.
1909 Having just one profile is desired as this also clearly defines the
1911 profile of newly allocated block groups, otherwise this depends on in‐
1912 ternal allocation policy. When there are multiple profiles present, the
1913 order of selection is RAID56, RAID10, RAID1, RAID0 as long as the de‐
1914 vice number constraints are satisfied.
1915 Commands that print the warning were chosen so they're brought to user
1917 attention when the filesystem state is being changed in that regard.
1918 This is: device add, device delete, balance cancel, balance pause. Com‐
1919 mands that report space usage: filesystem df, device usage. The command
1920 filesystem usage provides a line in the overall summary:
1921 Multiple profiles: yes (data, metadata)
1923
1925 The COW mechanism and multiple devices under one hood enable an inter‐
1926 esting concept, called a seeding device: extending a read-only filesys‐
1927 tem on a device with another device that captures all writes. For exam‐
1928 ple imagine an immutable golden image of an operating system enhanced
1929 with another device that allows to use the data from the golden image
1930 and normal operation. This idea originated on CD-ROMs with base OS and
1931 allowing to use them for live systems, but this became obsolete. There
1932 are technologies providing similar functionality, like unionmount,
1933 overlayfs or qcow2 image snapshot.
1934 The seeding device starts as a normal filesystem, once the contents is
1936 ready, btrfstune -S 1 is used to flag it as a seeding device. Mounting
1937 such device will not allow any writes, except adding a new device by
1938 btrfs device add. Then the filesystem can be remounted as read-write.
1939 Given that the filesystem on the seeding device is always recognized as
1941 read-only, it can be used to seed multiple filesystems from one device
1942 at the same time. The UUID that is normally attached to a device is au‐
1943 tomatically changed to a random UUID on each mount.
1944 Once the seeding device is mounted, it needs the writable device. After
1946 adding it, unmounting and mounting with umount /path; mount
1947 /dev/writable /path or remounting read-write with remount -o remount,rw
1948 makes the filesystem at /path ready for use.
1949 NOTE:
1951 There is a known bug with using remount to make the mount writeable:
1952 remount will leave the filesystem in a state where it is unable to
1953 clean deleted snapshots, so it will leak space until it is unmounted
1954 and mounted properly.
1955 Furthermore, deleting the seeding device from the filesystem can turn
1957 it into a normal filesystem, provided that the writable device can also
1958 contain all the data from the seeding device.
1959 The seeding device flag can be cleared again by btrfstune -f -S 0, e.g.
1961 allowing to update with newer data but please note that this will in‐
1962 validate all existing filesystems that use this particular seeding de‐
1963 vice. This works for some use cases, not for others, and the forcing
1964 flag to the command is mandatory to avoid accidental mistakes.
1965 Example how to create and use one seeding device:
1967 # mkfs.btrfs /dev/sda
1969 # mount /dev/sda /mnt/mnt1
1970 ... fill mnt1 with data
1971 # umount /mnt/mnt1
1972 # btrfstune -S 1 /dev/sda
1974 # mount /dev/sda /mnt/mnt1
1976 # btrfs device add /dev/sdb /mnt/mnt1
1977 # umount /mnt/mnt1
1978 # mount /dev/sdb /mnt/mnt1
1979 ... /mnt/mnt1 is now writable
1980 Now /mnt/mnt1 can be used normally. The device /dev/sda can be mounted
1982 again with a another writable device:
1983 # mount /dev/sda /mnt/mnt2
1985 # btrfs device add /dev/sdc /mnt/mnt2
1986 # umount /mnt/mnt2
1987 # mount /dev/sdc /mnt/mnt2
1988 ... /mnt/mnt2 is now writable
1989 The writable device (file:/dev/sdb) can be decoupled from the seeding
1991 device and used independently:
1992 # btrfs device delete /dev/sda /mnt/mnt1
1994 As the contents originated in the seeding device, it's possible to turn
1996 /dev/sdb to a seeding device again and repeat the whole process.
1997 A few things to note:
1999 • it's recommended to use only single device for the seeding device, it
2001 works for multiple devices but the single profile must be used in or‐
2002 der to make the seeding device deletion work
2003 • block group profiles single and dup support the use cases above
2005 • the label is copied from the seeding device and can be changed by
2007 btrfs filesystem label
2008 • each new mount of the seeding device gets a new random UUID
2010 • umount /path; mount /dev/writable /path can be replaced with mount -o
2012 remount,rw /path but it won't reclaim space of deleted subvolumes un‐
2013 til the seeding device is mounted read-write again before making it
2014 seeding again
2015 Chained seeding devices
2017 Though it's not recommended and is rather an obscure and untested use
2018 case, chaining seeding devices is possible. In the first example, the
2019 writable device /dev/sdb can be turned onto another seeding device
2020 again, depending on the unchanged seeding device /dev/sda. Then using
2021 /dev/sdb as the primary seeding device it can be extended with another
2022 writable device, say /dev/sdd, and it continues as before as a simple
2023 tree structure on devices.
2024 # mkfs.btrfs /dev/sda
2026 # mount /dev/sda /mnt/mnt1
2027 ... fill mnt1 with data
2028 # umount /mnt/mnt1
2029 # btrfstune -S 1 /dev/sda
2031 # mount /dev/sda /mnt/mnt1
2033 # btrfs device add /dev/sdb /mnt/mnt1
2034 # mount -o remount,rw /mnt/mnt1
2035 ... /mnt/mnt1 is now writable
2036 # umount /mnt/mnt1
2037 # btrfstune -S 1 /dev/sdb
2039 # mount /dev/sdb /mnt/mnt1
2041 # btrfs device add /dev/sdc /mnt
2042 # mount -o remount,rw /mnt/mnt1
2043 ... /mnt/mnt1 is now writable
2044 # umount /mnt/mnt1
2045 As a result we have:
2047 • sda is a single seeding device, with its initial contents
2049 • sdb is a seeding device but requires sda, the contents are from the
2051 time when sdb is made seeding, i.e. contents of sda with any later
2052 changes
2053 • sdc last writable, can be made a seeding one the same way as was sdb,
2055 preserving its contents and depending on sda and sdb
2056 As long as the seeding devices are unmodified and available, they can
2058 be used to start another branch.
2059
2061 The RAID56 feature provides striping and parity over several devices,
2062 same as the traditional RAID5/6. There are some implementation and de‐
2063 sign deficiencies that make it unreliable for some corner cases and the
2064 feature should not be used in production, only for evaluation or test‐
2065 ing. The power failure safety for metadata with RAID56 is not 100%.
2066 Metadata
2068 Do not use raid5 nor raid6 for metadata. Use raid1 or raid1c3 respec‐
2069 tively.
2070 The substitute profiles provide the same guarantees against loss of 1
2072 or 2 devices, and in some respect can be an improvement. Recovering
2073 from one missing device will only need to access the remaining 1st or
2074 2nd copy, that in general may be stored on some other devices due to
2075 the way RAID1 works on btrfs, unlike on a striped profile (similar to
2076 raid0) that would need all devices all the time.
2077 The space allocation pattern and consumption is different (e.g. on N
2079 devices): for raid5 as an example, a 1GiB chunk is reserved on each de‐
2080 vice, while with raid1 there's each 1GiB chunk stored on 2 devices. The
2081 consumption of each 1GiB of used metadata is then N * 1GiB for vs 2 *
2082 1GiB. Using raid1 is also more convenient for balancing/converting to
2083 other profile due to lower requirement on the available chunk space.
2084 Missing/incomplete support
2086 When RAID56 is on the same filesystem with different raid profiles, the
2087 space reporting is inaccurate, e.g. df, btrfs filesystem df or btrfs
2088 filesystem usage. When there's only a one profile per block group type
2089 (e.g. RAID5 for data) the reporting is accurate.
2090 When scrub is started on a RAID56 filesystem, it's started on all de‐
2092 vices that degrade the performance. The workaround is to start it on
2093 each device separately. Due to that the device stats may not match the
2094 actual state and some errors might get reported multiple times.
2095 The write hole problem. An unclean shutdown could leave a partially
2097 written stripe in a state where the some stripe ranges and the parity
2098 are from the old writes and some are new. The information which is
2099 which is not tracked. Write journal is not implemented. Alternatively a
2100 full read-modify-write would make sure that a full stripe is always
2101 written, avoiding the write hole completely, but performance in that
2102 case turned out to be too bad for use.
2103 The striping happens on all available devices (at the time the chunks
2105 were allocated), so in case a new device is added it may not be uti‐
2106 lized immediately and would require a rebalance. A fixed configured
2107 stripe width is not implemented.
2108
2110 Storage model
2111 A storage model is a model that captures key physical aspects of data
2112 structure in a data store. A filesystem is the logical structure orga‐
2113 nizing data on top of the storage device.
2114 The filesystem assumes several features or limitations of the storage
2116 device and utilizes them or applies measures to guarantee reliability.
2117 BTRFS in particular is based on a COW (copy on write) mode of writing,
2118 i.e. not updating data in place but rather writing a new copy to a dif‐
2119 ferent location and then atomically switching the pointers.
2120 In an ideal world, the device does what it promises. The filesystem as‐
2122 sumes that this may not be true so additional mechanisms are applied to
2123 either detect misbehaving hardware or get valid data by other means.
2124 The devices may (and do) apply their own detection and repair mecha‐
2125 nisms but we won't assume any.
2126 The following assumptions about storage devices are considered (sorted
2128 by importance, numbers are for further reference):
2129 1. atomicity of reads and writes of blocks/sectors (the smallest unit
2131 of data the device presents to the upper layers)
2132 2. there's a flush command that instructs the device to forcibly order
2134 writes before and after the command; alternatively there's a barrier
2135 command that facilitates the ordering but may not flush the data
2136 3. data sent to write to a given device offset will be written without
2138 further changes to the data and to the offset
2139 4. writes can be reordered by the device, unless explicitly serialized
2141 by the flush command
2142 5. reads and writes can be freely reordered and interleaved
2144 The consistency model of BTRFS builds on these assumptions. The logical
2146 data updates are grouped, into a generation, written on the device, se‐
2147 rialized by the flush command and then the super block is written end‐
2148 ing the generation. All logical links among metadata comprising a con‐
2149 sistent view of the data may not cross the generation boundary.
2150 When things go wrong
2152 No or partial atomicity of block reads/writes (1)
2153 • Problem: a partial block contents is written (torn write), e.g. due
2155 to a power glitch or other electronics failure during the read/write
2156 • Detection: checksum mismatch on read
2158 • Repair: use another copy or rebuild from multiple blocks using some
2160 encoding scheme
2161 The flush command does not flush (2)
2163 This is perhaps the most serious problem and impossible to mitigate by
2165 filesystem without limitations and design restrictions. What could hap‐
2166 pen in the worst case is that writes from one generation bleed to an‐
2167 other one, while still letting the filesystem consider the generations
2168 isolated. Crash at any point would leave data on the device in an in‐
2169 consistent state without any hint what exactly got written, what is
2170 missing and leading to stale metadata link information.
2171 Devices usually honor the flush command, but for performance reasons
2173 may do internal caching, where the flushed data are not yet persis‐
2174 tently stored. A power failure could lead to a similar scenario as
2175 above, although it's less likely that later writes would be written be‐
2176 fore the cached ones. This is beyond what a filesystem can take into
2177 account. Devices or controllers are usually equipped with batteries or
2178 capacitors to write the cache contents even after power is cut. (Bat‐
2179 tery backed write cache)
2180 Data get silently changed on write (3)
2182 Such thing should not happen frequently, but still can happen spuri‐
2184 ously due the complex internal workings of devices or physical effects
2185 of the storage media itself.
2186 • Problem: while the data are written atomically, the contents get
2188 changed
2189 • Detection: checksum mismatch on read
2191 • Repair: use another copy or rebuild from multiple blocks using some
2193 encoding scheme
2194 Data get silently written to another offset (3)
2196 This would be another serious problem as the filesystem has no informa‐
2198 tion when it happens. For that reason the measures have to be done
2199 ahead of time. This problem is also commonly called ghost write.
2200 The metadata blocks have the checksum embedded in the blocks, so a cor‐
2202 rect atomic write would not corrupt the checksum. It's likely that af‐
2203 ter reading such block the data inside would not be consistent with the
2204 rest. To rule that out there's embedded block number in the metadata
2205 block. It's the logical block number because this is what the logical
2206 structure expects and verifies.
2207 The following is based on information publicly available, user feed‐
2209 back, community discussions or bug report analyses. It's not complete
2210 and further research is encouraged when in doubt.
2211 Main memory
2213 The data structures and raw data blocks are temporarily stored in com‐
2214 puter memory before they get written to the device. It is critical that
2215 memory is reliable because even simple bit flips can have vast conse‐
2216 quences and lead to damaged structures, not only in the filesystem but
2217 in the whole operating system.
2218 Based on experience in the community, memory bit flips are more common
2220 than one would think. When it happens, it's reported by the
2221 tree-checker or by a checksum mismatch after reading blocks. There are
2222 some very obvious instances of bit flips that happen, e.g. in an or‐
2223 dered sequence of keys in metadata blocks. We can easily infer from the
2224 other data what values get damaged and how. However, fixing that is not
2225 straightforward and would require cross-referencing data from the en‐
2226 tire filesystem to see the scope.
2227 If available, ECC memory should lower the chances of bit flips, but
2229 this type of memory is not available in all cases. A memory test should
2230 be performed in case there's a visible bit flip pattern, though this
2231 may not detect a faulty memory module because the actual load of the
2232 system could be the factor making the problems appear. In recent years
2233 attacks on how the memory modules operate have been demonstrated
2234 (rowhammer) achieving specific bits to be flipped. While these were
2235 targeted, this shows that a series of reads or writes can affect unre‐
2236 lated parts of memory.
2237 Further reading:
2239 • https://en.wikipedia.org/wiki/Row_hammer
2241 What to do:
2243 • run memtest, note that sometimes memory errors happen only when the
2245 system is under heavy load that the default memtest cannot trigger
2246 • memory errors may appear as filesystem going read-only due to "pre
2248 write" check, that verify meta data before they get written but fail
2249 some basic consistency checks
2250 Direct memory access (DMA)
2252 Another class of errors is related to DMA (direct memory access) per‐
2253 formed by device drivers. While this could be considered a software er‐
2254 ror, the data transfers that happen without CPU assistance may acciden‐
2255 tally corrupt other pages. Storage devices utilize DMA for performance
2256 reasons, the filesystem structures and data pages are passed back and
2257 forth, making errors possible in case page life time is not properly
2258 tracked.
2259 There are lots of quirks (device-specific workarounds) in Linux kernel
2261 drivers (regarding not only DMA) that are added when found. The quirks
2262 may avoid specific errors or disable some features to avoid worse prob‐
2263 lems.
2264 What to do:
2266 • use up-to-date kernel (recent releases or maintained long term sup‐
2268 port versions)
2269 • as this may be caused by faulty drivers, keep the systems up-to-date
2271 Rotational disks (HDD)
2273 Rotational HDDs typically fail at the level of individual sectors or
2274 small clusters. Read failures are caught on the levels below the
2275 filesystem and are returned to the user as EIO - Input/output error.
2276 Reading the blocks repeatedly may return the data eventually, but this
2277 is better done by specialized tools and filesystem takes the result of
2278 the lower layers. Rewriting the sectors may trigger internal remapping
2279 but this inevitably leads to data loss.
2280 Disk firmware is technically software but from the filesystem perspec‐
2282 tive is part of the hardware. IO requests are processed, and caching or
2283 various other optimizations are performed, which may lead to bugs under
2284 high load or unexpected physical conditions or unsupported use cases.
2285 Disks are connected by cables with two ends, both of which can cause
2287 problems when not attached properly. Data transfers are protected by
2288 checksums and the lower layers try hard to transfer the data correctly
2289 or not at all. The errors from badly-connecting cables may manifest as
2290 large amount of failed read or write requests, or as short error bursts
2291 depending on physical conditions.
2292 What to do:
2294 • check smartctl for potential issues
2296 Solid state drives (SSD)
2298 The mechanism of information storage is different from HDDs and this
2299 affects the failure mode as well. The data are stored in cells grouped
2300 in large blocks with limited number of resets and other write con‐
2301 straints. The firmware tries to avoid unnecessary resets and performs
2302 optimizations to maximize the storage media lifetime. The known tech‐
2303 niques are deduplication (blocks with same fingerprint/hash are mapped
2304 to same physical block), compression or internal remapping and garbage
2305 collection of used memory cells. Due to the additional processing there
2306 are measures to verity the data e.g. by ECC codes.
2307 The observations of failing SSDs show that the whole electronic fails
2309 at once or affects a lot of data (e.g. stored on one chip). Recovering
2310 such data may need specialized equipment and reading data repeatedly
2311 does not help as it's possible with HDDs.
2312 There are several technologies of the memory cells with different char‐
2314 acteristics and price. The lifetime is directly affected by the type
2315 and frequency of data written. Writing "too much" distinct data (e.g.
2316 encrypted) may render the internal deduplication ineffective and lead
2317 to a lot of rewrites and increased wear of the memory cells.
2318 There are several technologies and manufacturers so it's hard to de‐
2320 scribe them but there are some that exhibit similar behaviour:
2321 • expensive SSD will use more durable memory cells and is optimized for
2323 reliability and high load
2324 • cheap SSD is projected for a lower load ("desktop user") and is opti‐
2326 mized for cost, it may employ the optimizations and/or extended error
2327 reporting partially or not at all
2328 It's not possible to reliably determine the expected lifetime of an SSD
2330 due to lack of information about how it works or due to lack of reli‐
2331 able stats provided by the device.
2332 Metadata writes tend to be the biggest component of lifetime writes to
2334 a SSD, so there is some value in reducing them. Depending on the device
2335 class (high end/low end) the features like DUP block group profiles may
2336 affect the reliability in both ways:
2337 • high end are typically more reliable and using single for data and
2339 metadata could be suitable to reduce device wear
2340 • low end could lack ability to identify errors so an additional redun‐
2342 dancy at the filesystem level (checksums, DUP) could help
2343 Only users who consume 50 to 100% of the SSD's actual lifetime writes
2345 need to be concerned by the write amplification of btrfs DUP metadata.
2346 Most users will be far below 50% of the actual lifetime, or will write
2347 the drive to death and discover how many writes 100% of the actual
2348 lifetime was. SSD firmware often adds its own write multipliers that
2349 can be arbitrary and unpredictable and dependent on application behav‐
2350 ior, and these will typically have far greater effect on SSD lifespan
2351 than DUP metadata. It's more or less impossible to predict when a SSD
2352 will run out of lifetime writes to within a factor of two, so it's hard
2353 to justify wear reduction as a benefit.
2354 Further reading:
2356 • https://www.snia.org/educational-library/ssd-and-deduplication-end-spinning-disk-2012
2358 • https://www.snia.org/educational-library/realities-solid-state-storage-2013-2013
2360 • https://www.snia.org/educational-library/ssd-performance-primer-2013
2362 • https://www.snia.org/educational-library/how-controllers-maximize-ssd-life-2013
2364 What to do:
2366 • run smartctl or self-tests to look for potential issues
2368 • keep the firmware up-to-date
2370 NVM express, non-volatile memory (NVMe)
2372 NVMe is a type of persistent memory usually connected over a system bus
2373 (PCIe) or similar interface and the speeds are an order of magnitude
2374 faster than SSD. It is also a non-rotating type of storage, and is not
2375 typically connected by a cable. It's not a SCSI type device either but
2376 rather a complete specification for logical device interface.
2377 In a way the errors could be compared to a combination of SSD class and
2379 regular memory. Errors may exhibit as random bit flips or IO failures.
2380 There are tools to access the internal log (nvme log and nvme-cli) for
2381 a more detailed analysis.
2382 There are separate error detection and correction steps performed e.g.
2384 on the bus level and in most cases never making in to the filesystem
2385 level. Once this happens it could mean there's some systematic error
2386 like overheating or bad physical connection of the device. You may want
2387 to run self-tests (using smartctl).
2388 • https://en.wikipedia.org/wiki/NVM_Express
2390 • https://www.smartmontools.org/wiki/NVMe_Support
2392 Drive firmware
2394 Firmware is technically still software but embedded into the hardware.
2395 As all software has bugs, so does firmware. Storage devices can update
2396 the firmware and fix known bugs. In some cases the it's possible to
2397 avoid certain bugs by quirks (device-specific workarounds) in Linux
2398 kernel.
2399 A faulty firmware can cause wide range of corruptions from small and
2401 localized to large affecting lots of data. Self-repair capabilities may
2402 not be sufficient.
2403 What to do:
2405 • check for firmware updates in case there are known problems, note
2407 that updating firmware can be risky on itself
2408 • use up-to-date kernel (recent releases or maintained long term sup‐
2410 port versions)
2411 SD flash cards
2413 There are a lot of devices with low power consumption and thus using
2414 storage media based on low power consumption too, typically flash mem‐
2415 ory stored on a chip enclosed in a detachable card package. An improp‐
2416 erly inserted card may be damaged by electrical spikes when the device
2417 is turned on or off. The chips storing data in turn may be damaged per‐
2418 manently. All types of flash memory have a limited number of rewrites,
2419 so the data are internally translated by FTL (flash translation layer).
2420 This is implemented in firmware (technically a software) and prone to
2421 bugs that manifest as hardware errors.
2422 Adding redundancy like using DUP profiles for both data and metadata
2424 can help in some cases but a full backup might be the best option once
2425 problems appear and replacing the card could be required as well.
2426 Hardware as the main source of filesystem corruptions
2428 If you use unreliable hardware and don't know about that, don't blame
2429 the filesystem when it tells you.
2430
2432 acl(5), btrfs(8), chattr(1), fstrim(8), ioctl(2), mkfs.btrfs(8),
2433 mount(8), swapon(8)
24346.6.2 Nov 24, 2023 BTRFS(5)