1BTRFS(5) BTRFS BTRFS(5)
2
6 btrfs - topics about the BTRFS filesystem (mount options, supported
7 file attributes and other)
8
10 This document describes topics related to BTRFS that are not specific
11 to the tools. Currently covers:
12 1. mount options
14 2. filesystem features
16 3. checksum algorithms
18 4. compression
20 5. filesystem exclusive operations
22 6. filesystem limits
24 7. bootloader support
26 8. file attributes
28 9. zoned mode
30 10. control device
32 11. filesystems with multiple block group profiles
34 12. seeding device
36 13. raid56 status and recommended practices
38 14. storage model, hardware considerations
40
42 This section describes mount options specific to BTRFS. For the
43 generic mount options please refer to mount(8) manpage. The options are
44 sorted alphabetically (discarding the no prefix).
45 NOTE:
47 Most mount options apply to the whole filesystem and only options in
48 the first mounted subvolume will take effect. This is due to lack of
49 implementation and may change in the future. This means that (for
50 example) you can't set per-subvolume nodatacow, nodatasum, or com‐
51 press using mount options. This should eventually be fixed, but it
52 has proved to be difficult to implement correctly within the Linux
53 VFS framework.
54 Mount options are processed in order, only the last occurrence of an
56 option takes effect and may disable other options due to constraints
57 (see eg. nodatacow and compress). The output of mount command shows
58 which options have been applied.
59 acl, noacl
61 (default: on)
62 Enable/disable support for Posix Access Control Lists (ACLs).
64 See the acl(5) manual page for more information about ACLs.
65 The support for ACL is build-time configurable
67 (BTRFS_FS_POSIX_ACL) and mount fails if acl is requested but the
68 feature is not compiled in.
69 autodefrag, noautodefrag
71 (since: 3.0, default: off)
72 Enable automatic file defragmentation. When enabled, small ran‐
74 dom writes into files (in a range of tens of kilobytes, cur‐
75 rently it's 64KiB) are detected and queued up for the defragmen‐
76 tation process. Not well suited for large database workloads.
77 The read latency may increase due to reading the adjacent blocks
79 that make up the range for defragmentation, successive write
80 will merge the blocks in the new location.
81 WARNING:
83 Defragmenting with Linux kernel versions < 3.9 or ≥ 3.14-rc2
84 as well as with Linux stable kernel versions ≥ 3.10.31, ≥
85 3.12.12 or ≥ 3.13.4 will break up the reflinks of COW data
86 (for example files copied with cp --reflink, snapshots or
87 de-duplicated data). This may cause considerable increase of
88 space usage depending on the broken up reflinks.
89 barrier, nobarrier
91 (default: on)
92 Ensure that all IO write operations make it through the device
94 cache and are stored permanently when the filesystem is at its
95 consistency checkpoint. This typically means that a flush com‐
96 mand is sent to the device that will synchronize all pending
97 data and ordinary metadata blocks, then writes the superblock
98 and issues another flush.
99 The write flushes incur a slight hit and also prevent the IO
101 block scheduler to reorder requests in a more effective way.
102 Disabling barriers gets rid of that penalty but will most cer‐
103 tainly lead to a corrupted filesystem in case of a crash or
104 power loss. The ordinary metadata blocks could be yet unwritten
105 at the time the new superblock is stored permanently, expecting
106 that the block pointers to metadata were stored permanently be‐
107 fore.
108 On a device with a volatile battery-backed write-back cache, the
110 nobarrier option will not lead to filesystem corruption as the
111 pending blocks are supposed to make it to the permanent storage.
112 check_int, check_int_data, check_int_print_mask=<value>
114 (since: 3.0, default: off)
115 These debugging options control the behavior of the integrity
117 checking module (the BTRFS_FS_CHECK_INTEGRITY config option re‐
118 quired). The main goal is to verify that all blocks from a given
119 transaction period are properly linked.
120 check_int enables the integrity checker module, which examines
122 all block write requests to ensure on-disk consistency, at a
123 large memory and CPU cost.
124 check_int_data includes extent data in the integrity checks, and
126 implies the check_int option.
127 check_int_print_mask takes a bitmask of BTRFSIC_PRINT_MASK_*
129 values as defined in fs/btrfs/check-integrity.c, to control the
130 integrity checker module behavior.
131 See comments at the top of fs/btrfs/check-integrity.c for more
133 information.
134 clear_cache
136 Force clearing and rebuilding of the disk space cache if some‐
137 thing has gone wrong. See also: space_cache.
138 commit=<seconds>
140 (since: 3.12, default: 30)
141 Set the interval of periodic transaction commit when data are
143 synchronized to permanent storage. Higher interval values lead
144 to larger amount of unwritten data, which has obvious conse‐
145 quences when the system crashes. The upper bound is not forced,
146 but a warning is printed if it's more than 300 seconds (5 min‐
147 utes). Use with care.
148 compress, compress=<type[:level]>, compress-force, com‐
150 press-force=<type[:level]>
151 (default: off, level support since: 5.1)
152 Control BTRFS file data compression. Type may be specified as
154 zlib, lzo, zstd or no (for no compression, used for remounting).
155 If no type is specified, zlib is used. If compress-force is
156 specified, then compression will always be attempted, but the
157 data may end up uncompressed if the compression would make them
158 larger.
159 Both zlib and zstd (since version 5.1) expose the compression
161 level as a tunable knob with higher levels trading speed and
162 memory (zstd) for higher compression ratios. This can be set by
163 appending a colon and the desired level. Zlib accepts the range
164 [1, 9] and zstd accepts [1, 15]. If no level is set, both cur‐
165 rently use a default level of 3. The value 0 is an alias for the
166 default level.
167 Otherwise some simple heuristics are applied to detect an incom‐
169 pressible file. If the first blocks written to a file are not
170 compressible, the whole file is permanently marked to skip com‐
171 pression. As this is too simple, the compress-force is a work‐
172 around that will compress most of the files at the cost of some
173 wasted CPU cycles on failed attempts. Since kernel 4.15, a set
174 of heuristic algorithms have been improved by using frequency
175 sampling, repeated pattern detection and Shannon entropy calcu‐
176 lation to avoid that.
177 NOTE:
179 If compression is enabled, nodatacow and nodatasum are dis‐
180 abled.
181 datacow, nodatacow
183 (default: on)
184 Enable data copy-on-write for newly created files. Nodatacow
186 implies nodatasum, and disables compression. All files created
187 under nodatacow are also set the NOCOW file attribute (see
188 chattr(1)).
189 NOTE:
191 If nodatacow or nodatasum are enabled, compression is dis‐
192 abled.
193 Updates in-place improve performance for workloads that do fre‐
195 quent overwrites, at the cost of potential partial writes, in
196 case the write is interrupted (system crash, device failure).
197 datasum, nodatasum
199 (default: on)
200 Enable data checksumming for newly created files. Datasum im‐
202 plies datacow, ie. the normal mode of operation. All files cre‐
203 ated under nodatasum inherit the "no checksums" property, how‐
204 ever there's no corresponding file attribute (see chattr(1)).
205 NOTE:
207 If nodatacow or nodatasum are enabled, compression is dis‐
208 abled.
209 There is a slight performance gain when checksums are turned
211 off, the corresponding metadata blocks holding the checksums do
212 not need to updated. The cost of checksumming of the blocks in
213 memory is much lower than the IO, modern CPUs feature hardware
214 support of the checksumming algorithm.
215 degraded
217 (default: off)
218 Allow mounts with less devices than the RAID profile constraints
220 require. A read-write mount (or remount) may fail when there
221 are too many devices missing, for example if a stripe member is
222 completely missing from RAID0.
223 Since 4.14, the constraint checks have been improved and are
225 verified on the chunk level, not an the device level. This al‐
226 lows degraded mounts of filesystems with mixed RAID profiles for
227 data and metadata, even if the device number constraints would
228 not be satisfied for some of the profiles.
229 Example: metadata -- raid1, data -- single, devices -- /dev/sda,
231 /dev/sdb
232 Suppose the data are completely stored on sda, then missing sdb
234 will not prevent the mount, even if 1 missing device would nor‐
235 mally prevent (any) single profile to mount. In case some of the
236 data chunks are stored on sdb, then the constraint of sin‐
237 gle/data is not satisfied and the filesystem cannot be mounted.
238 device=<devicepath>
240 Specify a path to a device that will be scanned for BTRFS
241 filesystem during mount. This is usually done automatically by a
242 device manager (like udev) or using the btrfs device scan com‐
243 mand (eg. run from the initial ramdisk). In cases where this is
244 not possible the device mount option can help.
245 NOTE:
247 Booting eg. a RAID1 system may fail even if all filesystem's
248 device paths are provided as the actual device nodes may not
249 be discovered by the system at that point.
250 discard, discard=sync, discard=async, nodiscard
252 (default: off, async support since: 5.6)
253 Enable discarding of freed file blocks. This is useful for SSD
255 devices, thinly provisioned LUNs, or virtual machine images;
256 however, every storage layer must support discard for it to
257 work.
258 In the synchronous mode (sync or without option value), lack of
260 asynchronous queued TRIM on the backing device TRIM can severely
261 degrade performance, because a synchronous TRIM operation will
262 be attempted instead. Queued TRIM requires newer than SATA revi‐
263 sion 3.1 chipsets and devices.
264 The asynchronous mode (async) gathers extents in larger chunks
266 before sending them to the devices for TRIM. The overhead and
267 performance impact should be negligible compared to the previous
268 mode and it's supposed to be the preferred mode if needed.
269 If it is not necessary to immediately discard freed blocks, then
271 the fstrim tool can be used to discard all free blocks in a
272 batch. Scheduling a TRIM during a period of low system activity
273 will prevent latent interference with the performance of other
274 operations. Also, a device may ignore the TRIM command if the
275 range is too small, so running a batch discard has a greater
276 probability of actually discarding the blocks.
277 enospc_debug, noenospc_debug
279 (default: off)
280 Enable verbose output for some ENOSPC conditions. It's safe to
282 use but can be noisy if the system reaches near-full state.
283 fatal_errors=<action>
285 (since: 3.4, default: bug)
286 Action to take when encountering a fatal error.
288 bug BUG() on a fatal error, the system will stay in the
290 crashed state and may be still partially usable, but re‐
291 boot is required for full operation
292 panic panic() on a fatal error, depending on other system con‐
294 figuration, this may be followed by a reboot. Please re‐
295 fer to the documentation of kernel boot parameters, eg.
296 panic, oops or crashkernel.
297 flushoncommit, noflushoncommit
299 (default: off)
300 This option forces any data dirtied by a write in a prior trans‐
302 action to commit as part of the current commit, effectively a
303 full filesystem sync.
304 This makes the committed state a fully consistent view of the
306 file system from the application's perspective (i.e. it includes
307 all completed file system operations). This was previously the
308 behavior only when a snapshot was created.
309 When off, the filesystem is consistent but buffered writes may
311 last more than one transaction commit.
312 fragment=<type>
314 (depends on compile-time option BTRFS_DEBUG, since: 4.4, de‐
315 fault: off)
316 A debugging helper to intentionally fragment given type of block
318 groups. The type can be data, metadata or all. This mount option
319 should not be used outside of debugging environments and is not
320 recognized if the kernel config option BTRFS_DEBUG is not en‐
321 abled.
322 nologreplay
324 (default: off, even read-only)
325 The tree-log contains pending updates to the filesystem until
327 the full commit. The log is replayed on next mount, this can be
328 disabled by this option. See also treelog. Note that nologre‐
329 play is the same as norecovery.
330 WARNING:
332 Currently, the tree log is replayed even with a read-only
333 mount! To disable that behaviour, mount also with nologre‐
334 play.
335 max_inline=<bytes>
337 (default: min(2048, page size) )
338 Specify the maximum amount of space, that can be inlined in a
340 metadata b-tree leaf. The value is specified in bytes, option‐
341 ally with a K suffix (case insensitive). In practice, this
342 value is limited by the filesystem block size (named sectorsize
343 at mkfs time), and memory page size of the system. In case of
344 sectorsize limit, there's some space unavailable due to leaf
345 headers. For example, a 4KiB sectorsize, maximum size of inline
346 data is about 3900 bytes.
347 Inlining can be completely turned off by specifying 0. This will
349 increase data block slack if file sizes are much smaller than
350 block size but will reduce metadata consumption in return.
351 NOTE:
353 The default value has changed to 2048 in kernel 4.6.
354 metadata_ratio=<value>
356 (default: 0, internal logic)
357 Specifies that 1 metadata chunk should be allocated after every
359 value data chunks. Default behaviour depends on internal logic,
360 some percent of unused metadata space is attempted to be main‐
361 tained but is not always possible if there's not enough space
362 left for chunk allocation. The option could be useful to over‐
363 ride the internal logic in favor of the metadata allocation if
364 the expected workload is supposed to be metadata intense (snap‐
365 shots, reflinks, xattrs, inlined files).
366 norecovery
368 (since: 4.5, default: off)
369 Do not attempt any data recovery at mount time. This will dis‐
371 able logreplay and avoids other write operations. Note that this
372 option is the same as nologreplay.
373 NOTE:
375 The opposite option recovery used to have different meaning
376 but was changed for consistency with other filesystems, where
377 norecovery is used for skipping log replay. BTRFS does the
378 same and in general will try to avoid any write operations.
379 rescan_uuid_tree
381 (since: 3.12, default: off)
382 Force check and rebuild procedure of the UUID tree. This should
384 not normally be needed.
385 rescue (since: 5.9)
387 Modes allowing mount with damaged filesystem structures.
389 • usebackuproot (since: 5.9, replaces standalone option useback‐
391 uproot)
392 • nologreplay (since: 5.9, replaces standalone option nologre‐
394 play)
395 • ignorebadroots, ibadroots (since: 5.11)
397 • ignoredatacsums, idatacsums (since: 5.11)
399 • all (since: 5.9)
401 skip_balance
403 (since: 3.3, default: off)
404 Skip automatic resume of an interrupted balance operation. The
406 operation can later be resumed with btrfs balance resume, or the
407 paused state can be removed with btrfs balance cancel. The de‐
408 fault behaviour is to resume an interrupted balance immediately
409 after a volume is mounted.
410 space_cache, space_cache=<version>, nospace_cache
412 (nospace_cache since: 3.2, space_cache=v1 and space_cache=v2
413 since 4.5, default: space_cache=v1)
414 Options to control the free space cache. The free space cache
416 greatly improves performance when reading block group free space
417 into memory. However, managing the space cache consumes some re‐
418 sources, including a small amount of disk space.
419 There are two implementations of the free space cache. The orig‐
421 inal one, referred to as v1, is the safe default. The v1 space
422 cache can be disabled at mount time with nospace_cache without
423 clearing.
424 On very large filesystems (many terabytes) and certain work‐
426 loads, the performance of the v1 space cache may degrade drasti‐
427 cally. The v2 implementation, which adds a new b-tree called the
428 free space tree, addresses this issue. Once enabled, the v2
429 space cache will always be used and cannot be disabled unless it
430 is cleared. Use clear_cache,space_cache=v1 or
431 clear_cache,nospace_cache to do so. If v2 is enabled, kernels
432 without v2 support will only be able to mount the filesystem in
433 read-only mode.
434 The btrfs-check(8) and `mkfs.btrfs(8) commands have full v2 free
436 space cache support since v4.19.
437 If a version is not explicitly specified, the default implemen‐
439 tation will be chosen, which is v1.
440 ssd, ssd_spread, nossd, nossd_spread
442 (default: SSD autodetected)
443 Options to control SSD allocation schemes. By default, BTRFS
445 will enable or disable SSD optimizations depending on status of
446 a device with respect to rotational or non-rotational type. This
447 is determined by the contents of /sys/block/DEV/queue/rota‐
448 tional). If it is 0, the ssd option is turned on. The option
449 nossd will disable the autodetection.
450 The optimizations make use of the absence of the seek penalty
452 that's inherent for the rotational devices. The blocks can be
453 typically written faster and are not offloaded to separate
454 threads.
455 NOTE:
457 Since 4.14, the block layout optimizations have been dropped.
458 This used to help with first generations of SSD devices.
459 Their FTL (flash translation layer) was not effective and the
460 optimization was supposed to improve the wear by better
461 aligning blocks. This is no longer true with modern SSD de‐
462 vices and the optimization had no real benefit. Furthermore
463 it caused increased fragmentation. The layout tuning has been
464 kept intact for the option ssd_spread.
465 The ssd_spread mount option attempts to allocate into bigger and
467 aligned chunks of unused space, and may perform better on
468 low-end SSDs. ssd_spread implies ssd, enabling all other SSD
469 heuristics as well. The option nossd will disable all SSD op‐
470 tions while nossd_spread only disables ssd_spread.
471 subvol=<path>
473 Mount subvolume from path rather than the toplevel subvolume.
474 The path is always treated as relative to the toplevel subvol‐
475 ume. This mount option overrides the default subvolume set for
476 the given filesystem.
477 subvolid=<subvolid>
479 Mount subvolume specified by a subvolid number rather than the
480 toplevel subvolume. You can use btrfs subvolume list of btrfs
481 subvolume show to see subvolume ID numbers. This mount option
482 overrides the default subvolume set for the given filesystem.
483 NOTE:
485 If both subvolid and subvol are specified, they must point at
486 the same subvolume, otherwise the mount will fail.
487 thread_pool=<number>
489 (default: min(NRCPUS + 2, 8) )
490 The number of worker threads to start. NRCPUS is number of
492 on-line CPUs detected at the time of mount. Small number leads
493 to less parallelism in processing data and metadata, higher num‐
494 bers could lead to a performance hit due to increased locking
495 contention, process scheduling, cache-line bouncing or costly
496 data transfers between local CPU memories.
497 treelog, notreelog
499 (default: on)
500 Enable the tree logging used for fsync and O_SYNC writes. The
502 tree log stores changes without the need of a full filesystem
503 sync. The log operations are flushed at sync and transaction
504 commit. If the system crashes between two such syncs, the pend‐
505 ing tree log operations are replayed during mount.
506 WARNING:
508 Currently, the tree log is replayed even with a read-only
509 mount! To disable that behaviour, also mount with nologre‐
510 play.
511 The tree log could contain new files/directories, these would
513 not exist on a mounted filesystem if the log is not replayed.
514 usebackuproot
516 (since: 4.6, default: off)
517 Enable autorecovery attempts if a bad tree root is found at
519 mount time. Currently this scans a backup list of several pre‐
520 vious tree roots and tries to use the first readable. This can
521 be used with read-only mounts as well.
522 NOTE:
524 This option has replaced recovery.
525 user_subvol_rm_allowed
527 (default: off)
528 Allow subvolumes to be deleted by their respective owner. Other‐
530 wise, only the root user can do that.
531 NOTE:
533 Historically, any user could create a snapshot even if he was
534 not owner of the source subvolume, the subvolume deletion has
535 been restricted for that reason. The subvolume creation has
536 been restricted but this mount option is still required. This
537 is a usability issue. Since 4.18, the rmdir(2) syscall can
538 delete an empty subvolume just like an ordinary directory.
539 Whether this is possible can be detected at runtime, see
540 rmdir_subvol feature in FILESYSTEM FEATURES.
541 DEPRECATED MOUNT OPTIONS
543 List of mount options that have been removed, kept for backward compat‐
544 ibility.
545 recovery
547 (since: 3.2, default: off, deprecated since: 4.5)
548 NOTE:
550 This option has been replaced by usebackuproot and should not
551 be used but will work on 4.5+ kernels.
552 inode_cache, noinode_cache
554 (removed in: 5.11, since: 3.0, default: off)
555 NOTE:
557 The functionality has been removed in 5.11, any stale data
558 created by previous use of the inode_cache option can be re‐
559 moved by btrfs check --clear-ino-cache.
560 NOTES ON GENERIC MOUNT OPTIONS
562 Some of the general mount options from mount(8) that affect BTRFS and
563 are worth mentioning.
564 noatime
566 under read intensive work-loads, specifying noatime signifi‐
567 cantly improves performance because no new access time informa‐
568 tion needs to be written. Without this option, the default is
569 relatime, which only reduces the number of inode atime updates
570 in comparison to the traditional strictatime. The worst case for
571 atime updates under 'relatime' occurs when many files are read
572 whose atime is older than 24 h and which are freshly snapshot‐
573 ted. In that case the atime is updated and COW happens - for
574 each file - in bulk. See also https://lwn.net/Articles/499293/ -
575 Atime and btrfs: a bad combination? (LWN, 2012-05-31).
576 Note that noatime may break applications that rely on atime up‐
578 times like the venerable Mutt (unless you use maildir mail‐
579 boxes).
580
582 The basic set of filesystem features gets extended over time. The back‐
583 ward compatibility is maintained and the features are optional, need to
584 be explicitly asked for so accidental use will not create incompatibil‐
585 ities.
586 There are several classes and the respective tools to manage the fea‐
588 tures:
589 at mkfs time only
591 This is namely for core structures, like the b-tree nodesize or
592 checksum algorithm, see mkfs.btrfs(8) for more details.
593 after mkfs, on an unmounted filesystem::
595 Features that may optimize internal structures or add new struc‐
596 tures to support new functionality, see btrfstune(8). The com‐
597 mand btrfs inspect-internal dump-super /dev/sdx will dump a su‐
598 perblock, you can map the value of incompat_flags to the fea‐
599 tures listed below
600 after mkfs, on a mounted filesystem
602 The features of a filesystem (with a given UUID) are listed in
603 /sys/fs/btrfs/UUID/features/, one file per feature. The status
604 is stored inside the file. The value 1 is for enabled and ac‐
605 tive, while 0 means the feature was enabled at mount time but
606 turned off afterwards.
607 Whether a particular feature can be turned on a mounted filesys‐
609 tem can be found in the directory /sys/fs/btrfs/features/, one
610 file per feature. The value 1 means the feature can be enabled.
611 List of features (see also mkfs.btrfs(8) section FILESYSTEM FEATURES):
613 big_metadata
615 (since: 3.4)
616 the filesystem uses nodesize for metadata blocks, this can be
618 bigger than the page size
619 compress_lzo
621 (since: 2.6.38)
622 the lzo compression has been used on the filesystem, either as a
624 mount option or via btrfs filesystem defrag.
625 compress_zstd
627 (since: 4.14)
628 the zstd compression has been used on the filesystem, either as
630 a mount option or via btrfs filesystem defrag.
631 default_subvol
633 (since: 2.6.34)
634 the default subvolume has been set on the filesystem
636 extended_iref
638 (since: 3.7)
639 increased hardlink limit per file in a directory to 65536, older
641 kernels supported a varying number of hardlinks depending on the
642 sum of all file name sizes that can be stored into one metadata
643 block
644 free_space_tree
646 (since: 4.5)
647 free space representation using a dedicated b-tree, successor of
649 v1 space cache
650 metadata_uuid
652 (since: 5.0)
653 the main filesystem UUID is the metadata_uuid, which stores the
655 new UUID only in the superblock while all metadata blocks still
656 have the UUID set at mkfs time, see btrfstune(8) for more
657 mixed_backref
659 (since: 2.6.31)
660 the last major disk format change, improved backreferences, now
662 default
663 mixed_groups
665 (since: 2.6.37)
666 mixed data and metadata block groups, ie. the data and metadata
668 are not separated and occupy the same block groups, this mode is
669 suitable for small volumes as there are no constraints how the
670 remaining space should be used (compared to the split mode,
671 where empty metadata space cannot be used for data and vice
672 versa)
673 on the other hand, the final layout is quite unpredictable and
675 possibly highly fragmented, which means worse performance
676 no_holes
678 (since: 3.14)
679 improved representation of file extents where holes are not ex‐
681 plicitly stored as an extent, saves a few percent of metadata if
682 sparse files are used
683 raid1c34
685 (since: 5.5)
686 extended RAID1 mode with copies on 3 or 4 devices respectively
688 raid56 (since: 3.9)
690 the filesystem contains or contained a raid56 profile of block
692 groups
693 rmdir_subvol
695 (since: 4.18)
696 indicate that rmdir(2) syscall can delete an empty subvolume
698 just like an ordinary directory. Note that this feature only de‐
699 pends on the kernel version.
700 skinny_metadata
702 (since: 3.10)
703 reduced-size metadata for extent references, saves a few percent
705 of metadata
706 send_stream_version
708 (since: 5.10)
709 number of the highest supported send stream version
711 supported_checksums
713 (since: 5.5)
714 list of checksum algorithms supported by the kernel module, the
716 respective modules or built-in implementing the algorithms need
717 to be present to mount the filesystem, see CHECKSUM ALGORITHMS
718 supported_sectorsizes
720 (since: 5.13)
721 list of values that are accepted as sector sizes (mkfs.btrfs
723 --sectorsize) by the running kernel
724 supported_rescue_options
726 (since: 5.11)
727 list of values for the mount option rescue that are supported by
729 the running kernel, see btrfs(5)
730 zoned (since: 5.12)
732 zoned mode is allocation/write friendly to host-managed zoned
734 devices, allocation space is partitioned into fixed-size zones
735 that must be updated sequentially, see ZONED MODE
736
738 A swapfile is file-backed memory that the system uses to temporarily
739 offload the RAM. It is supported since kernel 5.0. Use swapon(8) to
740 activate the swapfile. There are some limitations of the implementation
741 in BTRFS and linux swap subsystem:
742 • filesystem - must be only single device
744 • filesystem - must have only single data profile
746 • swapfile - the containing subvolume cannot be snapshotted
748 • swapfile - must be preallocated
750 • swapfile - must be nodatacow (ie. also nodatasum)
752 • swapfile - must not be compressed
754 The limitations come namely from the COW-based design and mapping layer
756 of blocks that allows the advanced features like relocation and
757 multi-device filesystems. However, the swap subsystem expects simpler
758 mapping and no background changes of the file blocks once they've been
759 attached to swap.
760 With active swapfiles, the following whole-filesystem operations will
762 skip swapfile extents or may fail:
763 • balance - block groups with swapfile extents are skipped and re‐
765 ported, the rest will be processed normally
766 • resize grow - unaffected
768 • resize shrink - works as long as the extents are outside of the
770 shrunk range
771 • device add - a new device does not interfere with existing swapfile
773 and this operation will work, though no new swapfile can be activated
774 afterwards
775 • device delete - if the device has been added as above, it can be also
777 deleted
778 • device replace - ditto
780 When there are no active swapfiles and a whole-filesystem exclusive op‐
782 eration is running (eg. balance, device delete, shrink), the swapfiles
783 cannot be temporarily activated. The operation must finish first.
784 To create and activate a swapfile run the following commands:
786 # truncate -s 0 swapfile
788 # chattr +C swapfile
789 # fallocate -l 2G swapfile
790 # chmod 0600 swapfile
791 # mkswap swapfile
792 # swapon swapfile
793 Please note that the UUID returned by the mkswap utility identifies the
795 swap "filesystem" and because it's stored in a file, it's not generally
796 visible and usable as an identifier unlike if it was on a block device.
797 The file will appear in /proc/swaps:
799 # cat /proc/swaps
801 Filename Type Size Used Priority
802 /path/swapfile file 2097152 0 -2
803 --------------------
804 The swapfile can be created as one-time operation or, once properly
806 created, activated on each boot by the swapon -a command (usually
807 started by the service manager). Add the following entry to /etc/fstab,
808 assuming the filesystem that provides the /path has been already
809 mounted at this point. Additional mount options relevant for the swap‐
810 file can be set too (like priority, not the BTRFS mount options).
811 /path/swapfile none swap defaults 0 0
813
815 Data and metadata are checksummed by default, the checksum is calcu‐
816 lated before write and verifed after reading the blocks from devices.
817 The whole metadata block has a checksum stored inline in the b-tree
818 node header, each data block has a detached checksum stored in the
819 checksum tree.
820 There are several checksum algorithms supported. The default and back‐
822 ward compatible is crc32c. Since kernel 5.5 there are three more with
823 different characteristics and trade-offs regarding speed and strength.
824 The following list may help you to decide which one to select.
825 CRC32C (32bit digest)
827 default, best backward compatibility, very fast, modern CPUs
828 have instruction-level support, not collision-resistant but
829 still good error detection capabilities
830 XXHASH (64bit digest)
832 can be used as CRC32C successor, very fast, optimized for modern
833 CPUs utilizing instruction pipelining, good collision resistance
834 and error detection
835 SHA256 (256bit digest)::
837 a cryptographic-strength hash, relatively slow but with possible
838 CPU instruction acceleration or specialized hardware cards, FIPS
839 certified and in wide use
840 BLAKE2b (256bit digest)
842 a cryptographic-strength hash, relatively fast with possible CPU
843 acceleration using SIMD extensions, not standardized but based
844 on BLAKE which was a SHA3 finalist, in wide use, the algorithm
845 used is BLAKE2b-256 that's optimized for 64bit platforms
846 The digest size affects overall size of data block checksums stored in
848 the filesystem. The metadata blocks have a fixed area up to 256 bits
849 (32 bytes), so there's no increase. Each data block has a separate
850 checksum stored, with additional overhead of the b-tree leaves.
851 Approximate relative performance of the algorithms, measured against
853 CRC32C using reference software implementations on a 3.5GHz intel CPU:
854 ┌────────┬─────────────┬───────┬─────────────────┐
856 │Digest │ Cycles/4KiB │ Ratio │ Implementation │
857 ├────────┼─────────────┼───────┼─────────────────┤
858 │CRC32C │ 1700 │ 1.00 │ CPU instruction │
859 ├────────┼─────────────┼───────┼─────────────────┤
860 │XXHASH │ 2500 │ 1.44 │ reference impl. │
861 ├────────┼─────────────┼───────┼─────────────────┤
862 │SHA256 │ 105000 │ 61 │ reference impl. │
863 ├────────┼─────────────┼───────┼─────────────────┤
864 │SHA256 │ 36000 │ 21 │ libgcrypt/AVX2 │
865 ├────────┼─────────────┼───────┼─────────────────┤
866 │SHA256 │ 63000 │ 37 │ libsodium/AVX2 │
867 ├────────┼─────────────┼───────┼─────────────────┤
868 │BLAKE2b │ 22000 │ 13 │ reference impl. │
869 ├────────┼─────────────┼───────┼─────────────────┤
870 │BLAKE2b │ 19000 │ 11 │ libgcrypt/AVX2 │
871 ├────────┼─────────────┼───────┼─────────────────┤
872 │BLAKE2b │ 19000 │ 11 │ libsodium/AVX2 │
873 └────────┴─────────────┴───────┴─────────────────┘
874 Many kernels are configured with SHA256 as built-in and not as a mod‐
876 ule. The accelerated versions are however provided by the modules and
877 must be loaded explicitly (modprobe sha256) before mounting the
878 filesystem to make use of them. You can check in
879 /sys/fs/btrfs/FSID/checksum which one is used. If you see
880 sha256-generic, then you may want to unmount and mount the filesystem
881 again, changing that on a mounted filesystem is not possible. Check
882 the file /proc/crypto, when the implementation is built-in, you'd find
883 name : sha256
885 driver : sha256-generic
886 module : kernel
887 priority : 100
888 ...
889 while accelerated implementation is e.g.
891 name : sha256
893 driver : sha256-avx2
894 module : sha256_ssse3
895 priority : 170
896 ...
897
899 Btrfs supports transparent file compression. There are three algorithms
900 available: ZLIB, LZO and ZSTD (since v4.14), with various levels. The
901 compression happens on the level of file extents and the algorithm is
902 selected by file property, mount option or by a defrag command. You
903 can have a single btrfs mount point that has some files that are uncom‐
904 pressed, some that are compressed with LZO, some with ZLIB, for in‐
905 stance (though you may not want it that way, it is supported).
906 Once the compression is set, all newly written data will be compressed,
908 ie. existing data are untouched. Data are split into smaller chunks
909 (128KiB) before compression to make random rewrites possible without a
910 high performance hit. Due to the increased number of extents the meta‐
911 data consumption is higher. The chunks are compressed in parallel.
912 The algorithms can be characterized as follows regarding the speed/ra‐
914 tio trade-offs:
915 ZLIB
917 • slower, higher compression ratio
919 • levels: 1 to 9, mapped directly, default level is 3
921 • good backward compatibility
923 LZO
925 • faster compression and decompression than zlib, worse compres‐
927 sion ratio, designed to be fast
928 • no levels
930 • good backward compatibility
932 ZSTD
934 • compression comparable to zlib with higher compression/decom‐
936 pression speeds and different ratio
937 • levels: 1 to 15, mapped directly (higher levels are not avail‐
939 able)
940 • since 4.14, levels since 5.1
942 The differences depend on the actual data set and cannot be expressed
944 by a single number or recommendation. Higher levels consume more CPU
945 time and may not bring a significant improvement, lower levels are
946 close to real time.
947
949 Typically the compression can be enabled on the whole filesystem, spec‐
950 ified for the mount point. Note that the compression mount options are
951 shared among all mounts of the same filesystem, either bind mounts or
952 subvolume mounts. Please refer to section MOUNT OPTIONS.
953 $ mount -o compress=zstd /dev/sdx /mnt
955 This will enable the zstd algorithm on the default level (which is 3).
957 The level can be specified manually too like zstd:3. Higher levels com‐
958 press better at the cost of time. This in turn may cause increased
959 write latency, low levels are suitable for real-time compression and on
960 reasonably fast CPU don't cause noticeable performance drops.
961 $ btrfs filesystem defrag -czstd file
963 The command above will start defragmentation of the whole file and ap‐
965 ply the compression, regardless of the mount option. (Note: specifying
966 level is not yet implemented). The compression algorithm is not persis‐
967 tent and applies only to the defragmentation command, for any other
968 writes other compression settings apply.
969 Persistent settings on a per-file basis can be set in two ways:
971 $ chattr +c file
973 $ btrfs property set file compression zstd
974 The first command is using legacy interface of file attributes inher‐
976 ited from ext2 filesystem and is not flexible, so by default the zlib
977 compression is set. The other command sets a property on the file with
978 the given algorithm. (Note: setting level that way is not yet imple‐
979 mented.)
980
982 The level support of ZLIB has been added in v4.14, LZO does not support
983 levels (the kernel implementation provides only one), ZSTD level sup‐
984 port has been added in v5.1.
985 There are 9 levels of ZLIB supported (1 to 9), mapping 1:1 from the
987 mount option to the algorithm defined level. The default is level 3,
988 which provides the reasonably good compression ratio and is still rea‐
989 sonably fast. The difference in compression gain of levels 7, 8 and 9
990 is comparable but the higher levels take longer.
991 The ZSTD support includes levels 1 to 15, a subset of full range of
993 what ZSTD provides. Levels 1-3 are real-time, 4-8 slower with improved
994 compression and 9-15 try even harder though the resulting size may not
995 be significantly improved.
996 Level 0 always maps to the default. The compression level does not af‐
998 fect compatibility.
999
1001 Files with already compressed data or with data that won't compress
1002 well with the CPU and memory constraints of the kernel implementations
1003 are using a simple decision logic. If the first portion of data being
1004 compressed is not smaller than the original, the compression of the
1005 file is disabled -- unless the filesystem is mounted with com‐
1006 press-force. In that case compression will always be attempted on the
1007 file only to be later discarded. This is not optimal and subject to op‐
1008 timizations and further development.
1009 If a file is identified as incompressible, a flag is set (NOCOMPRESS)
1011 and it's sticky. On that file compression won't be performed unless
1012 forced. The flag can be also set by chattr +m (since e2fsprogs 1.46.2)
1013 or by properties with value no or none. Empty value will reset it to
1014 the default that's currently applicable on the mounted filesystem.
1015 There are two ways to detect incompressible data:
1017 • actual compression attempt - data are compressed, if the result is
1019 not smaller, it's discarded, so this depends on the algorithm and
1020 level
1021 • pre-compression heuristics - a quick statistical evaluation on the
1023 data is performed and based on the result either compression is per‐
1024 formed or skipped, the NOCOMPRESS bit is not set just by the heuris‐
1025 tic, only if the compression algorithm does not make an improvement
1026 $ lsattr file
1028 ---------------------m file
1029 Using the forcing compression is not recommended, the heuristics are
1031 supposed to decide that and compression algorithms internally detect
1032 incompressible data too.
1033
1035 The heuristics aim to do a few quick statistical tests on the com‐
1036 pressed data in order to avoid probably costly compression that would
1037 turn out to be inefficient. Compression algorithms could have internal
1038 detection of incompressible data too but this leads to more overhead as
1039 the compression is done in another thread and has to write the data
1040 anyway. The heuristic is read-only and can utilize cached memory.
1041 The tests performed based on the following: data sampling, long re‐
1043 peated pattern detection, byte frequency, Shannon entropy.
1044
1046 Compression is done using the COW mechanism so it's incompatible with
1047 nodatacow. Direct IO works on compressed files but will fall back to
1048 buffered writes and leads to recompression. Currently nodatasum and
1049 compression don't work together.
1050 The compression algorithms have been added over time so the version
1052 compatibility should be also considered, together with other tools that
1053 may access the compressed data like bootloaders.
1054
1056 There are several operations that affect the whole filesystem and can‐
1057 not be run in parallel. Attempt to start one while another is running
1058 will fail (see exceptions below).
1059 Since kernel 5.10 the currently running operation can be obtained from
1061 /sys/fs/UUID/exclusive_operation with following values and operations:
1062 • balance
1064 • balance paused (since 5.17)
1066 • device add
1068 • device delete
1070 • device replace
1072 • resize
1074 • swapfile activate
1076 • none
1078 Enqueuing is supported for several btrfs subcommands so they can be
1080 started at once and then serialized.
1081 There's an exception when a paused balance allows to start a device add
1083 operation as they don't really collide and this can be used to add more
1084 space for the balance to finish.
1085
1087 maximum file name length
1088 255
1089 maximum symlink target length
1091 depends on the nodesize value, for 4KiB it's 3949 bytes, for
1092 larger nodesize it's 4095 due to the system limit PATH_MAX
1093 The symlink target may not be a valid path, ie. the path name
1095 components can exceed the limits (NAME_MAX), there's no content
1096 validation at symlink(3) creation.
1097 maximum number of inodes
1099 264 but depends on the available metadata space as the inodes
1100 are created dynamically
1101 inode numbers
1103 minimum number: 256 (for subvolumes), regular files and directo‐
1104 ries: 257
1105 maximum file length
1107 inherent limit of btrfs is 264 (16 EiB) but the linux VFS limit
1108 is 263 (8 EiB)
1109 maximum number of subvolumes
1111 the subvolume ids can go up to 264 but the number of actual sub‐
1112 volumes depends on the available metadata space, the space con‐
1113 sumed by all subvolume metadata includes bookkeeping of shared
1114 extents can be large (MiB, GiB)
1115 maximum number of hardlinks of a file in a directory
1117 65536 when the extref feature is turned on during mkfs (de‐
1118 fault), roughly 100 otherwise
1119 minimum filesystem size
1121 the minimal size of each device depends on the mixed-bg feature,
1122 without that (the default) it's about 109MiB, with mixed-bg it's
1123 is 16MiB
1124
1126 GRUB2 (https://www.gnu.org/software/grub) has the most advanced support
1127 of booting from BTRFS with respect to features.
1128 U-boot (https://www.denx.de/wiki/U-Boot/) has decent support for boot‐
1130 ing but not all BTRFS features are implemented, check the documenta‐
1131 tion.
1132 EXTLINUX (from the https://syslinux.org project) can boot but does not
1134 support all features. Please check the upstream documentation before
1135 you use it.
1136 The first 1MiB on each device is unused with the exception of primary
1138 superblock that is on the offset 64KiB and spans 4KiB.
1139
1141 The btrfs filesystem supports setting file attributes or flags. Note
1142 there are old and new interfaces, with confusing names. The following
1143 list should clarify that:
1144 • attributes: chattr(1) or lsattr(1) utilities (the ioctls are
1146 FS_IOC_GETFLAGS and FS_IOC_SETFLAGS), due to the ioctl names the at‐
1147 tributes are also called flags
1148 • xflags: to distinguish from the previous, it's extended flags, with
1150 tunable bits similar to the attributes but extensible and new bits
1151 will be added in the future (the ioctls are FS_IOC_FSGETXATTR and
1152 FS_IOC_FSSETXATTR but they are not related to extended attributes
1153 that are also called xattrs), there's no standard tool to change the
1154 bits, there's support in xfs_io(8) as command xfs_io -c chattr
1155 Attributes
1157 a append only, new writes are always written at the end of the
1158 file
1159 A no atime updates
1161 c compress data, all data written after this attribute is set will
1163 be compressed. Please note that compression is also affected by
1164 the mount options or the parent directory attributes.
1165 When set on a directory, all newly created files will inherit
1167 this attribute. This attribute cannot be set with 'm' at the
1168 same time.
1169 C no copy-on-write, file data modifications are done in-place
1171 When set on a directory, all newly created files will inherit
1173 this attribute.
1174 NOTE:
1176 Due to implementation limitations, this flag can be set/unset
1177 only on empty files.
1178 d no dump, makes sense with 3rd party tools like dump(8), on BTRFS
1180 the attribute can be set/unset but no other special handling is
1181 done
1182 D synchronous directory updates, for more details search open(2)
1184 for O_SYNC and O_DSYNC
1185 i immutable, no file data and metadata changes allowed even to the
1187 root user as long as this attribute is set (obviously the excep‐
1188 tion is unsetting the attribute)
1189 m no compression, permanently turn off compression on the given
1191 file. Any compression mount options will not affect this file.
1192 (chattr support added in 1.46.2)
1193 When set on a directory, all newly created files will inherit
1195 this attribute. This attribute cannot be set with c at the same
1196 time.
1197 S synchronous updates, for more details search open(2) for O_SYNC
1199 and O_DSYNC
1200 No other attributes are supported. For the complete list please refer
1202 to the chattr(1) manual page.
1203 XFLAGS
1205 There's overlap of letters assigned to the bits with the attributes,
1206 this list refers to what xfs_io(8) provides:
1207 i immutable, same as the attribute
1209 a append only, same as the attribute
1211 s synchronous updates, same as the attribute S
1213 A no atime updates, same as the attribute
1215 d no dump, same as the attribute
1217
1219 Since version 5.12 btrfs supports so called zoned mode. This is a spe‐
1220 cial on-disk format and allocation/write strategy that's friendly to
1221 zoned devices. In short, a device is partitioned into fixed-size zones
1222 and each zone can be updated by append-only manner, or reset. As btrfs
1223 has no fixed data structures, except the super blocks, the zoned mode
1224 only requires block placement that follows the device constraints. You
1225 can learn about the whole architecture at https://zonedstorage.io .
1226 The devices are also called SMR/ZBC/ZNS, in host-managed mode. Note
1228 that there are devices that appear as non-zoned but actually are, this
1229 is drive-managed and using zoned mode won't help.
1230 The zone size depends on the device, typical sizes are 256MiB or 1GiB.
1232 In general it must be a power of two. Emulated zoned devices like
1233 null_blk allow to set various zone sizes.
1234 Requirements, limitations
1236 • all devices must have the same zone size
1237 • maximum zone size is 8GiB
1239 • mixing zoned and non-zoned devices is possible, the zone writes are
1241 emulated, but this is namely for testing
1242 •
1244 the super block is handled in a special way and is at different loca‐
1246 tions than on a non-zoned filesystem:
1247 • primary: 0B (and the next two zones)
1249 • secondary: 512GiB (and the next two zones)
1251 • tertiary: 4TiB (4096GiB, and the next two zones)
1253 Incompatible features
1255 The main constraint of the zoned devices is lack of in-place update of
1256 the data. This is inherently incompatibile with some features:
1257 • nodatacow - overwrite in-place, cannot create such files
1259 • fallocate - preallocating space for in-place first write
1261 • mixed-bg - unordered writes to data and metadata, fixing that means
1263 using separate data and metadata block groups
1264 • booting - the zone at offset 0 contains superblock, resetting the
1266 zone would destroy the bootloader data
1267 Initial support lacks some features but they're planned:
1269 • only single profile is supported
1271 • fstrim - due to dependency on free space cache v1
1273 Super block
1275 As said above, super block is handled in a special way. In order to be
1276 crash safe, at least one zone in a known location must contain a valid
1277 superblock. This is implemented as a ring buffer in two consecutive
1278 zones, starting from known offsets 0B, 512GiB and 4TiB.
1279 The values are different than on non-zoned devices. Each new super
1281 block is appended to the end of the zone, once it's filled, the zone is
1282 reset and writes continue to the next one. Looking up the latest super
1283 block needs to read offsets of both zones and determine the last writ‐
1284 ten version.
1285 The amount of space reserved for super block depends on the zone size.
1287 The secondary and tertiary copies are at distant offsets as the capac‐
1288 ity of the devices is expected to be large, tens of terabytes. Maximum
1289 zone size supported is 8GiB, which would mean that eg. offset 0-16GiB
1290 would be reserved just for the super block on a hypothetical device of
1291 that zone size. This is wasteful but required to guarantee crash
1292 safety.
1293
1295 There's a character special device /dev/btrfs-control with major and
1296 minor numbers 10 and 234 (the device can be found under the 'misc' cat‐
1297 egory).
1298 $ ls -l /dev/btrfs-control
1300 crw------- 1 root root 10, 234 Jan 1 12:00 /dev/btrfs-control
1301 The device accepts some ioctl calls that can perform following actions
1303 on the filesystem module:
1304 • scan devices for btrfs filesystem (ie. to let multi-device filesys‐
1306 tems mount automatically) and register them with the kernel module
1307 • similar to scan, but also wait until the device scanning process is
1309 finished for a given filesystem
1310 • get the supported features (can be also found under
1312 /sys/fs/btrfs/features)
1313 The device is created when btrfs is initialized, either as a module or
1315 a built-in functionality and makes sense only in connection with that.
1316 Running eg. mkfs without the module loaded will not register the device
1317 and will probably warn about that.
1318 In rare cases when the module is loaded but the device is not present
1320 (most likely accidentally deleted), it's possible to recreate it by
1321 # mknod --mode=600 /dev/btrfs-control c 10 234
1323 or (since 5.11) by a convenience command
1325 # btrfs rescue create-control-device
1327 The control device is not strictly required but the device scanning
1329 will not work and a workaround would need to be used to mount a
1330 multi-device filesystem. The mount option device can trigger the de‐
1331 vice scanning during mount, see also btrfs device scan.
1332
1334 It is possible that a btrfs filesystem contains multiple block group
1335 profiles of the same type. This could happen when a profile conversion
1336 using balance filters is interrupted (see btrfs-balance(8)). Some
1337 btrfs commands perform a test to detect this kind of condition and
1338 print a warning like this:
1339 WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
1341 WARNING: Data: single, raid1
1342 WARNING: Metadata: single, raid1
1343 The corresponding output of btrfs filesystem df might look like:
1345 WARNING: Multiple block group profiles detected, see 'man btrfs(5)'.
1347 WARNING: Data: single, raid1
1348 WARNING: Metadata: single, raid1
1349 Data, RAID1: total=832.00MiB, used=0.00B
1350 Data, single: total=1.63GiB, used=0.00B
1351 System, single: total=4.00MiB, used=16.00KiB
1352 Metadata, single: total=8.00MiB, used=112.00KiB
1353 Metadata, RAID1: total=64.00MiB, used=32.00KiB
1354 GlobalReserve, single: total=16.25MiB, used=0.00B
1355 There's more than one line for type Data and Metadata, while the pro‐
1357 files are single and RAID1.
1358 This state of the filesystem OK but most likely needs the user/adminis‐
1360 trator to take an action and finish the interrupted tasks. This cannot
1361 be easily done automatically, also the user knows the expected final
1362 profiles.
1363 In the example above, the filesystem started as a single device and
1365 single block group profile. Then another device was added, followed by
1366 balance with convert=raid1 but for some reason hasn't finished.
1367 Restarting the balance with convert=raid1 will continue and end up with
1368 filesystem with all block group profiles RAID1.
1369 NOTE:
1371 If you're familiar with balance filters, you can use con‐
1372 vert=raid1,profiles=single,soft, which will take only the uncon‐
1373 verted single profiles and convert them to raid1. This may speed up
1374 the conversion as it would not try to rewrite the already convert
1375 raid1 profiles.
1376 Having just one profile is desired as this also clearly defines the
1378 profile of newly allocated block groups, otherwise this depends on in‐
1379 ternal allocation policy. When there are multiple profiles present, the
1380 order of selection is RAID6, RAID5, RAID10, RAID1, RAID0 as long as the
1381 device number constraints are satisfied.
1382 Commands that print the warning were chosen so they're brought to user
1384 attention when the filesystem state is being changed in that regard.
1385 This is: device add, device delete, balance cancel, balance pause. Com‐
1386 mands that report space usage: filesystem df, device usage. The command
1387 filesystem usage provides a line in the overall summary:
1388 Multiple profiles: yes (data, metadata)
1390
1392 The COW mechanism and multiple devices under one hood enable an inter‐
1393 esting concept, called a seeding device: extending a read-only filesys‐
1394 tem on a device with another device that captures all writes. For exam‐
1395 ple imagine an immutable golden image of an operating system enhanced
1396 with another device that allows to use the data from the golden image
1397 and normal operation. This idea originated on CD-ROMs with base OS and
1398 allowing to use them for live systems, but this became obsolete. There
1399 are technologies providing similar functionality, like unionmount,
1400 overlayfs or qcow2 image snapshot.
1401 The seeding device starts as a normal filesystem, once the contents is
1403 ready, btrfstune -S 1 is used to flag it as a seeding device. Mounting
1404 such device will not allow any writes, except adding a new device by
1405 btrfs device add. Then the filesystem can be remounted as read-write.
1406 Given that the filesystem on the seeding device is always recognized as
1408 read-only, it can be used to seed multiple filesystems from one device
1409 at the same time. The UUID that is normally attached to a device is au‐
1410 tomatically changed to a random UUID on each mount.
1411 Once the seeding device is mounted, it needs the writable device. After
1413 adding it, something like remount -o remount,rw /path makes the
1414 filesystem at /path ready for use. The simplest use case is to throw
1415 away all changes by unmounting the filesystem when convenient.
1416 Alternatively, deleting the seeding device from the filesystem can turn
1418 it into a normal filesystem, provided that the writable device can also
1419 contain all the data from the seeding device.
1420 The seeding device flag can be cleared again by btrfstune -f -S 0, eg.
1422 allowing to update with newer data but please note that this will in‐
1423 validate all existing filesystems that use this particular seeding de‐
1424 vice. This works for some use cases, not for others, and the forcing
1425 flag to the command is mandatory to avoid accidental mistakes.
1426 Example how to create and use one seeding device:
1428 # mkfs.btrfs /dev/sda
1430 # mount /dev/sda /mnt/mnt1
1431 ... fill mnt1 with data
1432 # umount /mnt/mnt1
1433 # btrfstune -S 1 /dev/sda
1435 # mount /dev/sda /mnt/mnt1
1437 # btrfs device add /dev/sdb /mnt/mnt1
1438 # mount -o remount,rw /mnt/mnt1
1439 ... /mnt/mnt1 is now writable
1440 Now /mnt/mnt1 can be used normally. The device /dev/sda can be mounted
1442 again with a another writable device:
1443 # mount /dev/sda /mnt/mnt2
1445 # btrfs device add /dev/sdc /mnt/mnt2
1446 # mount -o remount,rw /mnt/mnt2
1447 ... /mnt/mnt2 is now writable
1448 The writable device (/dev/sdb) can be decoupled from the seeding device
1450 and used independently:
1451 # btrfs device delete /dev/sda /mnt/mnt1
1453 As the contents originated in the seeding device, it's possible to turn
1455 /dev/sdb to a seeding device again and repeat the whole process.
1456 A few things to note:
1458 • it's recommended to use only single device for the seeding device, it
1460 works for multiple devices but the single profile must be used in or‐
1461 der to make the seeding device deletion work
1462 • block group profiles single and dup support the use cases above
1464 • the label is copied from the seeding device and can be changed by
1466 btrfs filesystem label
1467 • each new mount of the seeding device gets a new random UUID
1469 Chained seeding devices
1471 Though it's not recommended and is rather an obscure and untested use
1472 case, chaining seeding devices is possible. In the first example, the
1473 writable device /dev/sdb can be turned onto another seeding device
1474 again, depending on the unchanged seeding device /dev/sda. Then using
1475 /dev/sdb as the primary seeding device it can be extended with another
1476 writable device, say /dev/sdd, and it continues as before as a simple
1477 tree structure on devices.
1478 # mkfs.btrfs /dev/sda
1480 # mount /dev/sda /mnt/mnt1
1481 ... fill mnt1 with data
1482 # umount /mnt/mnt1
1483 # btrfstune -S 1 /dev/sda
1485 # mount /dev/sda /mnt/mnt1
1487 # btrfs device add /dev/sdb /mnt/mnt1
1488 # mount -o remount,rw /mnt/mnt1
1489 ... /mnt/mnt1 is now writable
1490 # umount /mnt/mnt1
1491 # btrfstune -S 1 /dev/sdb
1493 # mount /dev/sdb /mnt/mnt1
1495 # btrfs device add /dev/sdc /mnt
1496 # mount -o remount,rw /mnt/mnt1
1497 ... /mnt/mnt1 is now writable
1498 # umount /mnt/mnt1
1499 As a result we have:
1501 • sda is a single seeding device, with its initial contents
1503 • sdb is a seeding device but requires sda, the contents are from the
1505 time when sdb is made seeding, ie. contents of sda with any later
1506 changes
1507 • sdc last writable, can be made a seeding one the same way as was sdb,
1509 preserving its contents and depending on sda and sdb
1510 As long as the seeding devices are unmodified and available, they can
1512 be used to start another branch.
1513
1515 The RAID56 feature provides striping and parity over several devices,
1516 same as the traditional RAID5/6. There are some implementation and de‐
1517 sign deficiencies that make it unreliable for some corner cases and the
1518 feature should not be used in production, only for evaluation or test‐
1519 ing. The power failure safety for metadata with RAID56 is not 100%.
1520 Metadata
1522 Do not use raid5 nor raid6 for metadata. Use raid1 or raid1c3 respec‐
1523 tively.
1524 The substitute profiles provide the same guarantees against loss of 1
1526 or 2 devices, and in some respect can be an improvement. Recovering
1527 from one missing device will only need to access the remaining 1st or
1528 2nd copy, that in general may be stored on some other devices due to
1529 the way RAID1 works on btrfs, unlike on a striped profile (similar to
1530 raid0) that would need all devices all the time.
1531 The space allocation pattern and consumption is different (eg. on N de‐
1533 vices): for raid5 as an example, a 1GiB chunk is reserved on each de‐
1534 vice, while with raid1 there's each 1GiB chunk stored on 2 devices. The
1535 consumption of each 1GiB of used metadata is then N * 1GiB for vs 2 *
1536 1GiB. Using raid1 is also more convenient for balancing/converting to
1537 other profile due to lower requirement on the available chunk space.
1538 Missing/incomplete support
1540 When RAID56 is on the same filesystem with different raid profiles, the
1541 space reporting is inaccurate, eg. df, btrfs filesystem df or btrfs
1542 filesystem usage. When there's only a one profile per block group type
1543 (eg. raid5 for data) the reporting is accurate.
1544 When scrub is started on a RAID56 filesystem, it's started on all de‐
1546 vices that degrade the performance. The workaround is to start it on
1547 each device separately. Due to that the device stats may not match the
1548 actual state and some errors might get reported multiple times.
1549 The write hole problem. An unclean shutdown could leave a partially
1551 written stripe in a state where the some stripe ranges and the parity
1552 are from the old writes and some are new. The information which is
1553 which is not tracked. Write journal is not implemented. Alternatively a
1554 full read-modify-write would make sure that a full stripe is always
1555 written, avoiding the write hole completely, but performance in that
1556 case turned out to be too bad for use.
1557 The striping happens on all available devices (at the time the chunks
1559 were allocated), so in case a new device is added it may not be uti‐
1560 lized immediately and would require a rebalance. A fixed configured
1561 stripe width is not implemented.
1562
1564 Storage model
1565 A storage model is a model that captures key physical aspects of data
1566 structure in a data store. A filesystem is the logical structure orga‐
1567 nizing data on top of the storage device.
1568 The filesystem assumes several features or limitations of the storage
1570 device and utilizes them or applies measures to guarantee reliability.
1571 BTRFS in particular is based on a COW (copy on write) mode of writing,
1572 ie. not updating data in place but rather writing a new copy to a dif‐
1573 ferent location and then atomically switching the pointers.
1574 In an ideal world, the device does what it promises. The filesystem as‐
1576 sumes that this may not be true so additional mechanisms are applied to
1577 either detect misbehaving hardware or get valid data by other means.
1578 The devices may (and do) apply their own detection and repair mecha‐
1579 nisms but we won't assume any.
1580 The following assumptions about storage devices are considered (sorted
1582 by importance, numbers are for further reference):
1583 1. atomicity of reads and writes of blocks/sectors (the smallest unit
1585 of data the device presents to the upper layers)
1586 2. there's a flush command that instructs the device to forcibly order
1588 writes before and after the command; alternatively there's a barrier
1589 command that facilitates the ordering but may not flush the data
1590 3. data sent to write to a given device offset will be written without
1592 further changes to the data and to the offset
1593 4. writes can be reordered by the device, unless explicitly serialized
1595 by the flush command
1596 5. reads and writes can be freely reordered and interleaved
1598 The consistency model of BTRFS builds on these assumptions. The logical
1600 data updates are grouped, into a generation, written on the device, se‐
1601 rialized by the flush command and then the super block is written end‐
1602 ing the generation. All logical links among metadata comprising a con‐
1603 sistent view of the data may not cross the generation boundary.
1604 When things go wrong
1606 No or partial atomicity of block reads/writes (1)
1607 • Problem: a partial block contents is written (torn write), eg. due to
1609 a power glitch or other electronics failure during the read/write
1610 • Detection: checksum mismatch on read
1612 • Repair: use another copy or rebuild from multiple blocks using some
1614 encoding scheme
1615 The flush command does not flush (2)
1617 This is perhaps the most serious problem and impossible to mitigate by
1619 filesystem without limitations and design restrictions. What could hap‐
1620 pen in the worst case is that writes from one generation bleed to an‐
1621 other one, while still letting the filesystem consider the generations
1622 isolated. Crash at any point would leave data on the device in an in‐
1623 consistent state without any hint what exactly got written, what is
1624 missing and leading to stale metadata link information.
1625 Devices usually honor the flush command, but for performance reasons
1627 may do internal caching, where the flushed data are not yet persis‐
1628 tently stored. A power failure could lead to a similar scenario as
1629 above, although it's less likely that later writes would be written be‐
1630 fore the cached ones. This is beyond what a filesystem can take into
1631 account. Devices or controllers are usually equipped with batteries or
1632 capacitors to write the cache contents even after power is cut. (Bat‐
1633 tery backed write cache)
1634 Data get silently changed on write (3)
1636 Such thing should not happen frequently, but still can happen spuri‐
1638 ously due the complex internal workings of devices or physical effects
1639 of the storage media itself.
1640 • Problem: while the data are written atomically, the contents get
1642 changed
1643 • Detection: checksum mismatch on read
1645 • 'Repair*: use another copy or rebuild from multiple blocks using some
1647 encoding scheme
1648 Data get silently written to another offset (3)
1650 This would be another serious problem as the filesystem has no informa‐
1652 tion when it happens. For that reason the measures have to be done
1653 ahead of time. This problem is also commonly called 'ghost write'.
1654 The metadata blocks have the checksum embedded in the blocks, so a cor‐
1656 rect atomic write would not corrupt the checksum. It's likely that af‐
1657 ter reading such block the data inside would not be consistent with the
1658 rest. To rule that out there's embedded block number in the metadata
1659 block. It's the logical block number because this is what the logical
1660 structure expects and verifies.
1661 The following is based on information publicly available, user feed‐
1663 back, community discussions or bug report analyses. It's not complete
1664 and further research is encouraged when in doubt.
1665 Main memory
1667 The data structures and raw data blocks are temporarily stored in com‐
1668 puter memory before they get written to the device. It is critical that
1669 memory is reliable because even simple bit flips can have vast conse‐
1670 quences and lead to damaged structures, not only in the filesystem but
1671 in the whole operating system.
1672 Based on experience in the community, memory bit flips are more common
1674 than one would think. When it happens, it's reported by the
1675 tree-checker or by a checksum mismatch after reading blocks. There are
1676 some very obvious instances of bit flips that happen, e.g. in an or‐
1677 dered sequence of keys in metadata blocks. We can easily infer from the
1678 other data what values get damaged and how. However, fixing that is not
1679 straightforward and would require cross-referencing data from the en‐
1680 tire filesystem to see the scope.
1681 If available, ECC memory should lower the chances of bit flips, but
1683 this type of memory is not available in all cases. A memory test should
1684 be performed in case there's a visible bit flip pattern, though this
1685 may not detect a faulty memory module because the actual load of the
1686 system could be the factor making the problems appear. In recent years
1687 attacks on how the memory modules operate have been demonstrated
1688 ('rowhammer') achieving specific bits to be flipped. While these were
1689 targeted, this shows that a series of reads or writes can affect unre‐
1690 lated parts of memory.
1691 Further reading:
1693 • https://en.wikipedia.org/wiki/Row_hammer
1695 What to do:
1697 • run memtest, note that sometimes memory errors happen only when the
1699 system is under heavy load that the default memtest cannot trigger
1700 • memory errors may appear as filesystem going read-only due to "pre
1702 write" check, that verify meta data before they get written but fail
1703 some basic consistency checks
1704 Direct memory access (DMA)
1706 Another class of errors is related to DMA (direct memory access) per‐
1707 formed by device drivers. While this could be considered a software er‐
1708 ror, the data transfers that happen without CPU assistance may acciden‐
1709 tally corrupt other pages. Storage devices utilize DMA for performance
1710 reasons, the filesystem structures and data pages are passed back and
1711 forth, making errors possible in case page life time is not properly
1712 tracked.
1713 There are lots of quirks (device-specific workarounds) in Linux kernel
1715 drivers (regarding not only DMA) that are added when found. The quirks
1716 may avoid specific errors or disable some features to avoid worse prob‐
1717 lems.
1718 What to do:
1720 • use up-to-date kernel (recent releases or maintained long term sup‐
1722 port versions)
1723 • as this may be caused by faulty drivers, keep the systems up-to-date
1725 Rotational disks (HDD)
1727 Rotational HDDs typically fail at the level of individual sectors or
1728 small clusters. Read failures are caught on the levels below the
1729 filesystem and are returned to the user as EIO - Input/output error.
1730 Reading the blocks repeatedly may return the data eventually, but this
1731 is better done by specialized tools and filesystem takes the result of
1732 the lower layers. Rewriting the sectors may trigger internal remapping
1733 but this inevitably leads to data loss.
1734 Disk firmware is technically software but from the filesystem perspec‐
1736 tive is part of the hardware. IO requests are processed, and caching or
1737 various other optimizations are performed, which may lead to bugs under
1738 high load or unexpected physical conditions or unsupported use cases.
1739 Disks are connected by cables with two ends, both of which can cause
1741 problems when not attached properly. Data transfers are protected by
1742 checksums and the lower layers try hard to transfer the data correctly
1743 or not at all. The errors from badly-connecting cables may manifest as
1744 large amount of failed read or write requests, or as short error bursts
1745 depending on physical conditions.
1746 What to do:
1748 • check smartctl for potential issues
1750 Solid state drives (SSD)
1752 The mechanism of information storage is different from HDDs and this
1753 affects the failure mode as well. The data are stored in cells grouped
1754 in large blocks with limited number of resets and other write con‐
1755 straints. The firmware tries to avoid unnecessary resets and performs
1756 optimizations to maximize the storage media lifetime. The known tech‐
1757 niques are deduplication (blocks with same fingerprint/hash are mapped
1758 to same physical block), compression or internal remapping and garbage
1759 collection of used memory cells. Due to the additional processing there
1760 are measures to verity the data e.g. by ECC codes.
1761 The observations of failing SSDs show that the whole electronic fails
1763 at once or affects a lot of data (eg. stored on one chip). Recovering
1764 such data may need specialized equipment and reading data repeatedly
1765 does not help as it's possible with HDDs.
1766 There are several technologies of the memory cells with different char‐
1768 acteristics and price. The lifetime is directly affected by the type
1769 and frequency of data written. Writing "too much" distinct data (e.g.
1770 encrypted) may render the internal deduplication ineffective and lead
1771 to a lot of rewrites and increased wear of the memory cells.
1772 There are several technologies and manufacturers so it's hard to de‐
1774 scribe them but there are some that exhibit similar behaviour:
1775 • expensive SSD will use more durable memory cells and is optimized for
1777 reliability and high load
1778 • cheap SSD is projected for a lower load ("desktop user") and is opti‐
1780 mized for cost, it may employ the optimizations and/or extended error
1781 reporting partially or not at all
1782 It's not possible to reliably determine the expected lifetime of an SSD
1784 due to lack of information about how it works or due to lack of reli‐
1785 able stats provided by the device.
1786 Metadata writes tend to be the biggest component of lifetime writes to
1788 a SSD, so there is some value in reducing them. Depending on the device
1789 class (high end/low end) the features like DUP block group profiles may
1790 affect the reliability in both ways:
1791 • high end are typically more reliable and using 'single' for data and
1793 metadata could be suitable to reduce device wear
1794 • low end could lack ability to identify errors so an additional redun‐
1796 dancy at the filesystem level (checksums, DUP) could help
1797 Only users who consume 50 to 100% of the SSD's actual lifetime writes
1799 need to be concerned by the write amplification of btrfs DUP metadata.
1800 Most users will be far below 50% of the actual lifetime, or will write
1801 the drive to death and discover how many writes 100% of the actual
1802 lifetime was. SSD firmware often adds its own write multipliers that
1803 can be arbitrary and unpredictable and dependent on application behav‐
1804 ior, and these will typically have far greater effect on SSD lifespan
1805 than DUP metadata. It's more or less impossible to predict when a SSD
1806 will run out of lifetime writes to within a factor of two, so it's hard
1807 to justify wear reduction as a benefit.
1808 Further reading:
1810 • https://www.snia.org/educational-library/ssd-and-deduplication-end-spinning-disk-2012
1812 • https://www.snia.org/educational-library/realities-solid-state-storage-2013-2013
1814 • https://www.snia.org/educational-library/ssd-performance-primer-2013
1816 • https://www.snia.org/educational-library/how-controllers-maximize-ssd-life-2013
1818 What to do:
1820 • run smartctl or self-tests to look for potential issues
1822 • keep the firmware up-to-date
1824 NVM express, non-volatile memory (NVMe)
1826 NVMe is a type of persistent memory usually connected over a system bus
1827 (PCIe) or similar interface and the speeds are an order of magnitude
1828 faster than SSD. It is also a non-rotating type of storage, and is not
1829 typically connected by a cable. It's not a SCSI type device either but
1830 rather a complete specification for logical device interface.
1831 In a way the errors could be compared to a combination of SSD class and
1833 regular memory. Errors may exhibit as random bit flips or IO failures.
1834 There are tools to access the internal log (nvme log and nvme-cli) for
1835 a more detailed analysis.
1836 There are separate error detection and correction steps performed e.g.
1838 on the bus level and in most cases never making in to the filesystem
1839 level. Once this happens it could mean there's some systematic error
1840 like overheating or bad physical connection of the device. You may want
1841 to run self-tests (using smartctl).
1842 • https://en.wikipedia.org/wiki/NVM_Express
1844 • https://www.smartmontools.org/wiki/NVMe_Support
1846 Drive firmware
1848 Firmware is technically still software but embedded into the hardware.
1849 As all software has bugs, so does firmware. Storage devices can update
1850 the firmware and fix known bugs. In some cases the it's possible to
1851 avoid certain bugs by quirks (device-specific workarounds) in Linux
1852 kernel.
1853 A faulty firmware can cause wide range of corruptions from small and
1855 localized to large affecting lots of data. Self-repair capabilities may
1856 not be sufficient.
1857 What to do:
1859 • check for firmware updates in case there are known problems, note
1861 that updating firmware can be risky on itself
1862 • use up-to-date kernel (recent releases or maintained long term sup‐
1864 port versions)
1865 SD flash cards
1867 There are a lot of devices with low power consumption and thus using
1868 storage media based on low power consumption too, typically flash mem‐
1869 ory stored on a chip enclosed in a detachable card package. An improp‐
1870 erly inserted card may be damaged by electrical spikes when the device
1871 is turned on or off. The chips storing data in turn may be damaged per‐
1872 manently. All types of flash memory have a limited number of rewrites,
1873 so the data are internally translated by FTL (flash translation layer).
1874 This is implemented in firmware (technically a software) and prone to
1875 bugs that manifest as hardware errors.
1876 Adding redundancy like using DUP profiles for both data and metadata
1878 can help in some cases but a full backup might be the best option once
1879 problems appear and replacing the card could be required as well.
1880 Hardware as the main source of filesystem corruptions
1882 If you use unreliable hardware and don't know about that, don't blame
1883 the filesystem when it tells you.
1884
1886 acl(5), btrfs(8), chattr(1), fstrim(8), ioctl(2), mkfs.btrfs(8),
1887 mount(8), swapon(8)
1888
1890 2022
18915.18 May 25, 2022 BTRFS(5)