1SnapRAID Backup For Disk ArraGyesn(e1r)al CommandsSMnaanpuRaAlID Backup For Disk Arrays(1)
2
6 snapraid - SnapRAID Backup For Disk Arrays
7
9 snapraid [-c, --conf CONFIG]
10 [-f, --filter PATTERN] [-d, --filter-disk NAME]
11 [-m, --filter-missing] [-e, --filter-error]
12 [-a, --audit-only] [-h, --pre-hash] [-i, --import DIR]
13 [-p, --plan PERC|bad|new|full]
14 [-o, --older-than DAYS] [-l, --log FILE]
15 [-Z, --force-zero] [-E, --force-empty]
16 [-U, --force-uuid] [-D, --force-device]
17 [-N, --force-nocopy] [-F, --force-full]
18 [-R, --force-realloc]
19 [-S, --start BLKSTART] [-B, --count BLKCOUNT]
20 [-L, --error-limit NUMBER]
21 [-v, --verbose] [-q, --quiet]
22 status|smart|up|down|diff|sync|scrub|fix|check|list|dup
23 |pool|devices|touch|rehash
24 snapraid [-V, --version] [-H, --help] [-C, --gen-conf CONTENT]
26 SnapRAID is a backup program for disk arrays. It stores parity informa‐
27 tion of your data and it recovers from up to six disk failures.
28 SnapRAID is mainly targeted for a home media center, with a lot of big
30 files that rarely change.
31 Beside the ability to recover from disk failures, other features of
33 SnapRAID are:
34 · All your data is hashed to ensure data integrity and to avoid
35 silent corruption.
36 · If the failed disks are too many to allow a recovery, you lose
37 the data only on the failed disks. All the data in the other
38 disks is safe.
39 · If you accidentally delete some files in a disk, you can recover
40 them.
41 · You can start with already filled disks.
42 · The disks can have different sizes.
43 · You can add disks at any time.
44 · It doesn´t lock-in your data. You can stop using SnapRAID at any
45 time without the need to reformat or move data.
46 · To access a file, a single disk needs to spin, saving power and
47 producing less noise.
48 The official site of SnapRAID is:
50 http://www.snapraid.it/
53 SnapRAID is in between a RAID and a Backup program trying to get the
54 best benefits of them. Although it also has some limitations that you
55 should consider before using it.
56 The main one is that if a disk fails, and you haven´t recently synced,
58 you may be unable to do a complete recover. More specifically, you may
59 be unable to recover up to the size of the amount of the changed or
60 deleted files from the last sync operation. This happens even if the
61 files changed or deleted are not in the failed disk. This is the reason
62 because SnapRAID is better suited for data that rarely change.
63 Instead the new added files don´t prevent the recovering of the already
65 existing files. You may only lose the just added files, if they are on
66 the failed disk.
67 Other limitations are:
69 · You have different file-systems for each disk. Using a RAID you
70 have only a big file-system.
71 · It doesn´t stripe data. With RAID you get a speed boost with
72 striping.
73 · It doesn´t support real-time recovery. With RAID you do not
74 have to stop working when a disk fails.
75 · It´s able to recover damages only from a limited number of
76 disks. With a Backup you are able to recover from a complete
77 failure of the whole disk array.
78 · Only file, time-stamps, symlinks and hardlinks are saved. Per‐
79 missions, ownership and extended attributes are not saved.
80
82 To use SnapRAID you need to first select one disk of your disk array to
83 dedicate at the "parity" information. With one disk for parity you will
84 be able to recover from a single disk failure, like RAID5.
85 If you want to be able to recover from more disk failures, like RAID6,
87 you must reserve additional disks for parity. Any additional parity
88 disk allow to recover from one more disk failure.
89 As parity disks, you have to pick the biggest disks in the array, as
91 the parity information may grow in size as the biggest data disk in the
92 array.
93 These disks will be dedicated to store the "parity" files. You should
95 not store your data in them.
96 Then you have to define the "data" disks that you want to protect with
98 SnapRAID. The protection is more effective if these disks contain data
99 that rarely change. For this reason it´s better to DO NOT include the
100 Windows C:\ disk, or the Unix /home, /var and /tmp disks.
101 The list of files is saved in the "content" files, usually stored in
103 the data, parity or boot disks. These files contain the details of
104 your backup, with all the check-sums to verify its integrity. The
105 "content" file is stored in multiple copies, and each one must be in a
106 different disk, to ensure that in even in case of multiple disk fail‐
107 ures at least one copy is available.
108 For example, suppose that you are interested only at one parity level
110 of protection, and that your disks are present in:
111 /mnt/diskp <- selected disk for parity
113 /mnt/disk1 <- first disk to protect
114 /mnt/disk2 <- second disk to protect
115 /mnt/disk3 <- third disk to protect
116 you have to create the configuration file /etc/snapraid.conf with the
117 following options:
118 parity /mnt/diskp/snapraid.parity
120 content /var/snapraid/snapraid.content
121 content /mnt/disk1/snapraid.content
122 content /mnt/disk2/snapraid.content
123 data d1 /mnt/disk1/
124 data d2 /mnt/disk2/
125 data d3 /mnt/disk3/
126 If you are in Windows, you should use the Windows path format, with
127 drive letters and backslashes instead of slashes.
128 parity E:\snapraid.parity
130 content C:\snapraid\snapraid.content
131 content F:\array\snapraid.content
132 content G:\array\snapraid.content
133 data d1 F:\array\
134 data d2 G:\array\
135 data d3 H:\array\
136 If you have many disks, and you run out of drive letters, you can mount
137 disks directly in sub folders. See:
138 https://www.google.com/search?q=Windows+mount+point
140 At this point you are ready to start the "sync" command to build the
141 parity information.
142 snapraid sync
144 This process may take some hours the first time, depending on the size
145 of the data already present in the disks. If the disks are empty the
146 process is immediate.
147 You can stop it at any time pressing Ctrl+C, and at the next run it
149 will start where interrupted.
150 When this command completes, your data is SAFE.
152 Now you can start using your array as you like, and periodically update
154 the parity information running the "sync" command.
155 Scrubbing
157 To periodically check the data and parity for errors, you can run the
158 "scrub" command.
159 snapraid scrub
161 This command verifies the data in your array comparing it with the hash
162 computed in the "sync" command.
163 Every run of the command checks about the 8% of the array, but not data
165 already scrubbed in the previous 10 days. You can use the -p, --plan
166 option to specify a different amount, and the -o, --older-than option
167 to specify a different age in days. For example, to check 5% of the
168 array older than 20 days use:
169 snapraid -p 5 -o 20 scrub
171 If during the process, silent or input/output errors are found, the
172 corresponding blocks are marked as bad in the "content" file, and
173 listed in the "status" command.
174 snapraid status
176 To fix them, you can use the "fix" command filtering for bad blocks
177 with the -e, --filter-error options:
178 snapraid -e fix
180 At the next "scrub" the errors will disappear from the "status" report
181 if really fixed. To make it fast, you can use -p bad to scrub only
182 blocks marked as bad.
183 snapraid -p bad scrub
185 Take care that running "scrub" on a not synced array may result in
186 errors caused by removed or modified files. These errors are reported
187 in the "scrub" result, but related blocks are not marked as bad.
188 Pooling
190 To have all the files in your array shown in the same directory tree,
191 you can enable the "pooling" feature. It consists in creating a
192 read-only virtual view of all the files in your array using symbolic
193 links.
194 You can configure the "pooling" directory in the configuration file
196 with:
197 pool /pool
199 or, if you are in Windows, with:
200 pool C:\pool
202 and then run the "pool" command to create or update the virtual view.
203 snapraid pool
205 If you are using a Unix platform and you want to share such directory
206 in the network to either Windows or Unix machines, you should add to
207 your /etc/samba/smb.conf the following options:
208 # In the global section of smb.conf
210 unix extensions = no
211 # In the share section of smb.conf
212 [pool]
213 comment = Pool
214 path = /pool
215 read only = yes
216 guest ok = yes
217 wide links = yes
218 follow symlinks = yes
219 In Windows the same sharing operation is not so straightforward,
220 because Windows shares the symbolic links as they are, and that
221 requires the network clients to resolve them remotely.
222 To make it working, besides sharing in the network the pool directory,
224 you must also share all the disks independently, using as share points
225 the disk names as defined in the configuration file. You must also
226 specify in the "share" option of the configure file, the Windows UNC
227 path that remote clients needs to use to access such shared disks.
228 For example, operating from a server named "darkstar", you can use the
230 options:
231 data d1 F:\array\
233 data d2 G:\array\
234 data d3 H:\array\
235 pool C:\pool
236 share \\darkstar
237 and share the following dirs in the network:
238 \\darkstar\pool -> C:\pool
240 \\darkstar\d1 -> F:\array
241 \\darkstar\d2 -> G:\array
242 \\darkstar\d3 -> H:\array
243 to allow remote clients to access all the files at \\darkstar\\pool.
244 You may also need to configure remote clients enabling access at remote
246 symlinks with the command:
247 fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
249 Undeleting
250 SnapRAID is more like a backup program than a RAID system, and it can
251 be used to restore or undelete files to their previous state using the
252 -f, --filter option :
253 snapraid fix -f FILE
255 or for a directory:
256 snapraid fix -f DIR/
258 You can also use it to recover only accidentally deleted files inside a
259 directory using the -m, --filter-missing option, that restores only
260 missing files, leaving untouched all the others.
261 snapraid fix -m -f DIR/
263 Or to recover all the deleted files in all the drives with:
264 snapraid fix -m
266 Recovering
267 The worst happened, and you lost one or more disks!
268 DO NOT PANIC! You will be able to recover them!
270 The first thing you have to do is to avoid further changes at your disk
272 array. Disable any remote connection to it, any scheduled process,
273 including any scheduled SnapRAID nightly sync or scrub.
274 Then proceed with the following steps.
276 STEP 1 -> Reconfigure
278 You need some space to recover, even better if you already have addi‐
279 tional spare disks, but in case, also an external USB or remote disk is
280 enough.
281 Change the SnapRAID configuration file to make the "data" or "parity"
283 option of the failed disk to point to the place where you have enough
284 empty space to recover the files.
285 For example, if you have that disk "d1" failed, you can change from:
287 data d1 /mnt/disk1/
289 to:
290 data d1 /mnt/new_spare_disk/
292 If the disk to recover is a parity disk, change the appropriate "par‐
293 ity" option. If you have more broken disks, change all their configu‐
294 ration options.
295 STEP 2 -> Fix
297 Run the fix command, storing the log in an external file with:
298 snapraid -d NAME -l fix.log fix
300 Where NAME is the name of the disk, like "d1" as in our previous exam‐
301 ple. In case the disk to recover is a parity disk, use the "parity",
302 "2-parity" names. If you have more broken disks, use multiple -d
303 options to specify all of them.
304 This command will take a long time.
306 Take care that you need also few gigabytes free to store the fix.log
308 file. Run it from a disk with some free space.
309 Now you have recovered all the recoverable. If some file is partially
311 or totally unrecoverable, it will be renamed adding the ".unrecover‐
312 able" extension.
313 You can get a detailed list of all the unrecoverable blocks in the
315 fix.log file checking all the lines starting with "unrecoverable:"
316 If you are not satisfied of the recovering, you can retry it as many
318 time you wish.
319 For example, if you have removed files from the array after the last
321 "sync", this may result in some other files not recovered. In this
322 case, you can retry the "fix" using the -i, --import option, specifying
323 where these files are now, to include them again in the recovering
324 process.
325 If you are satisfied of the recovering, you can now proceed further,
327 but take care that after syncing you cannot retry the "fix" command
328 anymore!
329 STEP 3 -> Check
331 As paranoid check, you can now run a "check" command to ensure that
332 everything is OK on the recovered disk.
333 snapraid -d NAME -a check
335 Where NAME is the name of the disk, like "d1" as in our previous exam‐
336 ple.
337 The options -d and -a tell SnapRAID to check only the specified disk,
339 and ignore all the parity data.
340 This command will take a long time, but if you are not paranoid, you
342 can skip it.
343 STEP 4 -> Sync
345 Run the "sync" command to resynchronize the array with the new disk.
346 snapraid sync
348 If everything is recovered, this command is immediate.
349
351 SnapRAID provides a few simple commands that allow to:
352 · Prints the status of the array -> "status"
353 · Controls the disks -> "smart", "up", "down"
354 · Makes a backup/snapshot -> "sync"
355 · Periodically checks data -> "scrub"
356 · Restore the last backup/snapshot -> "fix".
357 Take care that the commands have to be written in lower case.
359 status
361 Prints a summary of the state of the disk array.
362 It includes information about the parity fragmentation, how old are the
364 blocks without checking, and all the recorded silent errors encountered
365 while scrubbing.
366 Note that the information presented refers at the latest time you run
368 "sync". Later modifications are not taken into account.
369 If bad blocks were detected, their block numbers are listed. To fix
371 them, you can use the "fix -e" command.
372 It also shows a graph representing the the last time each block was
374 scrubbed or synced. Scrubbed blocks are shown with ´*´, blocks synced
375 but not yet scrubbed with ´o´.
376 Nothing is modified.
378 smart
380 Prints a SMART report of all the disks of the array.
381 It includes an estimation of the probability of failure in the next
383 year allowing to plan maintenance replacements of the disks that show
384 suspicious attributes.
385 This probability estimation obtained correlating the SMART attributes
387 of the disks, with the Backblaze data available at:
388 https://www.backblaze.com/hard-drive-test-data.html
390 If SMART reports that a disk is failing, "FAIL" or "PREFAIL" is printed
391 for that disk, and SnapRAID returns with an error. In this case an
392 immediate replacement of the disk is highly recommended.
393 Other possible strings are:
395 logfail In the past some attributes were lower than the threshold.
396 logerr The device error log contains errors.
397 selferr The device self-test log contains errors.
398 If the -v, --verbose option is specified a deeper statistical analysis
400 is provided. This analysis can help you to decide if you need more or
401 less parity.
402 This command uses the "smartctl" tool, and it´s equivalent to run
404 "smartctl -a" on all the devices.
405 If your devices are not auto-detected correctly, you can configure a
407 custom command using the "smartctl" option in the configuration file.
408 Nothing is modified.
410 up
412 Spins up all the disks of the array.
413 You can spin-up only some specific disks using the -d, --filter-disk
415 option.
416 Take care that spinniup-up all the disks at the same time needs a lot
418 of power. Ensure that your power-supply can sustain that.
419 Nothing is modified.
421 down
423 Spins down all the disks of the array.
424 This command uses the "smartctl" tool, and it´s equivalent to run
426 "smartctl -s standby,now" on all the devices.
427 You can spin-down only some specific disks using the -d, --filter-disk
429 option.
430 Nothing is modified.
432 diff
434 Lists all the files modified from the last "sync" that need to have
435 their parity data recomputed.
436 This command doesn´t check the file data, but only the file time-stamp
438 size and inode.
439 At the end of the command, you´ll get a summary of the file changes
441 grouped by:
442 equal Files equal at before.
443 added Files added that were not present before.
444 removed Files removed.
445 updated Files with a different size or time-stamp, meaning that
446 they were modified.
447 moved Files moved to a different directory of the same disk. They
448 are identified by having the same name, size, time-stamp and
449 inode, but different directory.
450 copied Files copied in the same or different disk. Note that if in
451 true they are moved to a different disk, you´ll also have them
452 counted in "removed". They are identified by having the same
453 name, size, and time-stamp. But if the sub-second time-stamp is
454 zero, then the full path should match, and not only the name.
455 restored Files with a different inode but with name, size and
456 time-stamp matching. These are usually files restored after
457 being deleted.
458 If a "sync" is required, the process return code is 2, instead of the
460 default 0. The return code 1 is instead for a generic error condition.
461 Nothing is modified.
463 sync
465 Updates the parity information. All the modified files in the disk
466 array are read, and the corresponding parity data is updated.
467 You can stop this process at any time pressing Ctrl+C, without losing
469 the work already done. At the next run the "sync" process will start
470 where interrupted.
471 If during the process, silent or input/output errors are found, the
473 corresponding blocks are marked as bad.
474 Files are identified by path and/or inode and checked by size and
476 time-stamp. If the file size or time-stamp are different, the parity
477 data is recomputed for the whole file. If the file is moved or renamed
478 in the same disk, keeping the same inode, the parity is not recomputed.
479 If the file is moved to another disk, the parity is recomputed, but the
480 previously computed hash information is kept.
481 The "content" and "parity" files are modified if necessary. The files
483 in the array are NOT modified.
484 scrub
486 Scrubs the array, checking for silent or input/output errors in data
487 and parity disks.
488 For each command invocation, about the 8% of the array is checked, but
490 nothing that was already scrubbed in the last 10 days. This means that
491 scrubbing once a week, every bit of data is checked at least one time
492 every three months.
493 You can define a different scrub plan or amount using the -p, --plan
495 option that takes as argument: bad - Scrub blocks marked bad. new -
496 Scrub just synced blocks not yet scrubbed. full - Scrub everything.
497 0-100 - Scrub the exact percentage of blocks.
498 If you specify a percentage amount, you can also use the -o,
500 --older-than option to define how old the block should be. The oldest
501 blocks are scrubbed first ensuring an optimal check. If instead you
502 want to scrub the just synced blocks, not yet scrubbed, you should use
503 the "-p new" option.
504 To get the details of the scrub status use the "status" command.
506 For any silent or input/output error found the corresponding blocks are
508 marked as bad in the "content" file. These bad blocks are listed in
509 "status", and can be fixed with "fix -e". After the fix, at the next
510 scrub they will be rechecked, and if found corrected, the bad mark will
511 be removed. To scrub only the bad blocks, you can use the "scrub -p
512 bad" command.
513 It´s recommended to run "scrub" only on a synced array, to avoid to
515 have reported error caused by unsynced data. These errors are recog‐
516 nized as not being silent errors, and the blocks are not marked as bad,
517 but such errors are reported in the output of the command.
518 Files are identified only by path, and not by inode.
520 The "content" file is modified to update the time of the last check of
522 each block, and to mark bad blocks. The "parity" files are NOT modi‐
523 fied. The files in the array are NOT modified.
524 fix
526 Fix all the files and the parity data.
527 All the files and the parity data are compared with the snapshot state
529 saved in the last "sync". If a difference is found, it´s reverted to
530 the stored snapshot.
531 The "fix" command doesn´t differentiate between errors and intentional
533 modifications. It unconditionally reverts the file state at the last
534 "sync".
535 If no other option is specified the full array is processed. Use the
537 filter options to select a subset of files or disks to operate on.
538 To only fix the blocks marked bad during "sync" and "scrub", use the
540 -e, --filter-error option. As difference from other filter options,
541 with this one the fixes are applied only to files that are not modified
542 from the the latest "sync".
543 All the files that cannot be fixed are renamed adding the ".unrecover‐
545 able" extension.
546 Before fixing, the full array is scanned to find any moved file, after
548 the last "sync" operation. These files are identified by their
549 time-stamp, ignoring their name and directory, and are used in the
550 recovering process if necessary. If you moved some of them outside the
551 array, you can use the -i, --import option to specify additional direc‐
552 tories to scan.
553 Files are identified only by path, and not by inode.
555 The "content" file is NOT modified. The "parity" files are modified if
557 necessary. The files in the array are modified if necessary.
558 check
560 Verify all the files and the parity data.
561 It works like "fix", but it only simulates a recovery and no change is
563 written in the array.
564 This command is mostly intended for manual verifications, like after a
566 recovery process or in other special conditions. For periodic and
567 scheduled checks uses "scrub".
568 If you use the -a, --audit-only option, only the file data is checked,
570 and the parity data is ignored for a faster run.
571 Files are identified only by path, and not by inode.
573 Nothing is modified.
575 list
577 Lists all the files contained in the array at the time of the last
578 "sync".
579 Nothing is modified.
581 dup
583 Lists all the duplicate files. Two files are assumed equal if their
584 hashes are matching. The file data is not read, but only the precom‐
585 puted hashes are used.
586 Nothing is modified.
588 pool
590 Creates or updates in the "pooling" directory a virtual view of all the
591 files of your disk array.
592 The files are not really copied here, but just linked using symbolic
594 links.
595 When updating, all the present symbolic links and empty subdirectories
597 are deleted and replaced with the new view of the array. Any other reg‐
598 ular file is left in place.
599 Nothing is modified outside the pool directory.
601 devices
603 Prints the low level devices used by the array.
604 This command prints the devices associations in place in the array, and
606 it´s mainly intended as a script interface.
607 The first two columns are the low level device id and path. The next
609 two columns are the high level device id and path. The latest column
610 if the disk name in the array.
611 In most cases you have one low level device for each disk in the array,
613 but in some more complex configurations, you may have multiple low
614 level devices used by a single disk in the array.
615 Nothing is modified.
617 touch
619 Sets arbitrarely the sub-second timestamp of all the files that have it
620 at zero.
621 This improves the SnapRAID capability to recognize moved and copied
623 files as it makes the timestamp almost unique, removing possible dupli‐
624 cates.
625 More specifically, if the sub-second timestamp is not zero, a moved or
627 copied file is identified as such if it matches the name, size and
628 timestamp. If instead the sub-second timestamp is zero, it´s considered
629 a copy only if it matches the full path, size and timestamp.
630 Note that the second precision timestamp is not modified, and all the
632 dates and times of your files will be maintained.
633 rehash
635 Schedules a rehash of the whole array.
636 This command changes the hash kind used, typically when upgrading from
638 a 32 bits system to a 64 bits one, to switch from MurmurHash3 to the
639 faster SpookyHash.
640 If you are already using the optimal hash, this command does nothing
642 and tells you that nothing has to be done.
643 The rehash isn´t done immediately, but it takes place progressively
645 during "sync" and "scrub".
646 You can get the rehash state using "status".
648 During the rehash, SnapRAID maintains full functionality, with the only
650 exception of "dup" not able to detect duplicated files using a differ‐
651 ent hash.
652
654 SnapRAID provides the following options:
655 -c, --conf CONFIG
657 Selects the configuration file to use. If not specified in Unix
658 it´s used the file "/usr/local/etc/snapraid.conf" if it exists,
659 or "/etc/snapraid.conf" otherwise. In Windows it´s used the
660 file "snapraid.conf" in the same directory of "snapraid.exe".
661 -f, --filter PATTERN
663 Filters the files to process in "check" and "fix". Only the
664 files matching the entered pattern are processed. This option
665 can be used many times. See the PATTERN section for more
666 details in the pattern specifications. In Unix, ensure to quote
667 globbing chars if used. This option can be used only with
668 "check" and "fix". Note that it cannot be used with "sync" and
669 "scrub", because they always process the whole array.
670 -d, --filter-disk NAME
672 Filters the disks to process in "check", "fix", "up" and "down".
673 You must specify a disk name as named in the configuration file.
674 You can also specify parity disks with the names: "parity",
675 "2-parity", "3-parity", ... to limit the operations a specific
676 parity disk. If you combine more --filter, --filter-disk and
677 --filter-missing options, only files matching all the set of
678 filters are selected. This option can be used many times. This
679 option can be used only with "check", "fix", "up" and "down".
680 Note that it cannot be used with "sync" and "scrub", because
681 they always process the whole array.
682 -m, --filter-missing
684 Filters the files to process in "check" and "fix". Only the
685 files missing/deleted from the array are processed. When used
686 with "fix", this is a kind of "undelete" command. If you com‐
687 bine more --filter, --filter-disk and --filter-missing options,
688 only files matching all the set of filters are selected. This
689 option can be used only with "check" and "fix". Note that it
690 cannot be used with "sync" and "scrub", because they always
691 process the whole array.
692 -e, --filter-error
694 Filters the blocks to process in "check" and "fix". It pro‐
695 cesses only the blocks marked with silent or input/output errors
696 during "sync" and "scrub", and listed in "status". This option
697 can be used only with "check" and "fix".
698 -p, --plan PERC|bad|new|full
700 Selects the scrub plan. If PERC is a numeric value from 0 to
701 100, it´s interpreted as the percentage of blocks to scrub.
702 Instead of a percentage, you can also specify a plan: "bad"
703 scrubs bad blocks, "new" the blocks not yet scrubbed, and "full"
704 for everything. This option can be used only with "scrub".
705 -o, --older-than DAYS
707 Selects the older the part of the array to process in "scrub".
708 DAYS is the minimum age in days for a block to be scrubbed,
709 default is 10. Blocks marked as bad are always scrubbed despite
710 this option. This option can be used only with "scrub".
711 -a, --audit-only
713 In "check" verifies the hash of the files without doing any kind
714 of check on the parity data. If you are interested in checking
715 only the file data this option can speedup a lot the checking
716 process. This option can be used only with "check".
717 -h, --pre-hash
719 In "sync" runs a preliminary hashing phase of all the new data
720 to have an additional verification before the parity computa‐
721 tion. Usually in "sync" no preliminary hashing is done, and the
722 new data is hashed just before the parity computation when it´s
723 read for the first time. Unfortunately, this process happens
724 when the system is under heavy load, with all disks spinning and
725 with a busy CPU. This is an extreme condition for the machine,
726 and if it has a latent hardware problem, it´s possible to
727 encounter silent errors what cannot be detected because the data
728 is not yet hashed. To avoid this risk, you can enable the
729 "pre-hash" mode and have all the data read two times to ensure
730 its integrity. This option also verifies the files moved inside
731 the array, to ensure that the move operation went successfully,
732 and in case to block the sync and to allow to run a fix opera‐
733 tion. This option can be used only with "sync".
734 -i, --import DIR
736 Imports from the specified directory any file that you deleted
737 from the array after the last "sync". If you still have such
738 files, they could be used by "check" and "fix" to improve the
739 recover process. The files are read also in subdirectories and
740 they are identified regardless of their name. This option can
741 be used only with "check" and "fix".
742 -Z, --force-zero
744 Forces the insecure operation of syncing a file with zero size
745 that before was not. If SnapRAID detects a such condition, it
746 stops proceeding unless you specify this option. This allows to
747 easily detect when after a system crash, some accessed files
748 were truncated. This is a possible condition in Linux with the
749 ext3/ext4 file-systems. This option can be used only with
750 "sync".
751 -E, --force-empty
753 Forces the insecure operation of syncing a disk with all the
754 original files missing. If SnapRAID detects that all the files
755 originally present in the disk are missing or rewritten, it
756 stops proceeding unless you specify this option. This allows to
757 easily detect when a data file-system is not mounted. This
758 option can be used only with "sync".
759 -U, --force-uuid
761 Forces the insecure operation of syncing, checking and fixing
762 with disks that have changed their UUID. If SnapRAID detects
763 that some disks have changed UUID, it stops proceeding unless
764 you specify this option. This allows to detect when your disks
765 are mounted in the wrong mount points. It´s anyway allowed to
766 have a single UUID change with single parity, and more with mul‐
767 tiple parity, because it´s the normal case of replacing disks
768 after a recovery. This option can be used only with "sync",
769 "check" or "fix".
770 -D, --force-device
772 Forces the insecure operation of fixing with inaccessible disks,
773 or with disks on the same physical device. Like if you lost two
774 data disks, and you have a spare disk to recover only the first
775 one, and you want to ignore the second inaccessible disk. Or if
776 you want to recover a disk in the free space left in an already
777 used disk, sharing the same physical device. This option can be
778 used only with "fix".
779 -N, --force-nocopy
781 In "sync", "check and "fix", disables the copy detection heuris‐
782 tic. Without this option SnapRAID assumes that files with same
783 attributes, like name, size and time-stamp are copies with the
784 same data. This allows to identify copied or moved files from
785 one disk to another, and to reuse the already computed hash
786 information to detect silent errors or to recover missing files.
787 This behavior, in some rare cases, may result in false posi‐
788 tives, or in a slow process due the many hash verifications, and
789 this option allows to resolve them. This option can be used
790 only with "sync", "check" and "fix".
791 -F, --force-full
793 In "sync" forces a full rebuild of the parity. This option can
794 be used when you add a new parity level, or if you reverted back
795 to an old content file using a more recent parity data. Instead
796 of recomputing the parity from scratch, this allows to reuse the
797 hashes present in the content file to validate data, and to
798 maintain data protection during the "sync" process using the
799 parity data you have. This option can be used only with "sync".
800 -R, --force-realloc
802 In "sync" forces a full reallocation of files and rebuild of the
803 parity. This option can be used to completely reallocate all
804 the files removing the fragmentation, but reusing the hashes
805 present in the content file to validate data. Compared to -F,
806 --force-full, this option reallocates all the parity not having
807 data protection during the operation. This option can be used
808 only with "sync".
809 -l, --log FILE
811 Write a detailed log in the specified file. If this option is
812 not specified, unexpected errors are printed on the screen,
813 likely resulting in too much output in case of many errors. When
814 -l, --log is specified, on the screen, go only fatal errors that
815 makes SnapRAID to stop progress. If the path starts with ´>>´
816 the file is opened in append mode. Occurrences of ´%D´ and ´%T´
817 in the name are replaced with the date and time in the format
818 YYYYMMDD and HHMMSS. Note that in Windows batch files, you´ll
819 have to double the ´%´ char, like result-%%D.log. And to use
820 ´>>´ you´ll have to enclose the name in ", like ">>result.log".
821 To output the log to standard output or standard error, you can
822 use respectively ">&1" and ">&2".
823 -L, --error-limit
825 Sets a new error limit before stopping execution. By default
826 SnapRAID stops if it encounters more than 100 Input/Output
827 errors, meaning that likely a disk is going to die. This
828 options affects "sync" and "scrub", that are allowed to continue
829 after the first bunch of disk errors, to try to complete at most
830 their operations. Instead, "check" and "fix" always stop at the
831 first error.
832 -S, --start BLKSTART
834 Starts the processing from the specified block number. It could
835 be useful to retry to check or fix some specific block, in case
836 of a damaged disk. It´s present mainly for advanced manual
837 recovering.
838 -B, --count BLKCOUNT
840 Processes only the specified number of blocks. It´s present
841 mainly for advanced manual recovering.
842 -C, --gen-conf CONTENT_FILE
844 Generates a dummy configuration file from an existing content
845 file. The configuration file is written in the standard output,
846 and it doesn´t overwrite an existing one. This configuration
847 file also contains the information needed to reconstruct the
848 disk mount points, in case you lose the entire system.
849 -v, --verbose
851 Prints more information on the screen. If specified one time,
852 it prints excluded files and more stats. This option has no
853 effect on the log files.
854 -q, --quiet
856 Prints less information on the screen. If specified one time,
857 removes the progress bar, if two times, the running operations,
858 three times, the info messages, four times the status messages.
859 Fatal errors are always printed on the screen. This option has
860 no effect on the log files.
861 -H, --help
863 Prints a short help screen.
864 -V, --version
866 Prints the program version.
867
869 SnapRAID requires a configuration file to know where your disk array is
870 located, and where storing the parity information.
871 In Unix it´s used the file "/usr/local/etc/snapraid.conf" if it exists,
873 or "/etc/snapraid.conf" otherwise. In Windows it´s used the file
874 "snapraid.conf" in the same directory of "snapraid.exe".
875 It should contain the following options (case sensitive):
877 parity FILE [,FILE] ...
879 Defines the files to use to store the parity information. The parity
880 enables the protection from a single disk failure, like RAID5.
881 You can specify multiples files that should be in different disks.
883 When a file cannot grow anymore, the next one is used. The total space
884 available must be as big as the biggest data disk in the array.
885 You can add additional parity files at later time, but you cannot
887 reorder or remove them.
888 Leaving the parity disks reserved for parity ensures that it doesn´t
890 get fragmented, improving the performance.
891 In Windows 256 MB are left unused in each disk to avoid the warning
893 about full disks.
894 This option is mandatory and it can be used only one time.
896 (2,3,4,5,6)-parity FILE [,FILE] ...
898 Defines the files to use to store extra parity information.
899 For each parity specified, one additional level of protection is
901 enabled:
902 · 2-parity enables RAID6 dual parity.
903 · 3-parity enables triple parity
904 · 4-parity enables quad (four) parity
905 · 5-parity enables penta (five) parity
906 · 6-parity enables hexa (six) parity
907 Each parity level requires the precence of all the previous parity lev‐
909 els.
910 The same considerations of the ´parity´ option apply.
912 These options are optional and they can be used only one time.
914 z-parity FILE [,FILE] ...
916 Defines an alternate file and format to store the triple parity.
917 This option is an alternative at ´3-parity´ mainly intended for low-end
919 CPUs like ARM or AMD Phenom, Athlon and Opteron that don´t support the
920 SSSE3 instructions set. In such cases it provides a better performance.
921 This format is similar, but faster, at the one used by the ZFS RAIDZ3.
923 Like ZFS, it doesn´t work beyond triple parity.
924 When using ´3-parity´ you will be warned if it´s recommended to use the
926 ´z-parity´ format for a performance improvement.
927 It´s possible to convert from one format to another, adjusting the con‐
929 figuration file with the wanted z-parity or 3-parity file, and using
930 ´fix´ to recreate it.
931 content FILE
933 Defines the file to use to store the list and check-sums of all the
934 files present in your disk array.
935 It can be placed in the disk used to store data, parity, or any other
937 disk available. If you use a data disk, this file is automatically
938 excluded from the "sync" process.
939 This option is mandatory and it can be used more times to save more
941 copies of the same files.
942 You have to store at least one copy for each parity disk used plus one.
944 Using some more doesn´t hurt.
945 data NAME DIR
947 Defines the name and the mount point of the data disks of the array.
948 NAME is used to identify the disk, and it must be unique. DIR is the
949 mount point of the disk in the file-system.
950 You can change the mount point as you like, as long you keep the NAME
952 fixed.
953 You should use one option for each data disk of the array.
955 You can rename later a disk, changing the NAME directly in the configu‐
957 ration file, and then run a ´sync´ command. In the rename case, the
958 association is done using the stored UUID of the disks.
959 nohidden
961 Excludes all the hidden files and directory. In Unix hidden files are
962 the ones starting with ".". In Windows they are the ones with the hid‐
963 den attribute.
964 exclude/include PATTERN
966 Defines the file or directory patterns to exclude and include in the
967 sync process. All the patterns are processed in the specified order.
968 If the first pattern that matches is an "exclude" one, the file is
970 excluded. If it´s an "include" one, the file is included. If no pat‐
971 tern matches, the file is excluded if the last pattern specified is an
972 "include", or included if the last pattern specified is an "exclude".
973 See the PATTERN section for more details in the pattern specifications.
975 This option can be used many times.
977 blocksize SIZE_IN_KIBIBYTES
979 Defines the basic block size in kibi bytes of the parity. One kibi
980 bytes is 1024 bytes.
981 The default blocksize is 256 and it should work for most cases.
983 WARNING! This option is for experts only, and it´s highly recommended
985 to not change it. To change again this value in future you´ll have to
986 recreate the whole parity!
987 A reason to use a different hashsize is if you have a lot of small
989 files. In the order of many millions.
990 For each file, even of few bytes, a whole block of parity is allocated,
992 and with many files this may result in a lot of unused parity space.
993 And when you completely fill the parity disk, you are not allowed to
994 add more files in the data disks. Anyway, the wasted parity doesn´t
995 sum between data disks. Wasted space resulting from a high number of
996 files in a data disk, limits only the amount of data in such data disk
997 and not in others.
998 As approximation, you can assume that half of the block size is wasted
1000 for each file. For example, with 100000 files and a 256 KiB block size,
1001 you are going to waste 13 GB of parity, that may result in 13 GB less
1002 space available in the data disk.
1003 You can get the amount of wasted space in each disk using "status".
1005 This is the amount of space that you must leave free in the data disks,
1006 or use for files not included in the array. If this value is negative,
1007 it means that your are near to fill the parity, and it represents the
1008 space you can still waste.
1009 To avoid the problem, you can use a bigger partition for parity. For
1011 example, if you have the parity partition bigger than 13 GB than data
1012 disks, you have enough extra space to handle up to 100000 files in each
1013 data disk.
1014 A trick to get a bigger parity partition in Linux, is to format it with
1016 the command:
1017 mkfs.ext4 -m 0 -T largefile4 DEVICE
1019 This results in about 1.5% of extra space. Meaning about 60 GB for a 4
1020 TB disk, that allows about 460000 files in each data disk without any
1021 wasted space.
1022 hashsize SIZE_IN_BYTES
1024 Defines the hash size in bytes of the saved blocks.
1025 The default hashsize is 16 bytes (128 bits), and it should work for
1027 most cases.
1028 WARNING! This option is for experts only, and it´s highly recommended
1030 to not change it. To change again this value in future you´ll have to
1031 recreate the whole parity!
1032 A reason to use a different hashsize is if your system has small mem‐
1034 ory. As a rule of thumb SnapRAID usually requires 1 GiB of RAM memory
1035 for each 16 TB of data in the array.
1036 Specifically, to store the hashes of the data, SnapRAID requires about
1038 TS*(1+HS)/BS bytes of RAM memory. Where TS is the total size in bytes
1039 of your disk array, BS is the block size in bytes, and HS is the hash
1040 size in bytes.
1041 For example with 8 disks of 4 TB and a block size of 256 KiB (1 KiB =
1043 1024 bytes), and an hash size of 16, you get:
1044 RAM = (8 * 4 * 10^12) * (1+16) / (256 * 2^10) = 1.93 GiB
1046 Switching to a hash size of 8, you get:
1047 RAM = (8 * 4 * 10^12) * (1+8) / (256 * 2^10) = 1.02 GiB
1049 Switching to a block size of 512, you get:
1050 RAM = (8 * 4 * 10^12) * (1+16) / (512 * 2^10) = 0.96 GiB
1052 Switching to both a hash size of 8, and a block size of 512 you get:
1053 RAM = (8 * 4 * 10^12) * (1+8) / (512 * 2^10) = 0.51 GiB
1055 autosave SIZE_IN_GIGABYTES
1056 Automatically save the state when syncing or scrubbing after the speci‐
1057 fied amount of GB processed. This option is useful to avoid to restart
1058 from scratch long "sync" commands interrupted by a machine crash, or
1059 any other event that may interrupt SnapRAID.
1060 pool DIR
1062 Defines the pooling directory where the virtual view of the disk array
1063 is created using the "pool" command.
1064 The directory must already exist.
1066 share UNC_DIR
1068 Defines the Windows UNC path required to access the disks remotely.
1069 If this option is specified, the symbolic links created in the pool
1071 directory use this UNC path to access the disks. Without this option
1072 the symbolic links generated use only local paths, not allowing to
1073 share the pool directory in the network.
1074 The symbolic links are formed using the specified UNC path, adding the
1076 disk name as specified in the "disk" option, and finally adding the
1077 file dir and name.
1078 This option is only required for Windows.
1080 smartctl DISK/PARITY OPTIONS...
1082 Defines a custom smartctl command to obtain the SMART attributes for
1083 each disk. This may be required for RAID controllers and for some USB
1084 disk that cannot be auto-detected.
1085 DISK is the same disk name specified in the "disk" option. PARITY is
1087 one of the parity name as "parity,(1,2,3,4,5,6,z)-parity".
1088 In the specified OPTIONS, the "%s" string is replaced by the device
1090 name. Note that in case of RAID controllers the device is likely fixed,
1091 and you don´t have to use "%s".
1092 Refers at the smartmontools documentation about the possible options:
1094 https://www.smartmontools.org/wiki/Supported_RAID-Controllers
1096 https://www.smartmontools.org/wiki/Supported_USB-Devices
1097 Examples
1098 An example of a typical configuration for Unix is:
1099 parity /mnt/diskp/snapraid.parity
1101 content /mnt/diskp/snapraid.content
1102 content /var/snapraid/snapraid.content
1103 data d1 /mnt/disk1/
1104 data d2 /mnt/disk2/
1105 data d3 /mnt/disk3/
1106 exclude /lost+found/
1107 exclude /tmp/
1108 smartctl d1 -d sat %s
1109 smartctl d2 -d usbjmicron %s
1110 smartctl parity -d areca,1/1 /dev/sg0
1111 smartctl 2-parity -d areca,2/1 /dev/sg0
1112 An example of a typical configuration for Windows is:
1113 parity E:\snapraid.parity
1115 content E:\snapraid.content
1116 content C:\snapraid\snapraid.content
1117 data d1 G:\array\
1118 data d2 H:\array\
1119 data d3 I:\array\
1120 exclude Thumbs.db
1121 exclude \$RECYCLE.BIN
1122 exclude \System Volume Information
1123 smartctl d1 -d sat %s
1124 smartctl d2 -d usbjmicron %s
1125 smartctl parity -d areca,1/1 /dev/arcmsr0
1126 smartctl 2-parity -d areca,2/1 /dev/arcmsr0
1128 Patterns are used to select a subset of files to exclude or include in
1129 the process.
1130 There are four different types of patterns:
1132 FILE Selects any file named as FILE. You can use any globbing charac‐
1134 ter like * and ?, and char classes like [a-z]. This pattern is
1135 applied only to files and not to directories.
1136 DIR/ Selects any directory named DIR and everything inside. You can
1138 use any globbing character like * and ?. This pattern is
1139 applied only to directories and not to files.
1140 /PATH/FILE
1142 Selects the exact specified file path. You can use any globbing
1143 character like * and ? but they never match a directory slash.
1144 This pattern is applied only to files and not to directories.
1145 /PATH/DIR/
1147 Selects the exact specified directory path and everything
1148 inside. You can use any globbing character like * and ? but they
1149 never match a directory slash. This pattern is applied only to
1150 directories and not to files.
1151 Note that when you specify an absolute path starting with /, it´s
1153 applied at the array root dir and not at the local file-system root
1154 dir.
1155 In Windows you can use the backslash \ instead of the forward slash /.
1157 Note that Windows system directories, junctions, mount points, and any
1158 other Windows special directory are treated just as files, meaning that
1159 to exclude them you must use a file rule, and not a directory one.
1160 If the file name you want to use really contains a ´*´, ´?´, ´[´, or
1162 ´]´ char, you have to escape it to avoid to have interpreted as a glob‐
1163 bing character. In Unix the escape char is ´\´, in Windows it´s ´^´.
1164 Note that when the pattern is on the command line, you have to double
1165 the escape character to avoid to have it interpreted by the command
1166 shell.
1167 In the configuration file, you can use different strategies to filter
1169 the files to process. The simplest one is to use only "exclude" rules
1170 to remove all the files and directories you do not want to process. For
1171 example:
1172 # Excludes any file named "*.unrecoverable"
1174 exclude *.unrecoverable
1175 # Excludes the root directory "/lost+found"
1176 exclude /lost+found/
1177 # Excludes any sub-directory named "tmp"
1178 exclude tmp/
1179 The opposite way is to define only the file you want to process, using
1180 only "include" rules. For example:
1181 # Includes only some directories
1183 include /movies/
1184 include /musics/
1185 include /pictures/
1186 The final way, is to mix "exclude" and "include" rules. In this case
1187 take care that the order of rules is important. Previous rules have the
1188 precedence over the later ones. To get things simpler you can first
1189 have all the "exclude" rules and then all the "include" ones. For exam‐
1190 ple:
1191 # Excludes any file named "*.unrecoverable"
1193 exclude *.unrecoverable
1194 # Excludes any sub-directory named "tmp"
1195 exclude tmp/
1196 # Includes only some directories
1197 include /movies/
1198 include /musics/
1199 include /pictures/
1200 On the command line, using the -f option, you can only use "include"
1201 patterns. For example:
1202 # Checks only the .mp3 files.
1204 # Note the "" use to avoid globbing expansion by the shell in Unix.
1205 snapraid -f "*.mp3" check
1206 In Unix, when using globbing chars in the command line, you have to
1207 quote them. Otherwise the shell will try to expand them.
1208
1210 SnapRAID stores the list and check-sums of your files in the content
1211 file.
1212 It´s a binary file, listing all the files present in your disk array,
1214 with all the check-sums to verify their integrity.
1215 This file is read and written by the "sync" and "scrub" commands, and
1217 read by "fix", "check" and "status".
1218
1220 SnapRAID stores the parity information of your array in the parity
1221 files.
1222 They are binary files, containing the computed parity of all the blocks
1224 defined in the "content" file.
1225 These files are read and written by the "sync" and "fix" commands, and
1227 only read by "scrub" and "check".
1228
1230 SnapRAID in Unix ignores any encoding. It reads and stores the file
1231 names with the same encoding used by the file-system.
1232 In Windows all the names read from the file-system are converted and
1234 processed in the UTF-8 format.
1235 To have the file names printed correctly you have to set the Windows
1237 console in the UTF-8 mode, with the command "chcp 65001", and use a
1238 TrueType font like "Lucida Console" as console font. Note that it has
1239 effect only on the printed file names, if you redirect the console out‐
1240 put to a file, the resulting file is always in the UTF-8 format.
1241
1243 This file is Copyright (C) 2011 Andrea Mazzoleni
1244
1246 rsync(1)
1247 SnapRAID Backup For Disk Arrays(1)