1SnapRAID Backup For Disk ArraGyesn(e1r)al CommandsSMnaanpuRaAlID Backup For Disk Arrays(1)
2
6 snapraid - SnapRAID Backup For Disk Arrays
7
9 snapraid [-c, --conf CONFIG]
10 [-f, --filter PATTERN] [-d, --filter-disk NAME]
11 [-m, --filter-missing] [-e, --filter-error]
12 [-a, --audit-only] [-h, --pre-hash] [-i, --import DIR]
13 [-p, --plan PERC|bad|new|full]
14 [-o, --older-than DAYS] [-l, --log FILE]
15 [-Z, --force-zero] [-E, --force-empty]
16 [-U, --force-uuid] [-D, --force-device]
17 [-N, --force-nocopy] [-F, --force-full]
18 [-R, --force-realloc]
19 [-S, --start BLKSTART] [-B, --count BLKCOUNT]
20 [-L, --error-limit NUMBER]
21 [-v, --verbose] [-q, --quiet]
22 status|smart|up|down|diff|sync|scrub|fix|check|list|dup
23 |pool|devices|touch|rehash
24 snapraid [-V, --version] [-H, --help] [-C, --gen-conf CONTENT]
26 SnapRAID is a backup program for disk arrays. It stores parity informa‐
27 tion of your data and it recovers from up to six disk failures.
28 SnapRAID is mainly targeted for a home media center, with a lot of big
30 files that rarely change.
31 Beside the ability to recover from disk failures, other features of
33 SnapRAID are:
34 • You can use disk already filled with files, without the need to
35 reformat them. You will access them like now.
36 • All your data is hashed to ensure data integrity and to avoid
37 silent corruption.
38 • If the failed disks are too many to allow a recovery, you lose
39 the data only on the failed disks. All the data in the other
40 disks is safe.
41 • If you accidentally delete some files in a disk, you can recover
42 them.
43 • The disks can have different sizes.
44 • You can add disks at any time.
45 • It doesn´t lock-in your data. You can stop using SnapRAID at any
46 time without the need to reformat or move data.
47 • To access a file, only a single disk needs to spin, saving power
48 and producing less noise.
49 The official site of SnapRAID is:
51 http://www.snapraid.it/
54 SnapRAID is in between a RAID and a Backup program trying to get the
55 best benefits of them. Although it also has some limitations that you
56 should consider before using it.
57 The main one is that if a disk fails, and you haven´t recently synced,
59 you may be unable to do a complete recover. More specifically, you may
60 be unable to recover up to the size of the amount of the changed or
61 deleted files from the last sync operation. This happens even if the
62 files changed or deleted are not in the failed disk. This is the reason
63 because SnapRAID is better suited for data that rarely change.
64 Instead the new added files don´t prevent the recovering of the already
66 existing files. You may only lose the just added files, if they are on
67 the failed disk.
68 Other limitations are:
70 • You have different file-systems for each disk. Using a RAID you
71 have only a big file-system.
72 • It doesn´t stripe data. With RAID you get a speed boost with
73 striping.
74 • It doesn´t support real-time recovery. With RAID you do not
75 have to stop working when a disk fails.
76 • It´s able to recover damages only from a limited number of
77 disks. With a Backup you are able to recover from a complete
78 failure of the whole disk array.
79 • Only file, time-stamps, symlinks and hardlinks are saved. Per‐
80 missions, ownership and extended attributes are not saved.
81
83 To use SnapRAID you need to first select one disk of your disk array to
84 dedicate at the "parity" information. With one disk for parity you will
85 be able to recover from a single disk failure, like RAID5.
86 If you want to be able to recover from more disk failures, like RAID6,
88 you must reserve additional disks for parity. Any additional parity
89 disk allow to recover from one more disk failure.
90 As parity disks, you have to pick the biggest disks in the array, as
92 the parity information may grow in size as the biggest data disk in the
93 array.
94 These disks will be dedicated to store the "parity" files. You should
96 not store your data in them.
97 Then you have to define the "data" disks that you want to protect with
99 SnapRAID. The protection is more effective if these disks contain data
100 that rarely change. For this reason it´s better to DO NOT include the
101 Windows C:\ disk, or the Unix /home, /var and /tmp disks.
102 The list of files is saved in the "content" files, usually stored in
104 the data, parity or boot disks. These files contain the details of
105 your backup, with all the check-sums to verify its integrity. The
106 "content" file is stored in multiple copies, and each one must be in a
107 different disk, to ensure that in even in case of multiple disk fail‐
108 ures at least one copy is available.
109 For example, suppose that you are interested only at one parity level
111 of protection, and that your disks are present in:
112 /mnt/diskp <- selected disk for parity
114 /mnt/disk1 <- first disk to protect
115 /mnt/disk2 <- second disk to protect
116 /mnt/disk3 <- third disk to protect
117 you have to create the configuration file /etc/snapraid.conf with the
118 following options:
119 parity /mnt/diskp/snapraid.parity
121 content /var/snapraid/snapraid.content
122 content /mnt/disk1/snapraid.content
123 content /mnt/disk2/snapraid.content
124 data d1 /mnt/disk1/
125 data d2 /mnt/disk2/
126 data d3 /mnt/disk3/
127 If you are in Windows, you should use the Windows path format, with
128 drive letters and backslashes instead of slashes.
129 parity E:\snapraid.parity
131 content C:\snapraid\snapraid.content
132 content F:\array\snapraid.content
133 content G:\array\snapraid.content
134 data d1 F:\array\
135 data d2 G:\array\
136 data d3 H:\array\
137 If you have many disks, and you run out of drive letters, you can mount
138 disks directly in sub folders. See:
139 https://www.google.com/search?q=Windows+mount+point
141 At this point you are ready to start the "sync" command to build the
142 parity information.
143 snapraid sync
145 This process may take some hours the first time, depending on the size
146 of the data already present in the disks. If the disks are empty the
147 process is immediate.
148 You can stop it at any time pressing Ctrl+C, and at the next run it
150 will start where interrupted.
151 When this command completes, your data is SAFE.
153 Now you can start using your array as you like, and periodically update
155 the parity information running the "sync" command.
156 Scrubbing
158 To periodically check the data and parity for errors, you can run the
159 "scrub" command.
160 snapraid scrub
162 This command verifies the data in your array comparing it with the hash
163 computed in the "sync" command.
164 Every run of the command checks about the 8% of the array, but not data
166 already scrubbed in the previous 10 days. You can use the -p, --plan
167 option to specify a different amount, and the -o, --older-than option
168 to specify a different age in days. For example, to check 5% of the
169 array older than 20 days use:
170 snapraid -p 5 -o 20 scrub
172 If during the process, silent or input/output errors are found, the
173 corresponding blocks are marked as bad in the "content" file, and
174 listed in the "status" command.
175 snapraid status
177 To fix them, you can use the "fix" command filtering for bad blocks
178 with the -e, --filter-error options:
179 snapraid -e fix
181 At the next "scrub" the errors will disappear from the "status" report
182 if really fixed. To make it fast, you can use -p bad to scrub only
183 blocks marked as bad.
184 snapraid -p bad scrub
186 Take care that running "scrub" on a not synced array may result in er‐
187 rors caused by removed or modified files. These errors are reported in
188 the "scrub" result, but related blocks are not marked as bad.
189 Pooling
191 To have all the files in your array shown in the same directory tree,
192 you can enable the "pooling" feature. It consists in creating a
193 read-only virtual view of all the files in your array using symbolic
194 links.
195 You can configure the "pooling" directory in the configuration file
197 with:
198 pool /pool
200 or, if you are in Windows, with:
201 pool C:\pool
203 and then run the "pool" command to create or update the virtual view.
204 snapraid pool
206 If you are using a Unix platform and you want to share such directory
207 in the network to either Windows or Unix machines, you should add to
208 your /etc/samba/smb.conf the following options:
209 # In the global section of smb.conf
211 unix extensions = no
212 # In the share section of smb.conf
213 [pool]
214 comment = Pool
215 path = /pool
216 read only = yes
217 guest ok = yes
218 wide links = yes
219 follow symlinks = yes
220 In Windows the same sharing operation is not so straightforward, be‐
221 cause Windows shares the symbolic links as they are, and that requires
222 the network clients to resolve them remotely.
223 To make it working, besides sharing in the network the pool directory,
225 you must also share all the disks independently, using as share points
226 the disk names as defined in the configuration file. You must also
227 specify in the "share" option of the configure file, the Windows UNC
228 path that remote clients needs to use to access such shared disks.
229 For example, operating from a server named "darkstar", you can use the
231 options:
232 data d1 F:\array\
234 data d2 G:\array\
235 data d3 H:\array\
236 pool C:\pool
237 share \\darkstar
238 and share the following dirs in the network:
239 \\darkstar\pool -> C:\pool
241 \\darkstar\d1 -> F:\array
242 \\darkstar\d2 -> G:\array
243 \\darkstar\d3 -> H:\array
244 to allow remote clients to access all the files at \\darkstar\\pool.
245 You may also need to configure remote clients enabling access at remote
247 symlinks with the command:
248 fsutil behavior set SymlinkEvaluation L2L:1 R2R:1 L2R:1 R2L:1
250 Undeleting
251 SnapRAID is more like a backup program than a RAID system, and it can
252 be used to restore or undelete files to their previous state using the
253 -f, --filter option :
254 snapraid fix -f FILE
256 or for a directory:
257 snapraid fix -f DIR/
259 You can also use it to recover only accidentally deleted files inside a
260 directory using the -m, --filter-missing option, that restores only
261 missing files, leaving untouched all the others.
262 snapraid fix -m -f DIR/
264 Or to recover all the deleted files in all the drives with:
265 snapraid fix -m
267 Recovering
268 The worst happened, and you lost one or more disks!
269 DO NOT PANIC! You will be able to recover them!
271 The first thing you have to do is to avoid further changes at your disk
273 array. Disable any remote connection to it, any scheduled process, in‐
274 cluding any scheduled SnapRAID nightly sync or scrub.
275 Then proceed with the following steps.
277 STEP 1 -> Reconfigure
279 You need some space to recover, even better if you already have addi‐
280 tional spare disks, but in case, also an external USB or remote disk is
281 enough.
282 Change the SnapRAID configuration file to make the "data" or "parity"
284 option of the failed disk to point to the place where you have enough
285 empty space to recover the files.
286 For example, if you have that disk "d1" failed, you can change from:
288 data d1 /mnt/disk1/
290 to:
291 data d1 /mnt/new_spare_disk/
293 If the disk to recover is a parity disk, change the appropriate "par‐
294 ity" option. If you have more broken disks, change all their configu‐
295 ration options.
296 STEP 2 -> Fix
298 Run the fix command, storing the log in an external file with:
299 snapraid -d NAME -l fix.log fix
301 Where NAME is the name of the disk, like "d1" as in our previous exam‐
302 ple. In case the disk to recover is a parity disk, use the "parity",
303 "2-parity" names. If you have more broken disks, use multiple -d op‐
304 tions to specify all of them.
305 This command will take a long time.
307 Take care that you need also few gigabytes free to store the fix.log
309 file. Run it from a disk with some free space.
310 Now you have recovered all the recoverable. If some file is partially
312 or totally unrecoverable, it will be renamed adding the ".unrecover‐
313 able" extension.
314 You can get a detailed list of all the unrecoverable blocks in the
316 fix.log file checking all the lines starting with "unrecoverable:"
317 If you are not satisfied of the recovering, you can retry it as many
319 time you wish.
320 For example, if you have removed files from the array after the last
322 "sync", this may result in some other files not recovered. In this
323 case, you can retry the "fix" using the -i, --import option, specifying
324 where these files are now, to include them again in the recovering
325 process.
326 If you are satisfied of the recovering, you can now proceed further,
328 but take care that after syncing you cannot retry the "fix" command
329 anymore!
330 STEP 3 -> Check
332 As paranoid check, you can now run a "check" command to ensure that ev‐
333 erything is OK on the recovered disk.
334 snapraid -d NAME -a check
336 Where NAME is the name of the disk, like "d1" as in our previous exam‐
337 ple.
338 The options -d and -a tell SnapRAID to check only the specified disk,
340 and ignore all the parity data.
341 This command will take a long time, but if you are not paranoid, you
343 can skip it.
344 STEP 4 -> Sync
346 Run the "sync" command to re-synchronize the array with the new disk.
347 snapraid sync
349 If everything is recovered, this command is immediate.
350
352 SnapRAID provides a few simple commands that allow to:
353 • Prints the status of the array -> "status"
354 • Controls the disks -> "smart", "up", "down"
355 • Makes a backup/snapshot -> "sync"
356 • Periodically checks data -> "scrub"
357 • Restore the last backup/snapshot -> "fix".
358 Take care that the commands have to be written in lower case.
360 status
362 Prints a summary of the state of the disk array.
363 It includes information about the parity fragmentation, how old are the
365 blocks without checking, and all the recorded silent errors encountered
366 while scrubbing.
367 Note that the information presented refers at the latest time you run
369 "sync". Later modifications are not taken into account.
370 If bad blocks were detected, their block numbers are listed. To fix
372 them, you can use the "fix -e" command.
373 It also shows a graph representing the last time each block was
375 scrubbed or synced. Scrubbed blocks are shown with ´*´, blocks synced
376 but not yet scrubbed with ´o´.
377 Nothing is modified.
379 smart
381 Prints a SMART report of all the disks of the array.
382 It includes an estimation of the probability of failure in the next
384 year allowing to plan maintenance replacements of the disks that show
385 suspicious attributes.
386 This probability estimation obtained correlating the SMART attributes
388 of the disks, with the Backblaze data available at:
389 https://www.backblaze.com/hard-drive-test-data.html
391 If SMART reports that a disk is failing, "FAIL" or "PREFAIL" is printed
392 for that disk, and SnapRAID returns with an error. In this case an im‐
393 mediate replacement of the disk is highly recommended.
394 Other possible strings are:
396 logfail In the past some attributes were lower than the threshold.
397 logerr The device error log contains errors.
398 selferr The device self-test log contains errors.
399 If the -v, --verbose option is specified a deeper statistical analysis
401 is provided. This analysis can help you to decide if you need more or
402 less parity.
403 This command uses the "smartctl" tool, and it´s equivalent to run
405 "smartctl -a" on all the devices.
406 If your devices are not auto-detected correctly, you can configure a
408 custom command using the "smartctl" option in the configuration file.
409 Nothing is modified.
411 up
413 Spins up all the disks of the array.
414 You can spin-up only some specific disks using the -d, --filter-disk
416 option.
417 Take care that spinning-up all the disks at the same time needs a lot
419 of power. Ensure that your power-supply can sustain that.
420 Nothing is modified.
422 down
424 Spins down all the disks of the array.
425 This command uses the "smartctl" tool, and it´s equivalent to run
427 "smartctl -s standby,now" on all the devices.
428 You can spin-down only some specific disks using the -d, --filter-disk
430 option.
431 Nothing is modified.
433 diff
435 Lists all the files modified from the last "sync" that need to have
436 their parity data recomputed.
437 This command doesn´t check the file data, but only the file time-stamp
439 size and inode.
440 At the end of the command, you´ll get a summary of the file changes
442 grouped by:
443 equal Files equal at before.
444 added Files added that were not present before.
445 removed Files removed.
446 updated Files with a different size or time-stamp, meaning that
447 they were modified.
448 moved Files moved to a different directory of the same disk. They
449 are identified by having the same name, size, time-stamp and
450 inode, but different directory.
451 copied Files copied in the same or different disk. Note that if in
452 true they are moved to a different disk, you´ll also have them
453 counted in "removed". They are identified by having the same
454 name, size, and time-stamp. But if the sub-second time-stamp is
455 zero, then the full path should match, and not only the name.
456 restored Files with a different inode but with name, size and
457 time-stamp matching. These are usually files restored after be‐
458 ing deleted.
459 If a "sync" is required, the process return code is 2, instead of the
461 default 0. The return code 1 is instead for a generic error condition.
462 Nothing is modified.
464 sync
466 Updates the parity information. All the modified files in the disk ar‐
467 ray are read, and the corresponding parity data is updated.
468 You can stop this process at any time pressing Ctrl+C, without losing
470 the work already done. At the next run the "sync" process will start
471 where interrupted.
472 If during the process, silent or input/output errors are found, the
474 corresponding blocks are marked as bad.
475 Files are identified by path and/or inode and checked by size and
477 time-stamp. If the file size or time-stamp are different, the parity
478 data is recomputed for the whole file. If the file is moved or renamed
479 in the same disk, keeping the same inode, the parity is not recomputed.
480 If the file is moved to another disk, the parity is recomputed, but the
481 previously computed hash information is kept.
482 The "content" and "parity" files are modified if necessary. The files
484 in the array are NOT modified.
485 scrub
487 Scrubs the array, checking for silent or input/output errors in data
488 and parity disks.
489 For each command invocation, about the 8% of the array is checked, but
491 nothing that was already scrubbed in the last 10 days. This means that
492 scrubbing once a week, every bit of data is checked at least one time
493 every three months.
494 You can define a different scrub plan or amount using the -p, --plan
496 option that takes as argument: bad - Scrub blocks marked bad. new -
497 Scrub just synced blocks not yet scrubbed. full - Scrub everything.
498 0-100 - Scrub the exact percentage of blocks.
499 If you specify a percentage amount, you can also use the -o,
501 --older-than option to define how old the block should be. The oldest
502 blocks are scrubbed first ensuring an optimal check. If instead you
503 want to scrub the just synced blocks, not yet scrubbed, you should use
504 the "-p new" option.
505 To get the details of the scrub status use the "status" command.
507 For any silent or input/output error found the corresponding blocks are
509 marked as bad in the "content" file. These bad blocks are listed in
510 "status", and can be fixed with "fix -e". After the fix, at the next
511 scrub they will be rechecked, and if found corrected, the bad mark will
512 be removed. To scrub only the bad blocks, you can use the "scrub -p
513 bad" command.
514 It´s recommended to run "scrub" only on a synced array, to avoid to
516 have reported error caused by unsynced data. These errors are recog‐
517 nized as not being silent errors, and the blocks are not marked as bad,
518 but such errors are reported in the output of the command.
519 Files are identified only by path, and not by inode.
521 The "content" file is modified to update the time of the last check of
523 each block, and to mark bad blocks. The "parity" files are NOT modi‐
524 fied. The files in the array are NOT modified.
525 fix
527 Fix all the files and the parity data.
528 All the files and the parity data are compared with the snapshot state
530 saved in the last "sync". If a difference is found, it´s reverted to
531 the stored snapshot.
532 The "fix" command doesn´t differentiate between errors and intentional
534 modifications. It unconditionally reverts the file state at the last
535 "sync".
536 If no other option is specified the full array is processed. Use the
538 filter options to select a subset of files or disks to operate on.
539 To only fix the blocks marked bad during "sync" and "scrub", use the
541 -e, --filter-error option. As difference from other filter options,
542 with this one the fixes are applied only to files that are not modified
543 from the latest "sync".
544 All the files that cannot be fixed are renamed adding the ".unrecover‐
546 able" extension.
547 Before fixing, the full array is scanned to find any moved file, after
549 the last "sync" operation. These files are identified by their
550 time-stamp, ignoring their name and directory, and are used in the re‐
551 covering process if necessary. If you moved some of them outside the
552 array, you can use the -i, --import option to specify additional direc‐
553 tories to scan.
554 Files are identified only by path, and not by inode.
556 The "content" file is NOT modified. The "parity" files are modified if
558 necessary. The files in the array are modified if necessary.
559 check
561 Verify all the files and the parity data.
562 It works like "fix", but it only simulates a recovery and no change is
564 written in the array.
565 This command is mostly intended for manual verification, like after a
567 recovery process or in other special conditions. For periodic and
568 scheduled checks uses "scrub".
569 If you use the -a, --audit-only option, only the file data is checked,
571 and the parity data is ignored for a faster run.
572 Files are identified only by path, and not by inode.
574 Nothing is modified.
576 list
578 Lists all the files contained in the array at the time of the last
579 "sync".
580 Nothing is modified.
582 dup
584 Lists all the duplicate files. Two files are assumed equal if their
585 hashes are matching. The file data is not read, but only the pre-com‐
586 puted hashes are used.
587 Nothing is modified.
589 pool
591 Creates or updates in the "pooling" directory a virtual view of all the
592 files of your disk array.
593 The files are not really copied here, but just linked using symbolic
595 links.
596 When updating, all the present symbolic links and empty sub-directories
598 are deleted and replaced with the new view of the array. Any other reg‐
599 ular file is left in place.
600 Nothing is modified outside the pool directory.
602 devices
604 Prints the low level devices used by the array.
605 This command prints the devices associations in place in the array, and
607 it´s mainly intended as a script interface.
608 The first two columns are the low level device id and path. The next
610 two columns are the high level device id and path. The latest column
611 if the disk name in the array.
612 In most cases you have one low level device for each disk in the array,
614 but in some more complex configurations, you may have multiple low
615 level devices used by a single disk in the array.
616 Nothing is modified.
618 touch
620 Sets arbitrarily the sub-second time-stamp of all the files that have
621 it at zero.
622 This improves the SnapRAID capability to recognize moved and copied
624 files as it makes the time-stamp almost unique, removing possible du‐
625 plicates.
626 More specifically, if the sub-second time-stamp is not zero, a moved or
628 copied file is identified as such if it matches the name, size and
629 time-stamp. If instead the sub-second time-stamp is zero, it´s consid‐
630 ered a copy only if it matches the full path, size and time-stamp.
631 Note that the second precision time-stamp is not modified, and all the
633 dates and times of your files will be maintained.
634 rehash
636 Schedules a rehash of the whole array.
637 This command changes the hash kind used, typically when upgrading from
639 a 32 bits system to a 64 bits one, to switch from MurmurHash3 to the
640 faster SpookyHash.
641 If you are already using the optimal hash, this command does nothing
643 and tells you that nothing has to be done.
644 The rehash isn´t done immediately, but it takes place progressively
646 during "sync" and "scrub".
647 You can get the rehash state using "status".
649 During the rehash, SnapRAID maintains full functionality, with the only
651 exception of "dup" not able to detect duplicated files using a differ‐
652 ent hash.
653
655 SnapRAID provides the following options:
656 -c, --conf CONFIG
658 Selects the configuration file to use. If not specified in Unix
659 it´s used the file "/usr/local/etc/snapraid.conf" if it exists,
660 or "/etc/snapraid.conf" otherwise. In Windows it´s used the
661 file "snapraid.conf" in the same directory of "snapraid.exe".
662 -f, --filter PATTERN
664 Filters the files to process in "check" and "fix". Only the
665 files matching the entered pattern are processed. This option
666 can be used many times. See the PATTERN section for more de‐
667 tails in the pattern specifications. In Unix, ensure to quote
668 globbing chars if used. This option can be used only with
669 "check" and "fix". Note that it cannot be used with "sync" and
670 "scrub", because they always process the whole array.
671 -d, --filter-disk NAME
673 Filters the disks to process in "check", "fix", "up" and "down".
674 You must specify a disk name as named in the configuration file.
675 You can also specify parity disks with the names: "parity",
676 "2-parity", "3-parity", ... to limit the operations a specific
677 parity disk. If you combine more --filter, --filter-disk and
678 --filter-missing options, only files matching all the set of
679 filters are selected. This option can be used many times. This
680 option can be used only with "check", "fix", "up" and "down".
681 Note that it cannot be used with "sync" and "scrub", because
682 they always process the whole array.
683 -m, --filter-missing
685 Filters the files to process in "check" and "fix". Only the
686 files missing/deleted from the array are processed. When used
687 with "fix", this is a kind of "undelete" command. If you com‐
688 bine more --filter, --filter-disk and --filter-missing options,
689 only files matching all the set of filters are selected. This
690 option can be used only with "check" and "fix". Note that it
691 cannot be used with "sync" and "scrub", because they always
692 process the whole array.
693 -e, --filter-error
695 Process the files with errors in "check" and "fix". It pro‐
696 cesses only files that have blocks marked with silent or in‐
697 put/output errors during "sync" and "scrub", and listed in "sta‐
698 tus". This option can be used only with "check" and "fix".
699 -p, --plan PERC|bad|new|full
701 Selects the scrub plan. If PERC is a numeric value from 0 to
702 100, it´s interpreted as the percentage of blocks to scrub. In‐
703 stead of a percentage, you can also specify a plan: "bad" scrubs
704 bad blocks, "new" the blocks not yet scrubbed, and "full" for
705 everything. This option can be used only with "scrub".
706 -o, --older-than DAYS
708 Selects the older the part of the array to process in "scrub".
709 DAYS is the minimum age in days for a block to be scrubbed, de‐
710 fault is 10. Blocks marked as bad are always scrubbed despite
711 this option. This option can be used only with "scrub".
712 -a, --audit-only
714 In "check" verifies the hash of the files without doing any kind
715 of check on the parity data. If you are interested in checking
716 only the file data this option can speedup a lot the checking
717 process. This option can be used only with "check".
718 -h, --pre-hash
720 In "sync" runs a preliminary hashing phase of all the new data
721 to have an additional verification before the parity computa‐
722 tion. Usually in "sync" no preliminary hashing is done, and the
723 new data is hashed just before the parity computation when it´s
724 read for the first time. Unfortunately, this process happens
725 when the system is under heavy load, with all disks spinning and
726 with a busy CPU. This is an extreme condition for the machine,
727 and if it has a latent hardware problem, it´s possible to en‐
728 counter silent errors what cannot be detected because the data
729 is not yet hashed. To avoid this risk, you can enable the
730 "pre-hash" mode and have all the data read two times to ensure
731 its integrity. This option also verifies the files moved inside
732 the array, to ensure that the move operation went successfully,
733 and in case to block the sync and to allow to run a fix opera‐
734 tion. This option can be used only with "sync".
735 -i, --import DIR
737 Imports from the specified directory any file that you deleted
738 from the array after the last "sync". If you still have such
739 files, they could be used by "check" and "fix" to improve the
740 recover process. The files are read also in sub-directories and
741 they are identified regardless of their name. This option can
742 be used only with "check" and "fix".
743 -Z, --force-zero
745 Forces the insecure operation of syncing a file with zero size
746 that before was not. If SnapRAID detects a such condition, it
747 stops proceeding unless you specify this option. This allows to
748 easily detect when after a system crash, some accessed files
749 were truncated. This is a possible condition in Linux with the
750 ext3/ext4 file-systems. This option can be used only with
751 "sync".
752 -E, --force-empty
754 Forces the insecure operation of syncing a disk with all the
755 original files missing. If SnapRAID detects that all the files
756 originally present in the disk are missing or rewritten, it
757 stops proceeding unless you specify this option. This allows to
758 easily detect when a data file-system is not mounted. This op‐
759 tion can be used only with "sync".
760 -U, --force-uuid
762 Forces the insecure operation of syncing, checking and fixing
763 with disks that have changed their UUID. If SnapRAID detects
764 that some disks have changed UUID, it stops proceeding unless
765 you specify this option. This allows to detect when your disks
766 are mounted in the wrong mount points. It´s anyway allowed to
767 have a single UUID change with single parity, and more with mul‐
768 tiple parity, because it´s the normal case of replacing disks
769 after a recovery. This option can be used only with "sync",
770 "check" or "fix".
771 -D, --force-device
773 Forces the insecure operation of fixing with inaccessible disks,
774 or with disks on the same physical device. Like if you lost two
775 data disks, and you have a spare disk to recover only the first
776 one, and you want to ignore the second inaccessible disk. Or if
777 you want to recover a disk in the free space left in an already
778 used disk, sharing the same physical device. This option can be
779 used only with "fix".
780 -N, --force-nocopy
782 In "sync", "check and "fix", disables the copy detection heuris‐
783 tic. Without this option SnapRAID assumes that files with same
784 attributes, like name, size and time-stamp are copies with the
785 same data. This allows to identify copied or moved files from
786 one disk to another, and to reuse the already computed hash in‐
787 formation to detect silent errors or to recover missing files.
788 This behavior, in some rare cases, may result in false posi‐
789 tives, or in a slow process due the many hash verification, and
790 this option allows to resolve them. This option can be used
791 only with "sync", "check" and "fix".
792 -F, --force-full
794 In "sync" forces a full recomputation of the parity. This op‐
795 tion can be used when you add a new parity level, or if you re‐
796 verted back to an old content file using a more recent parity
797 data. Instead of recreating the parity from scratch, this al‐
798 lows to reuse the hashes present in the content file to validate
799 data, and to maintain data protection during the "sync" process
800 using the parity data you have. This option can be used only
801 with "sync".
802 -R, --force-realloc
804 In "sync" forces a full reallocation of files and rebuild of the
805 parity. This option can be used to completely reallocate all
806 the files removing the fragmentation, but reusing the hashes
807 present in the content file to validate data. This option can
808 be used only with "sync". WARNING! This option is for experts
809 only, and it´s highly recommended to not use it. You DO NOT
810 have data protection during the "sync" operation.
811 -l, --log FILE
813 Write a detailed log in the specified file. If this option is
814 not specified, unexpected errors are printed on the screen,
815 likely resulting in too much output in case of many errors. When
816 -l, --log is specified, on the screen, go only fatal errors that
817 makes SnapRAID to stop progress. If the path starts with ´>>´
818 the file is opened in append mode. Occurrences of ´%D´ and ´%T´
819 in the name are replaced with the date and time in the format
820 YYYYMMDD and HHMMSS. Note that in Windows batch files, you´ll
821 have to double the ´%´ char, like result-%%D.log. And to use
822 ´>>´ you´ll have to enclose the name in ", like ">>result.log".
823 To output the log to standard output or standard error, you can
824 use respectively ">&1" and ">&2".
825 -L, --error-limit
827 Sets a new error limit before stopping execution. By default
828 SnapRAID stops if it encounters more than 100 Input/Output er‐
829 rors, meaning that likely a disk is going to die. This options
830 affects "sync" and "scrub", that are allowed to continue after
831 the first bunch of disk errors, to try to complete at most their
832 operations. Instead, "check" and "fix" always stop at the first
833 error.
834 -S, --start BLKSTART
836 Starts the processing from the specified block number. It could
837 be useful to retry to check or fix some specific block, in case
838 of a damaged disk. It´s present mainly for advanced manual re‐
839 covering.
840 -B, --count BLKCOUNT
842 Processes only the specified number of blocks. It´s present
843 mainly for advanced manual recovering.
844 -C, --gen-conf CONTENT_FILE
846 Generates a dummy configuration file from an existing content
847 file. The configuration file is written in the standard output,
848 and it doesn´t overwrite an existing one. This configuration
849 file also contains the information needed to reconstruct the
850 disk mount points, in case you lose the entire system.
851 -v, --verbose
853 Prints more information on the screen. If specified one time,
854 it prints excluded files and more stats. This option has no ef‐
855 fect on the log files.
856 -q, --quiet
858 Prints less information on the screen. If specified one time,
859 removes the progress bar, if two times, the running operations,
860 three times, the info messages, four times the status messages.
861 Fatal errors are always printed on the screen. This option has
862 no effect on the log files.
863 -H, --help
865 Prints a short help screen.
866 -V, --version
868 Prints the program version.
869
871 SnapRAID requires a configuration file to know where your disk array is
872 located, and where storing the parity information.
873 In Unix it´s used the file "/usr/local/etc/snapraid.conf" if it exists,
875 or "/etc/snapraid.conf" otherwise. In Windows it´s used the file
876 "snapraid.conf" in the same directory of "snapraid.exe".
877 It should contain the following options (case sensitive):
879 parity FILE [,FILE] ...
881 Defines the files to use to store the parity information. The parity
882 enables the protection from a single disk failure, like RAID5.
883 You can specify multiples files that should be in different disks.
885 When a file cannot grow anymore, the next one is used. The total space
886 available must be as big as the biggest data disk in the array.
887 You can add additional parity files at later time, but you cannot re‐
889 order or remove them.
890 Leaving the parity disks reserved for parity ensures that it doesn´t
892 get fragmented, improving the performance.
893 In Windows 256 MB are left unused in each disk to avoid the warning
895 about full disks.
896 This option is mandatory and it can be used only one time.
898 (2,3,4,5,6)-parity FILE [,FILE] ...
900 Defines the files to use to store extra parity information.
901 For each parity specified, one additional level of protection is en‐
903 abled:
904 • 2-parity enables RAID6 dual parity.
905 • 3-parity enables triple parity
906 • 4-parity enables quad (four) parity
907 • 5-parity enables penta (five) parity
908 • 6-parity enables hexa (six) parity
909 Each parity level requires the presence of all the previous parity lev‐
911 els.
912 The same considerations of the ´parity´ option apply.
914 These options are optional and they can be used only one time.
916 z-parity FILE [,FILE] ...
918 Defines an alternate file and format to store the triple parity.
919 This option is an alternative at ´3-parity´ mainly intended for low-end
921 CPUs like ARM or AMD Phenom, Athlon and Opteron that don´t support the
922 SSSE3 instructions set. In such cases it provides a better performance.
923 This format is similar, but faster, at the one used by the ZFS RAIDZ3.
925 Like ZFS, it doesn´t work beyond triple parity.
926 When using ´3-parity´ you will be warned if it´s recommended to use the
928 ´z-parity´ format for a performance improvement.
929 It´s possible to convert from one format to another, adjusting the con‐
931 figuration file with the wanted z-parity or 3-parity file, and using
932 ´fix´ to recreate it.
933 content FILE
935 Defines the file to use to store the list and check-sums of all the
936 files present in your disk array.
937 It can be placed in the disk used to store data, parity, or any other
939 disk available. If you use a data disk, this file is automatically ex‐
940 cluded from the "sync" process.
941 This option is mandatory and it can be used more times to save more
943 copies of the same files.
944 You have to store at least one copy for each parity disk used plus one.
946 Using some more doesn´t hurt.
947 data NAME DIR
949 Defines the name and the mount point of the data disks of the array.
950 NAME is used to identify the disk, and it must be unique. DIR is the
951 mount point of the disk in the file-system.
952 You can change the mount point as you like, as long you keep the NAME
954 fixed.
955 You should use one option for each data disk of the array.
957 You can rename later a disk, changing the NAME directly in the configu‐
959 ration file, and then run a ´sync´ command. In the rename case, the
960 association is done using the stored UUID of the disks.
961 nohidden
963 Excludes all the hidden files and directory. In Unix hidden files are
964 the ones starting with ".". In Windows they are the ones with the hid‐
965 den attribute.
966 exclude/include PATTERN
968 Defines the file or directory patterns to exclude and include in the
969 sync process. All the patterns are processed in the specified order.
970 If the first pattern that matches is an "exclude" one, the file is ex‐
972 cluded. If it´s an "include" one, the file is included. If no pattern
973 matches, the file is excluded if the last pattern specified is an "in‐
974 clude", or included if the last pattern specified is an "exclude".
975 See the PATTERN section for more details in the pattern specifications.
977 This option can be used many times.
979 blocksize SIZE_IN_KIBIBYTES
981 Defines the basic block size in kibi bytes of the parity. One kibi
982 bytes is 1024 bytes.
983 The default blocksize is 256 and it should work for most cases.
985 WARNING! This option is for experts only, and it´s highly recommended
987 to not change it. To change again this value in future you´ll have to
988 recreate the whole parity!
989 A reason to use a different hashsize is if you have a lot of small
991 files. In the order of many millions.
992 For each file, even of few bytes, a whole block of parity is allocated,
994 and with many files this may result in a lot of unused parity space.
995 And when you completely fill the parity disk, you are not allowed to
996 add more files in the data disks. Anyway, the wasted parity doesn´t
997 sum between data disks. Wasted space resulting from a high number of
998 files in a data disk, limits only the amount of data in such data disk
999 and not in others.
1000 As approximation, you can assume that half of the block size is wasted
1002 for each file. For example, with 100000 files and a 256 KiB block size,
1003 you are going to waste 13 GB of parity, that may result in 13 GB less
1004 space available in the data disk.
1005 You can get the amount of wasted space in each disk using "status".
1007 This is the amount of space that you must leave free in the data disks,
1008 or use for files not included in the array. If this value is negative,
1009 it means that your are near to fill the parity, and it represents the
1010 space you can still waste.
1011 To avoid the problem, you can use a bigger partition for parity. For
1013 example, if you have the parity partition bigger than 13 GB than data
1014 disks, you have enough extra space to handle up to 100000 files in each
1015 data disk.
1016 A trick to get a bigger parity partition in Linux, is to format it with
1018 the command:
1019 mkfs.ext4 -m 0 -T largefile4 DEVICE
1021 This results in about 1.5% of extra space. Meaning about 60 GB for a 4
1022 TB disk, that allows about 460000 files in each data disk without any
1023 wasted space.
1024 hashsize SIZE_IN_BYTES
1026 Defines the hash size in bytes of the saved blocks.
1027 The default hashsize is 16 bytes (128 bits), and it should work for
1029 most cases.
1030 WARNING! This option is for experts only, and it´s highly recommended
1032 to not change it. To change again this value in future you´ll have to
1033 recreate the whole parity!
1034 A reason to use a different hashsize is if your system has small mem‐
1036 ory. As a rule of thumb SnapRAID usually requires 1 GiB of RAM memory
1037 for each 16 TB of data in the array.
1038 Specifically, to store the hashes of the data, SnapRAID requires about
1040 TS*(1+HS)/BS bytes of RAM memory. Where TS is the total size in bytes
1041 of your disk array, BS is the block size in bytes, and HS is the hash
1042 size in bytes.
1043 For example with 8 disks of 4 TB and a block size of 256 KiB (1 KiB =
1045 1024 bytes), and an hash size of 16, you get:
1046 RAM = (8 * 4 * 10^12) * (1+16) / (256 * 2^10) = 1.93 GiB
1048 Switching to a hash size of 8, you get:
1049 RAM = (8 * 4 * 10^12) * (1+8) / (256 * 2^10) = 1.02 GiB
1051 Switching to a block size of 512, you get:
1052 RAM = (8 * 4 * 10^12) * (1+16) / (512 * 2^10) = 0.96 GiB
1054 Switching to both a hash size of 8, and a block size of 512 you get:
1055 RAM = (8 * 4 * 10^12) * (1+8) / (512 * 2^10) = 0.51 GiB
1057 autosave SIZE_IN_GIGABYTES
1058 Automatically save the state when syncing or scrubbing after the speci‐
1059 fied amount of GB processed. This option is useful to avoid to restart
1060 from scratch long "sync" commands interrupted by a machine crash, or
1061 any other event that may interrupt SnapRAID.
1062 pool DIR
1064 Defines the pooling directory where the virtual view of the disk array
1065 is created using the "pool" command.
1066 The directory must already exist.
1068 share UNC_DIR
1070 Defines the Windows UNC path required to access the disks remotely.
1071 If this option is specified, the symbolic links created in the pool di‐
1073 rectory use this UNC path to access the disks. Without this option the
1074 symbolic links generated use only local paths, not allowing to share
1075 the pool directory in the network.
1076 The symbolic links are formed using the specified UNC path, adding the
1078 disk name as specified in the "disk" option, and finally adding the
1079 file dir and name.
1080 This option is only required for Windows.
1082 smartctl DISK/PARITY OPTIONS...
1084 Defines a custom smartctl command to obtain the SMART attributes for
1085 each disk. This may be required for RAID controllers and for some USB
1086 disk that cannot be auto-detected.
1087 DISK is the same disk name specified in the "disk" option. PARITY is
1089 one of the parity name as "parity,(1,2,3,4,5,6,z)-parity".
1090 In the specified OPTIONS, the "%s" string is replaced by the device
1092 name. Note that in case of RAID controllers the device is likely fixed,
1093 and you don´t have to use "%s".
1094 Refers at the smartmontools documentation about the possible options:
1096 https://www.smartmontools.org/wiki/Supported_RAID-Controllers
1098 https://www.smartmontools.org/wiki/Supported_USB-Devices
1099 Examples
1100 An example of a typical configuration for Unix is:
1101 parity /mnt/diskp/snapraid.parity
1103 content /mnt/diskp/snapraid.content
1104 content /var/snapraid/snapraid.content
1105 data d1 /mnt/disk1/
1106 data d2 /mnt/disk2/
1107 data d3 /mnt/disk3/
1108 exclude /lost+found/
1109 exclude /tmp/
1110 smartctl d1 -d sat %s
1111 smartctl d2 -d usbjmicron %s
1112 smartctl parity -d areca,1/1 /dev/sg0
1113 smartctl 2-parity -d areca,2/1 /dev/sg0
1114 An example of a typical configuration for Windows is:
1115 parity E:\snapraid.parity
1117 content E:\snapraid.content
1118 content C:\snapraid\snapraid.content
1119 data d1 G:\array\
1120 data d2 H:\array\
1121 data d3 I:\array\
1122 exclude Thumbs.db
1123 exclude \$RECYCLE.BIN
1124 exclude \System Volume Information
1125 smartctl d1 -d sat %s
1126 smartctl d2 -d usbjmicron %s
1127 smartctl parity -d areca,1/1 /dev/arcmsr0
1128 smartctl 2-parity -d areca,2/1 /dev/arcmsr0
1130 Patterns are used to select a subset of files to exclude or include in
1131 the process.
1132 There are four different types of patterns:
1134 FILE Selects any file named as FILE. You can use any globbing charac‐
1136 ter like * and ?, and char classes like [a-z]. This pattern is
1137 applied only to files and not to directories.
1138 DIR/ Selects any directory named DIR and everything inside. You can
1140 use any globbing character like * and ?. This pattern is ap‐
1141 plied only to directories and not to files.
1142 /PATH/FILE
1144 Selects the exact specified file path. You can use any globbing
1145 character like * and ? but they never match a directory slash.
1146 This pattern is applied only to files and not to directories.
1147 /PATH/DIR/
1149 Selects the exact specified directory path and everything in‐
1150 side. You can use any globbing character like * and ? but they
1151 never match a directory slash. This pattern is applied only to
1152 directories and not to files.
1153 Note that when you specify an absolute path starting with /, it´s ap‐
1155 plied at the array root dir and not at the local file-system root dir.
1156 In Windows you can use the backslash \ instead of the forward slash /.
1158 Note that Windows system directories, junctions, mount points, and any
1159 other Windows special directory are treated just as files, meaning that
1160 to exclude them you must use a file rule, and not a directory one.
1161 If the file name you want to use really contains a ´*´, ´?´, ´[´, or
1163 ´]´ char, you have to escape it to avoid to have interpreted as a glob‐
1164 bing character. In Unix the escape char is ´\´, in Windows it´s ´^´.
1165 Note that when the pattern is on the command line, you have to double
1166 the escape character to avoid to have it interpreted by the command
1167 shell.
1168 In the configuration file, you can use different strategies to filter
1170 the files to process. The simplest one is to use only "exclude" rules
1171 to remove all the files and directories you do not want to process. For
1172 example:
1173 # Excludes any file named "*.unrecoverable"
1175 exclude *.unrecoverable
1176 # Excludes the root directory "/lost+found"
1177 exclude /lost+found/
1178 # Excludes any sub-directory named "tmp"
1179 exclude tmp/
1180 The opposite way is to define only the file you want to process, using
1181 only "include" rules. For example:
1182 # Includes only some directories
1184 include /movies/
1185 include /musics/
1186 include /pictures/
1187 The final way, is to mix "exclude" and "include" rules. In this case
1188 take care that the order of rules is important. Previous rules have the
1189 precedence over the later ones. To get things simpler you can first
1190 have all the "exclude" rules and then all the "include" ones. For exam‐
1191 ple:
1192 # Excludes any file named "*.unrecoverable"
1194 exclude *.unrecoverable
1195 # Excludes any sub-directory named "tmp"
1196 exclude tmp/
1197 # Includes only some directories
1198 include /movies/
1199 include /musics/
1200 include /pictures/
1201 On the command line, using the -f option, you can only use "include"
1202 patterns. For example:
1203 # Checks only the .mp3 files.
1205 # Note the "" use to avoid globbing expansion by the shell in Unix.
1206 snapraid -f "*.mp3" check
1207 In Unix, when using globbing chars in the command line, you have to
1208 quote them. Otherwise the shell will try to expand them.
1209
1211 SnapRAID stores the list and check-sums of your files in the content
1212 file.
1213 It´s a binary file, listing all the files present in your disk array,
1215 with all the check-sums to verify their integrity.
1216 This file is read and written by the "sync" and "scrub" commands, and
1218 read by "fix", "check" and "status".
1219
1221 SnapRAID stores the parity information of your array in the parity
1222 files.
1223 They are binary files, containing the computed parity of all the blocks
1225 defined in the "content" file.
1226 These files are read and written by the "sync" and "fix" commands, and
1228 only read by "scrub" and "check".
1229
1231 SnapRAID in Unix ignores any encoding. It reads and stores the file
1232 names with the same encoding used by the file-system.
1233 In Windows all the names read from the file-system are converted and
1235 processed in the UTF-8 format.
1236 To have the file names printed correctly you have to set the Windows
1238 console in the UTF-8 mode, with the command "chcp 65001", and use a
1239 TrueType font like "Lucida Console" as console font. Note that it has
1240 effect only on the printed file names, if you redirect the console out‐
1241 put to a file, the resulting file is always in the UTF-8 format.
1242
1244 This file is Copyright (C) 2011 Andrea Mazzoleni
1245
1247 rsync(1)
1248 SnapRAID Backup For Disk Arrays(1)