1MOGTOOL(1) User Contributed Perl Documentation MOGTOOL(1)
2
6 mogtool -- Inject/extract data to/from a MogileFS installation
7
9 $ mogtool [general-opts] <command> [command-opts] <command-args>
10 $ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
12 $ mogtool --conf=foo.conf ...
13 $ mogtool inject thefile.tgz thefilekey
15 $ mogtool inject --bigfile thebigfile.tgz thefilekey
16 $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
17 $ mogtool inject --bigfile --gzip mydirectory thedirkey
18 $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
19 $ mogtool inject --bigfile --gzip --verify \
21 --description="Description" \
22 --receipt="foo@bar.com, baz@bar.com" \
23 --concurrent=5 --chunksize=32M \
24 somehugefile thefilekey
25 $ mogtool extract thefilekey thenewfile.tgz
27 $ mogtool extract thefilekey -
28 $ mogtool extract --bigfile thedirkey .
29 $ mogtool extract --bigfile --asfile thedirkey thefile.tgz
30 $ mogtool extract --bigfile thedevkey /dev/hda4
31 $ mogtool delete thekey
33 $ mogtool locate thekey
35 $ mogtool locate --verify --bigfile thekey
36 $ mogtool list
38 $ mogtool listkey
39
41 --debug
42 Turn on MogileFS debug output.
43 --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
45 Specify one or more trackers for your MogileFS installation. Note
46 that you can specify preferred IPs to override the default IPs
47 with. So it would look something like 10.10.0.1/10.0.0.1:8081.
48 --domain=<domain>
50 Set the MogileFS domain to use.
51 --class=<class>
53 Set the class within the domain to use. Defaults to _default.
54 --conf=<file>
56 Specify a configuration file to load from.
57 --lib=<directory>
59 Specify a directory to use as a library path. Right now, this
60 should be the directory where you expect to find the MogileFS.pm
61 file, if it's not actually installed.
62
64 inject⎪i
65 Insert a resource into MogileFS. See "INJECT OPTIONS" and "INJECT
66 ARGUMENTS" for the rest of how to use the inject mode.
67 extract⎪x
69 Extract a resource from MogileFS. See "EXTRACT OPTIONS" and
70 "EXTRACT ARGUMENTS" for how to use extract.
71 delete⎪rm
73 Delete a resource. See "DELETE OPTIONS" and "DELETE ARGUMENTS".
74 locate⎪lo key
76 List the paths to the file identified by the given key.
77 list⎪ls
79 List all big files contained in MogileFS. No options, no argu‐
80 ments.
81 listkey⎪lsk key
83 List all files which match the key. Key is just a prefix, and this
84 will list all keys which match the prefix. So if you specify key as
85 "ABC1" then you'll get all keys which start with the characters
86 "ABC1"
87
89 The following options are used to control the behavior of the injector.
90 --bigfile⎪-b
92 If specified, use chunking to break the resource into manageable
93 pieces.
94 --chunksize=<size>[B⎪K⎪M⎪G]
96 When instructed to break files into chunks, the injector will use
97 the specified chunk size as the maximum chunk size. Defaults to
98 64M. You can specify the chunk size manually and specify the
99 units--defaults to bytes.
100 --gzip⎪-z
102 If specified, mogtool will gzip the data as it's going into
103 MogileFS. The resource will be marked as compressed.
104 Note that you do not need to specify this if the resource is
106 already gzipped, but it doesn't hurt. (We automatically detect
107 that and mark it as compressed.)
108 --overwrite
110 If you previously were working on injecting a big file as chunks
111 and the process died, normally mogtool refuses to do it again.
112 Specify this option to force the overwrite of that file.
113 NOTE: Other than in the above case (partial failure), mogtool will
115 not prompt before overwriting an existing file.
116 --verify
118 If on, we do a full MD5 verification of every chunk after it is
119 replicated. This can take a while on large files!
120 --description=<text>
122 Specifies a description for this file. Optional, but assists in
123 reporting and listing the large files in MogileFS. (This is also
124 displayed in any receipts that are created.)
125 --receipt=<email address>[, <email address>]*
127 If specified, emails a copy of the receipt file to the specified
128 comma-separated email addresses. Also creates a local filesystem
129 copy of the receipt file.
130 --concurrent=<number>
132 Specifies the number of concurrent processes to run for MogileFS
133 insertion. If you are noticing mogtool spend most of it's time
134 waiting for children and not actually buffering data, you may wish
135 to raise this number. The default is 1 but we've found 3 or 4 work
136 well.
137
139 resource
140 What you actually want to inject. This can be a file, directory,
141 or a raw partition in the format /dev/X.
142 Please see "USAGE EXAMPLES" for more information on how to inject
144 these different types of resources and the differences thereof.
145 key Specifies the key to save this file to. For big files, the key is
147 actually "_big_N:key" and "key,#" where N is one of a bunch of
148 things we use and # is the chunk number.
149 Generally, you want this to be descriptive so you remember what it
151 is later and can identify the file just by looking at the key.
152
154 --bigfile⎪-b
155 If specified, indicates that this resource was chunked on injection
156 and should be reassembled for extraction.
157 --gzip⎪-z
159 Specifies to mogtool that it should ungzip the output if and only
160 if it was compressed when inserted into the MogileFS system. So,
161 if you're extracting a file that wasn't gzipped to begin with, this
162 doesn't do anything.
163 --asfile
165 Useful when extracting something previously inserted as a direc‐
166 tory--this option instructs mogtool to treat the resource as a file
167 and not actually run it through tar for decompression.
168
170 key Specifies the key to get the file from.
171 destination
173 What destination means varies depending on what type of resource
174 you're extracting. However, no matter what, you can specify a sin‐
175 gle dash (-) to mean STDOUT.
176 Please see the usage examples for more information on how extract
178 works.
179
181 --bigfile⎪-b
182 The resource is a "big file" and all chunks should be deleted.
183
185 key Specifies the key of the file to delete.
186
188 --verify
189 Verify that the returned paths actually contain the file.
190 --bigfile⎪-b
192 The resource is a "big file" and the locations of the information
193 key should be printed.
194
196 key Specifies the key of the file to locate
197
199 0 Sucess during operation.
200 1 During the locate, list, or listkey operation, the key was not
202 found.
203 2 Some fatal error occured.
205
207 Please note that all examples assume you have a default config file
208 that contains the tracker and domain to use. Saves us from having to
209 clutter up the command line.
210 Small Files (<64MB)
212 When it comes to using small files, mogtool is very, very easy.
214 Injection
216 $ mogtool inject foo.dbm foo.dbm.2004.12
218 Injects the file foo.dbm into MogileFS under the key of
220 foo.dbm.2004.12.
221 $ mogtool inject --gzip foo.dbm foo.dbm.2004.12
223 Injects the same file to the same key, but compresses it on the fly for
225 you.
226 Extraction
228 $ mogtool extract foo.dbm.2004.12 newfoo.dbm
230 Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.
232 $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
234 Gets the file and automatically decompresses it, if and only if it was
236 compressed. So basically, you can turn on gzip in your config file and
237 mogtool will do the smart thing each time.
238 $ mogtool extract foo.dbm.2004.12 -
240 Print the resource to standard out. If you want, you can pipe it some‐
242 where or redirect to a file (but why not just specify the filename?).
243 Large Files (>64MB)
245 Given mogtool's ability to break files into chunks and later reassemble
247 them, inserting large files (even files over the 4GB barrier) is rela‐
248 tively easy.
249 Injection
251 $ mogtool inject --bigfile largefile.dat largefile.dat
253 As expected, inserts the file largefile.dat into the MogileFS system
255 under the name largefile.dat. Not very creative. Uses the default
256 64MB chunks.
257 $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
259 Specify to use 16MB chunks instead of the default. Otherwise, the
261 same.
262 $ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
264 Do it again, but specify 1000KB chunks, gzip automatically, and upload
266 it under a different key somekey.
267 $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
269 Same as above, but use 5 children processes for uploading chunks to
271 MogileFS. This can take up to 300MB of memory in this example! (It
272 tends to use about (concurrency + 1) * chunksize bytes.)
273 $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
275 --receipt="foo@bar.com" --verify --description="A large file" \
276 largefile.dat somekey
277 Break this file into 128MB chunks, set a description, use 3 children to
279 upload them, gzip the file as you go, do a full MD5 verification of
280 every chunk, then email a receipt with all of the MogileFS paths to me.
281 Lots of flexibility with mogtool.
283 Extraction
285 $ mogtool extract --bigfile somekey newfile.dat
287 In its basic form, extracts the previously inserted large file and
289 saves it as newfile.dat.
290 $ mogtool extract --bigfile --gzip somekey newfile.dat
292 If the file was gzipped on entry, ungzip it and save the result. If it
294 wasn't gzipped, then we just save it.
295 Directories
297 Directories are easily injected and extracted with mogtool. To create
299 the data stream that is inserted into MogileFS, we use tar.
300 Injection
302 $ mogtool inject --bigfile mydir mykey
304 Run mydir through tar and then save it as mykey.
306 $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
308 Inject, but also gzip and use multiple injectors.
310 Note how this is just like injecting a large file. See injection exam‐
312 ples for large files for more examples.
313 Extraction
315 $ mogtool extract --bigfile mykey .
317 Extract the previously injected directory mykey to your local direc‐
319 tory.
320 $ mogtool extract --bigfile --asfile mykey foo.tar
322 Take the previously generated tarball and save it as foo.tar. Simply
324 creates the file instead of extracting everything inside.
325 Partitions/Devices
327 mogtool has the ability to inject raw partitions into MogileFS and to
329 retrieve them later and write them back to a partition. They're
330 treated just like directories for the most part, we just don't pipe
331 things through tar.
332 Injection
334 $ mogtool inject --bigfile /dev/hda3 hda3.backup
336 Save a raw copy of your partition /dev/hda3 to the key hda3.backup.
338 $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
340 Same, but compress on the fly during injection.
342 Extraction
344 $ mogtool extract --bigfile hda3.backup /dev/hda4
346 Extract the partition at hda3.backup to the partition /dev/hda4. WARN‐
348 ING: mogtool won't ask for confirmation, make sure you don't mistype
349 partition numbers!
350 Deleting a Resource
352 WARNING: Please make sure you're specifying the right parameter, as
354 delete does not prompt for confirmation of the request!
355 $ mogtool delete thekey
357 Delete a normal file.
359 $ mogtool delete --bigfile thekey
361 Delete a chunked file--this deletes all chunks and the receipt, so the
363 file is gone.
364 Listing Big Files
366 $ mogtool list backup
368 Lists all large files stored in MogileFS. It is not possible to list
370 all normal files at this time.
371 Listing Files Matching a Key
373 $ mogtool listkey abc1
375 Lists all files in MogileFS whose keys start with the characters
377 "abc1".
378
380 Instead of adding a ton of options to the command line every time, mog‐
381 tool enables you to create a default configuration file that it will
382 read all of the options from. It searches two locations for a default
383 configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf. (Alter‐
384 nately, you can specify --conf=whatever as an option on the command
385 line.)
386 The file can consist of any number of the following items:
388 trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
390 domain = mogiledomain
391 class = fileclass
392 lib = /home/foo/lib
393 gzip = 1
394 big = 1
395 overwrite = 1
396 chunksize = 32M
397 receipt = foo@bar.com, baz@bar.com
398 verify = 1
399 concurrent = 3
400
402 None? Send me any you find! :)
403
405 --concurrent for extract
406 It would be nice to have concurrent extraction going on.
407 recover mode
409 If the receipt file is ever corrupt in MogileFS it would be useful
410 to recover a file given just a receipt. It would have the same
411 arguments as the extract mode, except use a receipt file as the
412 data source.
413 partition size verification
415 We can easily get the partition size when we save one to MogileFS,
416 so we should use that information to determine during extraction if
417 a target partition is going to be big enough.
418 on the fly gzip extraction
420 Right now we can gzip on an injection, but we should support doing
421 decompression on the fly coming out of MogileFS.
422 make list take a prefix
424 If you can specify a prefix, that makes things easier for finding
425 small files that are stored in MogileFS.
426 more information on list
428 Have list load up the info file and parse it for information about
429 each of the big files being stored. Maybe have this as an option
430 (-l). (This means the reading and parsing of info files should be
431 abstracted into a function.)
432
434 Mark Smith <junior@danga.com> - most of the implementation and mainte‐
435 nance.
436 Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.
438 Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.
440 Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
442perl v5.8.8 2007-08-01 MOGTOOL(1)