1MOGTOOL(1) User Contributed Perl Documentation MOGTOOL(1)
2
6 mogtool -- Inject/extract data to/from a MogileFS installation
7 WARNING: this utility is deprecated! See MogileFS::Utils
9
11 $ mogtool [general-opts] <command> [command-opts] <command-args>
12 $ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
14 $ mogtool --conf=foo.conf ...
15 $ mogtool inject thefile.tgz thefilekey
17 $ mogtool inject --bigfile thebigfile.tgz thefilekey
18 $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
19 $ mogtool inject --bigfile --gzip mydirectory thedirkey
20 $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
21 $ mogtool inject --nobigfile bigcontiguousfile bigcfilekey
22 $ mogtool inject --bigfile --gzip --verify \
24 --description="Description" \
25 --receipt="foo@bar.com, baz@bar.com" \
26 --concurrent=5 --chunksize=32M \
27 somehugefile thefilekey
28 $ mogtool extract thefilekey thenewfile.tgz
30 $ mogtool extract thefilekey -
31 $ mogtool extract --bigfile thedirkey .
32 $ mogtool extract --bigfile --asfile thedirkey thefile.tgz
33 $ mogtool extract --bigfile thedevkey /dev/hda4
34 $ mogtool delete thekey
36 $ mogtool locate --noverify thekey
38 $ mogtool locate --bigfile thekey
39 $ mogtool list
41 $ mogtool listkey key
42
44 --debug
45 Turn on MogileFS debug output.
46 --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
48 Specify one or more trackers for your MogileFS installation. Note
49 that you can specify preferred IPs to override the default IPs
50 with. So it would look something like 10.10.0.1/10.0.0.1:8081.
51 --domain=<domain>
53 Set the MogileFS domain to use.
54 --class=<class>
56 Set the class within the domain to use. Defaults to _default.
57 --conf=<file>
59 Specify a configuration file to load from.
60 --lib=<directory>
62 Specify a directory to use as a library path. Right now, this
63 should be the directory where you expect to find the MogileFS.pm
64 file, if it's not actually installed.
65
67 inject|i
68 Insert a resource into MogileFS. See "INJECT OPTIONS" and "INJECT
69 ARGUMENTS" for the rest of how to use the inject mode.
70 extract|x
72 Extract a resource from MogileFS. See "EXTRACT OPTIONS" and
73 "EXTRACT ARGUMENTS" for how to use extract.
74 delete|rm
76 Delete a resource. See "DELETE OPTIONS" and "DELETE ARGUMENTS".
77 locate|lo key
79 List the paths to the file identified by the given key.
80 list|ls
82 List all big files contained in MogileFS. No options, no
83 arguments.
84 listkey|lsk key
86 List all files which match the key. Key is just a prefix, and this
87 will list all keys which match the prefix. So if you specify key as
88 "ABC1" then you'll get all keys which start with the characters
89 "ABC1"
90
92 The following options are used to control the behavior of the injector.
93 --bigfile|-b
95 If specified, use chunking to break the resource into manageable
96 pieces.
97 --chunksize=<size>[B|K|M|G]
99 When instructed to break files into chunks, the injector will use
100 the specified chunk size as the maximum chunk size. Defaults to
101 64M. You can specify the chunk size manually and specify the
102 units--defaults to bytes.
103 --gzip|-z
105 If specified, mogtool will gzip the data as it's going into
106 MogileFS. The resource will be marked as compressed.
107 Note that you do not need to specify this if the resource is
109 already gzipped, but it doesn't hurt. (We automatically detect
110 that and mark it as compressed.)
111 --overwrite
113 If you previously were working on injecting a big file as chunks
114 and the process died, normally mogtool refuses to do it again.
115 Specify this option to force the overwrite of that file.
116 NOTE: Other than in the above case (partial failure), mogtool will
118 not prompt before overwriting an existing file.
119 --verify
121 If on, we do a full MD5 verification of every chunk after it is
122 replicated. This can take a while on large files!
123 --description=<text>
125 Specifies a description for this file. Optional, but assists in
126 reporting and listing the large files in MogileFS. (This is also
127 displayed in any receipts that are created.)
128 --receipt=<email address>[, <email address>]*
130 If specified, emails a copy of the receipt file to the specified
131 comma-separated email addresses. Also creates a local filesystem
132 copy of the receipt file.
133 --concurrent=<number>
135 Specifies the number of concurrent processes to run for MogileFS
136 insertion. If you are noticing mogtool spend most of it's time
137 waiting for children and not actually buffering data, you may wish
138 to raise this number. The default is 1 but we've found 3 or 4 work
139 well.
140
142 resource
143 What you actually want to inject. This can be a file, directory,
144 or a raw partition in the format /dev/X.
145 Please see "USAGE EXAMPLES" for more information on how to inject
147 these different types of resources and the differences thereof.
148 key Specifies the key to save this file to. For big files, the key is
150 actually "_big_N:key" and "key,#" where N is one of a bunch of
151 things we use and # is the chunk number.
152 Generally, you want this to be descriptive so you remember what it
154 is later and can identify the file just by looking at the key.
155
157 --bigfile|-b
158 If specified, indicates that this resource was chunked on injection
159 and should be reassembled for extraction.
160 --gzip|-z
162 Specifies to mogtool that it should ungzip the output if and only
163 if it was compressed when inserted into the MogileFS system. So,
164 if you're extracting a file that wasn't gzipped to begin with, this
165 doesn't do anything.
166 --asfile
168 Useful when extracting something previously inserted as a
169 directory--this option instructs mogtool to treat the resource as a
170 file and not actually run it through tar for decompression.
171
173 key Specifies the key to get the file from.
174 destination
176 What destination means varies depending on what type of resource
177 you're extracting. However, no matter what, you can specify a
178 single dash (-) to mean STDOUT.
179 Please see the usage examples for more information on how extract
181 works.
182
184 --bigfile|-b
185 The resource is a "big file" and all chunks should be deleted.
186
188 key Specifies the key of the file to delete.
189
191 --verify
192 Verify that the returned paths actually contain the file. The
193 locate commands defaults to verify, you can disable it with
194 --noverify.
195 --bigfile|-b
197 The resource is a "big file" and the locations of the information
198 key should be printed.
199
201 key Specifies the key of the file to locate
202
204 0 Success during operation.
205 1 During the locate, list, or listkey operation, the key was not
207 found.
208 2 Some fatal error occurred.
210
212 Please note that all examples assume you have a default config file
213 that contains the tracker and domain to use. Saves us from having to
214 clutter up the command line.
215 Small Files (<64MB)
217 When it comes to using small files, mogtool is very, very easy.
218 Injection
220 $ mogtool inject foo.dbm foo.dbm.2004.12
222 Injects the file foo.dbm into MogileFS under the key of
224 foo.dbm.2004.12.
225 $ mogtool inject --gzip foo.dbm foo.dbm.2004.12
227 Injects the same file to the same key, but compresses it on the fly for
229 you.
230 Extraction
232 $ mogtool extract foo.dbm.2004.12 newfoo.dbm
234 Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.
236 $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
238 Gets the file and automatically decompresses it, if and only if it was
240 compressed. So basically, you can turn on gzip in your config file and
241 mogtool will do the smart thing each time.
242 $ mogtool extract foo.dbm.2004.12 -
244 Print the resource to standard out. If you want, you can pipe it
246 somewhere or redirect to a file (but why not just specify the
247 filename?).
248 Large Files (>64MB)
250 Given mogtool's ability to break files into chunks and later reassemble
251 them, inserting large files (even files over the 4GB barrier) is
252 relatively easy.
253 Injection
255 $ mogtool inject --bigfile largefile.dat largefile.dat
257 As expected, inserts the file largefile.dat into the MogileFS system
259 under the name largefile.dat. Not very creative. Uses the default
260 64MB chunks.
261 $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
263 Specify to use 16MB chunks instead of the default. Otherwise, the
265 same.
266 $ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
268 Do it again, but specify 1000KB chunks, gzip automatically, and upload
270 it under a different key somekey.
271 $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
273 Same as above, but use 5 children processes for uploading chunks to
275 MogileFS. This can take up to 300MB of memory in this example! (It
276 tends to use about (concurrency + 1) * chunksize bytes.)
277 $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
279 --receipt="foo@bar.com" --verify --description="A large file" \
280 largefile.dat somekey
281 Break this file into 128MB chunks, set a description, use 3 children to
283 upload them, gzip the file as you go, do a full MD5 verification of
284 every chunk, then email a receipt with all of the MogileFS paths to me.
285 Lots of flexibility with mogtool.
287 Extraction
289 $ mogtool extract --bigfile somekey newfile.dat
291 In its basic form, extracts the previously inserted large file and
293 saves it as newfile.dat.
294 $ mogtool extract --bigfile --gzip somekey newfile.dat
296 If the file was gzipped on entry, ungzip it and save the result. If it
298 wasn't gzipped, then we just save it.
299 Directories
301 Directories are easily injected and extracted with mogtool. To create
302 the data stream that is inserted into MogileFS, we use tar.
303 Injection
305 $ mogtool inject --bigfile mydir mykey
307 Run mydir through tar and then save it as mykey.
309 $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
311 Inject, but also gzip and use multiple injectors.
313 Note how this is just like injecting a large file. See injection
315 examples for large files for more examples.
316 Extraction
318 $ mogtool extract --bigfile mykey .
320 Extract the previously injected directory mykey to your local
322 directory.
323 $ mogtool extract --bigfile --asfile mykey foo.tar
325 Take the previously generated tarball and save it as foo.tar. Simply
327 creates the file instead of extracting everything inside.
328 Partitions/Devices
330 mogtool has the ability to inject raw partitions into MogileFS and to
331 retrieve them later and write them back to a partition. They're
332 treated just like directories for the most part, we just don't pipe
333 things through tar.
334 Injection
336 $ mogtool inject --bigfile /dev/hda3 hda3.backup
338 Save a raw copy of your partition /dev/hda3 to the key hda3.backup.
340 $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
342 Same, but compress on the fly during injection.
344 Extraction
346 $ mogtool extract --bigfile hda3.backup /dev/hda4
348 Extract the partition at hda3.backup to the partition /dev/hda4.
350 WARNING: mogtool won't ask for confirmation, make sure you don't
351 mistype partition numbers!
352 Deleting a Resource
354 WARNING: Please make sure you're specifying the right parameter, as
355 delete does not prompt for confirmation of the request!
356 $ mogtool delete thekey
358 Delete a normal file.
360 $ mogtool delete --bigfile thekey
362 Delete a chunked file--this deletes all chunks and the receipt, so the
364 file is gone.
365 Listing Big Files
367 $ mogtool list backup
368 Lists all large files stored in MogileFS. It is not possible to list
370 all normal files at this time.
371 Listing Files Matching a Key
373 $ mogtool listkey abc1
374 Lists all files in MogileFS whose keys start with the characters
376 "abc1".
377
379 Instead of adding a ton of options to the command line every time,
380 mogtool enables you to create a default configuration file that it will
381 read all of the options from. It searches two locations for a default
382 configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.
383 (Alternately, you can specify --conf=whatever as an option on the
384 command line.)
385 The file can consist of any number of the following items:
387 trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
389 domain = mogiledomain
390 class = fileclass
391 lib = /home/foo/lib
392 gzip = 1
393 big = 1
394 overwrite = 1
395 chunksize = 32M
396 receipt = foo@bar.com, baz@bar.com
397 verify = 1
398 concurrent = 3
399
401 None? Send me any you find! :)
402
404 --concurrent for extract
405 It would be nice to have concurrent extraction going on.
406 recover mode
408 If the receipt file is ever corrupt in MogileFS it would be useful
409 to recover a file given just a receipt. It would have the same
410 arguments as the extract mode, except use a receipt file as the
411 data source.
412 partition size verification
414 We can easily get the partition size when we save one to MogileFS,
415 so we should use that information to determine during extraction if
416 a target partition is going to be big enough.
417 on the fly gzip extraction
419 Right now we can gzip on an injection, but we should support doing
420 decompression on the fly coming out of MogileFS.
421 make list take a prefix
423 If you can specify a prefix, that makes things easier for finding
424 small files that are stored in MogileFS.
425 more information on list
427 Have list load up the info file and parse it for information about
428 each of the big files being stored. Maybe have this as an option
429 (-l). (This means the reading and parsing of info files should be
430 abstracted into a function.)
431
433 Mark Smith <junior@danga.com> - most of the implementation and
434 maintenance.
435 Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.
437 Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.
439 Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
441
443 Hey! The above document had some coding errors, which are explained
444 below:
445 Around line 289:
447 Expected text after =item, not a number
448 Around line 293:
450 Expected text after =item, not a number
451perl v5.32.0 2020-07-28 MOGTOOL(1)