1MOGTOOL(1)            User Contributed Perl Documentation           MOGTOOL(1)
2
3
4

NAME

6       mogtool -- Inject/extract data to/from a MogileFS installation
7
8       WARNING: this utility is deprecated! See MogileFS::Utils
9

SYNOPSIS

11           $ mogtool [general-opts] <command> [command-opts] <command-args>
12
13           $ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
14           $ mogtool --conf=foo.conf ...
15
16           $ mogtool inject thefile.tgz thefilekey
17           $ mogtool inject --bigfile thebigfile.tgz thefilekey
18           $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
19           $ mogtool inject --bigfile --gzip mydirectory thedirkey
20           $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
21           $ mogtool inject --nobigfile bigcontiguousfile bigcfilekey
22
23           $ mogtool inject --bigfile --gzip --verify \
24                            --description="Description" \
25                            --receipt="foo@bar.com, baz@bar.com" \
26                            --concurrent=5 --chunksize=32M \
27                            somehugefile thefilekey
28
29           $ mogtool extract thefilekey thenewfile.tgz
30           $ mogtool extract thefilekey -
31           $ mogtool extract --bigfile thedirkey .
32           $ mogtool extract --bigfile --asfile thedirkey thefile.tgz
33           $ mogtool extract --bigfile thedevkey /dev/hda4
34
35           $ mogtool delete thekey
36
37           $ mogtool locate --noverify thekey
38           $ mogtool locate --bigfile thekey
39
40           $ mogtool list
41           $ mogtool listkey key
42

GENERAL OPTIONS

44       --debug
45           Turn on MogileFS debug output.
46
47       --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
48           Specify one or more trackers for your MogileFS installation.  Note
49           that you can specify preferred IPs to override the default IPs
50           with.  So it would look something like 10.10.0.1/10.0.0.1:8081.
51
52       --domain=<domain>
53           Set the MogileFS domain to use.
54
55       --class=<class>
56           Set the class within the domain to use.  Defaults to _default.
57
58       --conf=<file>
59           Specify a configuration file to load from.
60
61       --lib=<directory>
62           Specify a directory to use as a library path.  Right now, this
63           should be the directory where you expect to find the MogileFS.pm
64           file, if it's not actually installed.
65

COMMANDS

67       inject|i
68           Insert a resource into MogileFS.  See "INJECT OPTIONS" and "INJECT
69           ARGUMENTS" for the rest of how to use the inject mode.
70
71       extract|x
72           Extract a resource from MogileFS.  See "EXTRACT OPTIONS" and
73           "EXTRACT ARGUMENTS" for how to use extract.
74
75       delete|rm
76           Delete a resource.  See "DELETE OPTIONS" and "DELETE ARGUMENTS".
77
78       locate|lo key
79           List the paths to the file identified by the given key.
80
81       list|ls
82           List all big files contained in MogileFS.  No options, no
83           arguments.
84
85       listkey|lsk key
86           List all files which match the key. Key is just a prefix, and this
87           will list all keys which match the prefix. So if you specify key as
88           "ABC1" then you'll get all keys which start with the characters
89           "ABC1"
90

INJECT OPTIONS

92       The following options are used to control the behavior of the injector.
93
94       --bigfile|-b
95           If specified, use chunking to break the resource into manageable
96           pieces.
97
98       --chunksize=<size>[B|K|M|G]
99           When instructed to break files into chunks, the injector will use
100           the specified chunk size as the maximum chunk size.  Defaults to
101           64M.  You can specify the chunk size manually and specify the
102           units--defaults to bytes.
103
104       --gzip|-z
105           If specified, mogtool will gzip the data as it's going into
106           MogileFS.  The resource will be marked as compressed.
107
108           Note that you do not need to specify this if the resource is
109           already gzipped, but it doesn't hurt.  (We automatically detect
110           that and mark it as compressed.)
111
112       --overwrite
113           If you previously were working on injecting a big file as chunks
114           and the process died, normally mogtool refuses to do it again.
115           Specify this option to force the overwrite of that file.
116
117           NOTE: Other than in the above case (partial failure), mogtool will
118           not prompt before overwriting an existing file.
119
120       --verify
121           If on, we do a full MD5 verification of every chunk after it is
122           replicated.  This can take a while on large files!
123
124       --description=<text>
125           Specifies a description for this file.  Optional, but assists in
126           reporting and listing the large files in MogileFS.  (This is also
127           displayed in any receipts that are created.)
128
129       --receipt=<email address>[, <email address>]*
130           If specified, emails a copy of the receipt file to the specified
131           comma-separated email addresses.  Also creates a local filesystem
132           copy of the receipt file.
133
134       --concurrent=<number>
135           Specifies the number of concurrent processes to run for MogileFS
136           insertion.  If you are noticing mogtool spend most of it's time
137           waiting for children and not actually buffering data, you may wish
138           to raise this number.  The default is 1 but we've found 3 or 4 work
139           well.
140

INJECT ARGUMENTS

142       resource
143           What you actually want to inject.  This can be a file, directory,
144           or a raw partition in the format /dev/X.
145
146           Please see "USAGE EXAMPLES" for more information on how to inject
147           these different types of resources and the differences thereof.
148
149       key Specifies the key to save this file to.  For big files, the key is
150           actually "_big_N:key" and "key,#" where N is one of a bunch of
151           things we use and # is the chunk number.
152
153           Generally, you want this to be descriptive so you remember what it
154           is later and can identify the file just by looking at the key.
155

EXTRACT OPTIONS

157       --bigfile|-b
158           If specified, indicates that this resource was chunked on injection
159           and should be reassembled for extraction.
160
161       --gzip|-z
162           Specifies to mogtool that it should ungzip the output if and only
163           if it was compressed when inserted into the MogileFS system.  So,
164           if you're extracting a file that wasn't gzipped to begin with, this
165           doesn't do anything.
166
167       --asfile
168           Useful when extracting something previously inserted as a
169           directory--this option instructs mogtool to treat the resource as a
170           file and not actually run it through tar for decompression.
171

EXTRACT ARGUMENTS

173       key Specifies the key to get the file from.
174
175       destination
176           What destination means varies depending on what type of resource
177           you're extracting.  However, no matter what, you can specify a
178           single dash (-) to mean STDOUT.
179
180           Please see the usage examples for more information on how extract
181           works.
182

DELETE OPTIONS

184       --bigfile|-b
185           The resource is a "big file" and all chunks should be deleted.
186

DELETE ARGUMENTS

188       key Specifies the key of the file to delete.
189

LOCATE OPTIONS

191       --verify
192           Verify that the returned paths actually contain the file.  The
193           locate commands defaults to verify, you can disable it with
194           --noverify.
195
196       --bigfile|-b
197           The resource is a "big file" and the locations of the information
198           key should be printed.
199

LOCATE ARGUMENTS

201       key Specifies the key of the file to locate
202

RETURN VALUES

204       0   Success during operation.
205
206       1   During the locate, list, or listkey operation, the key was not
207           found.
208
209       2   Some fatal error occurred.
210

USAGE EXAMPLES

212       Please note that all examples assume you have a default config file
213       that contains the tracker and domain to use.  Saves us from having to
214       clutter up the command line.
215
216   Small Files (<64MB)
217       When it comes to using small files, mogtool is very, very easy.
218
219       Injection
220
221           $ mogtool inject foo.dbm foo.dbm.2004.12
222
223       Injects the file foo.dbm into MogileFS under the key of
224       foo.dbm.2004.12.
225
226           $ mogtool inject --gzip foo.dbm foo.dbm.2004.12
227
228       Injects the same file to the same key, but compresses it on the fly for
229       you.
230
231       Extraction
232
233           $ mogtool extract foo.dbm.2004.12 newfoo.dbm
234
235       Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.
236
237           $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
238
239       Gets the file and automatically decompresses it, if and only if it was
240       compressed.  So basically, you can turn on gzip in your config file and
241       mogtool will do the smart thing each time.
242
243           $ mogtool extract foo.dbm.2004.12 -
244
245       Print the resource to standard out.  If you want, you can pipe it
246       somewhere or redirect to a file (but why not just specify the
247       filename?).
248
249   Large Files (>64MB)
250       Given mogtool's ability to break files into chunks and later reassemble
251       them, inserting large files (even files over the 4GB barrier) is
252       relatively easy.
253
254       Injection
255
256           $ mogtool inject --bigfile largefile.dat largefile.dat
257
258       As expected, inserts the file largefile.dat into the MogileFS system
259       under the name largefile.dat.  Not very creative.  Uses the default
260       64MB chunks.
261
262           $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
263
264       Specify to use 16MB chunks instead of the default.  Otherwise, the
265       same.
266
267           $ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
268
269       Do it again, but specify 1000KB chunks, gzip automatically, and upload
270       it under a different key somekey.
271
272           $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
273
274       Same as above, but use 5 children processes for uploading chunks to
275       MogileFS.  This can take up to 300MB of memory in this example!  (It
276       tends to use about (concurrency + 1) * chunksize bytes.)
277
278           $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
279                     --receipt="foo@bar.com" --verify --description="A large file" \
280                     largefile.dat somekey
281
282       Break this file into 128MB chunks, set a description, use 3 children to
283       upload them, gzip the file as you go, do a full MD5 verification of
284       every chunk, then email a receipt with all of the MogileFS paths to me.
285
286       Lots of flexibility with mogtool.
287
288       Extraction
289
290           $ mogtool extract --bigfile somekey newfile.dat
291
292       In its basic form, extracts the previously inserted large file and
293       saves it as newfile.dat.
294
295           $ mogtool extract --bigfile --gzip somekey newfile.dat
296
297       If the file was gzipped on entry, ungzip it and save the result.  If it
298       wasn't gzipped, then we just save it.
299
300   Directories
301       Directories are easily injected and extracted with mogtool.  To create
302       the data stream that is inserted into MogileFS, we use tar.
303
304       Injection
305
306           $ mogtool inject --bigfile mydir mykey
307
308       Run mydir through tar and then save it as mykey.
309
310           $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
311
312       Inject, but also gzip and use multiple injectors.
313
314       Note how this is just like injecting a large file.  See injection
315       examples for large files for more examples.
316
317       Extraction
318
319           $ mogtool extract --bigfile mykey .
320
321       Extract the previously injected directory mykey to your local
322       directory.
323
324           $ mogtool extract --bigfile --asfile mykey foo.tar
325
326       Take the previously generated tarball and save it as foo.tar.  Simply
327       creates the file instead of extracting everything inside.
328
329   Partitions/Devices
330       mogtool has the ability to inject raw partitions into MogileFS and to
331       retrieve them later and write them back to a partition.  They're
332       treated just like directories for the most part, we just don't pipe
333       things through tar.
334
335       Injection
336
337           $ mogtool inject --bigfile /dev/hda3 hda3.backup
338
339       Save a raw copy of your partition /dev/hda3 to the key hda3.backup.
340
341           $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
342
343       Same, but compress on the fly during injection.
344
345       Extraction
346
347           $ mogtool extract --bigfile hda3.backup /dev/hda4
348
349       Extract the partition at hda3.backup to the partition /dev/hda4.
350       WARNING: mogtool won't ask for confirmation, make sure you don't
351       mistype partition numbers!
352
353   Deleting a Resource
354       WARNING: Please make sure you're specifying the right parameter, as
355       delete does not prompt for confirmation of the request!
356
357           $ mogtool delete thekey
358
359       Delete a normal file.
360
361           $ mogtool delete --bigfile thekey
362
363       Delete a chunked file--this deletes all chunks and the receipt, so the
364       file is gone.
365
366   Listing Big Files
367           $ mogtool list backup
368
369       Lists all large files stored in MogileFS.  It is not possible to list
370       all normal files at this time.
371
372   Listing Files Matching a Key
373           $ mogtool listkey abc1
374
375       Lists all files in MogileFS whose keys start with the characters
376       "abc1".
377

CONFIGURATION FILE

379       Instead of adding a ton of options to the command line every time,
380       mogtool enables you to create a default configuration file that it will
381       read all of the options from.  It searches two locations for a default
382       configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.
383       (Alternately, you can specify --conf=whatever as an option on the
384       command line.)
385
386       The file can consist of any number of the following items:
387
388           trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
389           domain = mogiledomain
390           class = fileclass
391           lib = /home/foo/lib
392           gzip = 1
393           big = 1
394           overwrite = 1
395           chunksize = 32M
396           receipt = foo@bar.com, baz@bar.com
397           verify = 1
398           concurrent = 3
399

KNOWN BUGS

401       None?  Send me any you find!  :)
402

PLANNED FEATURES

404       --concurrent for extract
405           It would be nice to have concurrent extraction going on.
406
407       recover mode
408           If the receipt file is ever corrupt in MogileFS it would be useful
409           to recover a file given just a receipt.  It would have the same
410           arguments as the extract mode, except use a receipt file as the
411           data source.
412
413       partition size verification
414           We can easily get the partition size when we save one to MogileFS,
415           so we should use that information to determine during extraction if
416           a target partition is going to be big enough.
417
418       on the fly gzip extraction
419           Right now we can gzip on an injection, but we should support doing
420           decompression on the fly coming out of MogileFS.
421
422       make list take a prefix
423           If you can specify a prefix, that makes things easier for finding
424           small files that are stored in MogileFS.
425
426       more information on list
427           Have list load up the info file and parse it for information about
428           each of the big files being stored.  Maybe have this as an option
429           (-l).  (This means the reading and parsing of info files should be
430           abstracted into a function.)
431

AUTHOR

433       Mark Smith <junior@danga.com> - most of the implementation and
434       maintenance.
435
436       Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.
437
438       Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.
439
440       Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
441

POD ERRORS

443       Hey! The above document had some coding errors, which are explained
444       below:
445
446       Around line 289:
447           Expected text after =item, not a number
448
449       Around line 293:
450           Expected text after =item, not a number
451
452
453
454perl v5.34.0                      2022-01-21                        MOGTOOL(1)
Impressum