1MOGTOOL(1) User Contributed Perl Documentation MOGTOOL(1)
2
6 mogtool -- Inject/extract data to/from a MogileFS installation
7
9 $ mogtool [general-opts] <command> [command-opts] <command-args>
10 $ mogtool --trackers=127.0.0.1:6001 --domain=foo --class=bar ...
12 $ mogtool --conf=foo.conf ...
13 $ mogtool inject thefile.tgz thefilekey
15 $ mogtool inject --bigfile thebigfile.tgz thefilekey
16 $ mogtool inject --bigfile --gzip thebigfile.tar thefilekey
17 $ mogtool inject --bigfile --gzip mydirectory thedirkey
18 $ mogtool inject --bigfile --gzip /dev/hda4 thedevkey
19 $ mogtool inject --nobigfile bigcontiguousfile bigcfilekey
20 $ mogtool inject --bigfile --gzip --verify \
22 --description="Description" \
23 --receipt="foo@bar.com, baz@bar.com" \
24 --concurrent=5 --chunksize=32M \
25 somehugefile thefilekey
26 $ mogtool extract thefilekey thenewfile.tgz
28 $ mogtool extract thefilekey -
29 $ mogtool extract --bigfile thedirkey .
30 $ mogtool extract --bigfile --asfile thedirkey thefile.tgz
31 $ mogtool extract --bigfile thedevkey /dev/hda4
32 $ mogtool delete thekey
34 $ mogtool locate --noverify thekey
36 $ mogtool locate --bigfile thekey
37 $ mogtool list
39 $ mogtool listkey key
40
42 --debug
43 Turn on MogileFS debug output.
44 --trackers=<[preferred_ip/]ip:port>[,<[preferred_ip/]ip:port>]*
46 Specify one or more trackers for your MogileFS installation. Note
47 that you can specify preferred IPs to override the default IPs
48 with. So it would look something like 10.10.0.1/10.0.0.1:8081.
49 --domain=<domain>
51 Set the MogileFS domain to use.
52 --class=<class>
54 Set the class within the domain to use. Defaults to _default.
55 --conf=<file>
57 Specify a configuration file to load from.
58 --lib=<directory>
60 Specify a directory to use as a library path. Right now, this
61 should be the directory where you expect to find the MogileFS.pm
62 file, if it's not actually installed.
63
65 inject|i
66 Insert a resource into MogileFS. See "INJECT OPTIONS" and "INJECT
67 ARGUMENTS" for the rest of how to use the inject mode.
68 extract|x
70 Extract a resource from MogileFS. See "EXTRACT OPTIONS" and
71 "EXTRACT ARGUMENTS" for how to use extract.
72 delete|rm
74 Delete a resource. See "DELETE OPTIONS" and "DELETE ARGUMENTS".
75 locate|lo key
77 List the paths to the file identified by the given key.
78 list|ls
80 List all big files contained in MogileFS. No options, no
81 arguments.
82 listkey|lsk key
84 List all files which match the key. Key is just a prefix, and this
85 will list all keys which match the prefix. So if you specify key as
86 "ABC1" then you'll get all keys which start with the characters
87 "ABC1"
88
90 The following options are used to control the behavior of the injector.
91 --bigfile|-b
93 If specified, use chunking to break the resource into manageable
94 pieces.
95 --chunksize=<size>[B|K|M|G]
97 When instructed to break files into chunks, the injector will use
98 the specified chunk size as the maximum chunk size. Defaults to
99 64M. You can specify the chunk size manually and specify the
100 units--defaults to bytes.
101 --gzip|-z
103 If specified, mogtool will gzip the data as it's going into
104 MogileFS. The resource will be marked as compressed.
105 Note that you do not need to specify this if the resource is
107 already gzipped, but it doesn't hurt. (We automatically detect
108 that and mark it as compressed.)
109 --overwrite
111 If you previously were working on injecting a big file as chunks
112 and the process died, normally mogtool refuses to do it again.
113 Specify this option to force the overwrite of that file.
114 NOTE: Other than in the above case (partial failure), mogtool will
116 not prompt before overwriting an existing file.
117 --verify
119 If on, we do a full MD5 verification of every chunk after it is
120 replicated. This can take a while on large files!
121 --description=<text>
123 Specifies a description for this file. Optional, but assists in
124 reporting and listing the large files in MogileFS. (This is also
125 displayed in any receipts that are created.)
126 --receipt=<email address>[, <email address>]*
128 If specified, emails a copy of the receipt file to the specified
129 comma-separated email addresses. Also creates a local filesystem
130 copy of the receipt file.
131 --concurrent=<number>
133 Specifies the number of concurrent processes to run for MogileFS
134 insertion. If you are noticing mogtool spend most of it's time
135 waiting for children and not actually buffering data, you may wish
136 to raise this number. The default is 1 but we've found 3 or 4 work
137 well.
138
140 resource
141 What you actually want to inject. This can be a file, directory,
142 or a raw partition in the format /dev/X.
143 Please see "USAGE EXAMPLES" for more information on how to inject
145 these different types of resources and the differences thereof.
146 key Specifies the key to save this file to. For big files, the key is
148 actually "_big_N:key" and "key,#" where N is one of a bunch of
149 things we use and # is the chunk number.
150 Generally, you want this to be descriptive so you remember what it
152 is later and can identify the file just by looking at the key.
153
155 --bigfile|-b
156 If specified, indicates that this resource was chunked on injection
157 and should be reassembled for extraction.
158 --gzip|-z
160 Specifies to mogtool that it should ungzip the output if and only
161 if it was compressed when inserted into the MogileFS system. So,
162 if you're extracting a file that wasn't gzipped to begin with, this
163 doesn't do anything.
164 --asfile
166 Useful when extracting something previously inserted as a
167 directory--this option instructs mogtool to treat the resource as a
168 file and not actually run it through tar for decompression.
169
171 key Specifies the key to get the file from.
172 destination
174 What destination means varies depending on what type of resource
175 you're extracting. However, no matter what, you can specify a
176 single dash (-) to mean STDOUT.
177 Please see the usage examples for more information on how extract
179 works.
180
182 --bigfile|-b
183 The resource is a "big file" and all chunks should be deleted.
184
186 key Specifies the key of the file to delete.
187
189 --verify
190 Verify that the returned paths actually contain the file. The
191 locate commands defaults to verify, you can disable it with
192 --noverify.
193 --bigfile|-b
195 The resource is a "big file" and the locations of the information
196 key should be printed.
197
199 key Specifies the key of the file to locate
200
202 0 Success during operation.
203 1 During the locate, list, or listkey operation, the key was not
205 found.
206 2 Some fatal error occurred.
208
210 Please note that all examples assume you have a default config file
211 that contains the tracker and domain to use. Saves us from having to
212 clutter up the command line.
213 Small Files (<64MB)
215 When it comes to using small files, mogtool is very, very easy.
216 Injection
218 $ mogtool inject foo.dbm foo.dbm.2004.12
220 Injects the file foo.dbm into MogileFS under the key of
222 foo.dbm.2004.12.
223 $ mogtool inject --gzip foo.dbm foo.dbm.2004.12
225 Injects the same file to the same key, but compresses it on the fly for
227 you.
228 Extraction
230 $ mogtool extract foo.dbm.2004.12 newfoo.dbm
232 Retrieves the key foo.dbm.2004.12 and saves it as newfoo.dbm.
234 $ mogtool extract --gzip foo.dbm.2004.12 newfoo.dbm
236 Gets the file and automatically decompresses it, if and only if it was
238 compressed. So basically, you can turn on gzip in your config file and
239 mogtool will do the smart thing each time.
240 $ mogtool extract foo.dbm.2004.12 -
242 Print the resource to standard out. If you want, you can pipe it
244 somewhere or redirect to a file (but why not just specify the
245 filename?).
246 Large Files (>64MB)
248 Given mogtool's ability to break files into chunks and later reassemble
249 them, inserting large files (even files over the 4GB barrier) is
250 relatively easy.
251 Injection
253 $ mogtool inject --bigfile largefile.dat largefile.dat
255 As expected, inserts the file largefile.dat into the MogileFS system
257 under the name largefile.dat. Not very creative. Uses the default
258 64MB chunks.
259 $ mogtool inject --bigfile --chunksize=16M largefile.dat largefile.dat
261 Specify to use 16MB chunks instead of the default. Otherwise, the
263 same.
264 $ mogtool inject --bigfile --chunksize=1000K --gzip largefile.dat somekey
266 Do it again, but specify 1000KB chunks, gzip automatically, and upload
268 it under a different key somekey.
269 $ mogtool inject --bigfile --concurrent=5 --gzip largefile.dat somekey
271 Same as above, but use 5 children processes for uploading chunks to
273 MogileFS. This can take up to 300MB of memory in this example! (It
274 tends to use about (concurrency + 1) * chunksize bytes.)
275 $ mogtool inject --bigfile --chunksize=32M --concurrent=3 --gzip \
277 --receipt="foo@bar.com" --verify --description="A large file" \
278 largefile.dat somekey
279 Break this file into 128MB chunks, set a description, use 3 children to
281 upload them, gzip the file as you go, do a full MD5 verification of
282 every chunk, then email a receipt with all of the MogileFS paths to me.
283 Lots of flexibility with mogtool.
285 Extraction
287 $ mogtool extract --bigfile somekey newfile.dat
289 In its basic form, extracts the previously inserted large file and
291 saves it as newfile.dat.
292 $ mogtool extract --bigfile --gzip somekey newfile.dat
294 If the file was gzipped on entry, ungzip it and save the result. If it
296 wasn't gzipped, then we just save it.
297 Directories
299 Directories are easily injected and extracted with mogtool. To create
300 the data stream that is inserted into MogileFS, we use tar.
301 Injection
303 $ mogtool inject --bigfile mydir mykey
305 Run mydir through tar and then save it as mykey.
307 $ mogtool inject --bigfile --gzip --concurrent=5 mydir mykey
309 Inject, but also gzip and use multiple injectors.
311 Note how this is just like injecting a large file. See injection
313 examples for large files for more examples.
314 Extraction
316 $ mogtool extract --bigfile mykey .
318 Extract the previously injected directory mykey to your local
320 directory.
321 $ mogtool extract --bigfile --asfile mykey foo.tar
323 Take the previously generated tarball and save it as foo.tar. Simply
325 creates the file instead of extracting everything inside.
326 Partitions/Devices
328 mogtool has the ability to inject raw partitions into MogileFS and to
329 retrieve them later and write them back to a partition. They're
330 treated just like directories for the most part, we just don't pipe
331 things through tar.
332 Injection
334 $ mogtool inject --bigfile /dev/hda3 hda3.backup
336 Save a raw copy of your partition /dev/hda3 to the key hda3.backup.
338 $ mogtool inject --bigfile --gzip /dev/hda3 hda3.backup
340 Same, but compress on the fly during injection.
342 Extraction
344 $ mogtool extract --bigfile hda3.backup /dev/hda4
346 Extract the partition at hda3.backup to the partition /dev/hda4.
348 WARNING: mogtool won't ask for confirmation, make sure you don't
349 mistype partition numbers!
350 Deleting a Resource
352 WARNING: Please make sure you're specifying the right parameter, as
353 delete does not prompt for confirmation of the request!
354 $ mogtool delete thekey
356 Delete a normal file.
358 $ mogtool delete --bigfile thekey
360 Delete a chunked file--this deletes all chunks and the receipt, so the
362 file is gone.
363 Listing Big Files
365 $ mogtool list backup
366 Lists all large files stored in MogileFS. It is not possible to list
368 all normal files at this time.
369 Listing Files Matching a Key
371 $ mogtool listkey abc1
372 Lists all files in MogileFS whose keys start with the characters
374 "abc1".
375
377 Instead of adding a ton of options to the command line every time,
378 mogtool enables you to create a default configuration file that it will
379 read all of the options from. It searches two locations for a default
380 configuration file: ~/.mogtool and /etc/mogilefs/mogtool.conf.
381 (Alternately, you can specify --conf=whatever as an option on the
382 command line.)
383 The file can consist of any number of the following items:
385 trackers = 10.0.0.3:7001, 10.10.0.5/10.0.0.5:7001
387 domain = mogiledomain
388 class = fileclass
389 lib = /home/foo/lib
390 gzip = 1
391 big = 1
392 overwrite = 1
393 chunksize = 32M
394 receipt = foo@bar.com, baz@bar.com
395 verify = 1
396 concurrent = 3
397
399 None? Send me any you find! :)
400
402 --concurrent for extract
403 It would be nice to have concurrent extraction going on.
404 recover mode
406 If the receipt file is ever corrupt in MogileFS it would be useful
407 to recover a file given just a receipt. It would have the same
408 arguments as the extract mode, except use a receipt file as the
409 data source.
410 partition size verification
412 We can easily get the partition size when we save one to MogileFS,
413 so we should use that information to determine during extraction if
414 a target partition is going to be big enough.
415 on the fly gzip extraction
417 Right now we can gzip on an injection, but we should support doing
418 decompression on the fly coming out of MogileFS.
419 make list take a prefix
421 If you can specify a prefix, that makes things easier for finding
422 small files that are stored in MogileFS.
423 more information on list
425 Have list load up the info file and parse it for information about
426 each of the big files being stored. Maybe have this as an option
427 (-l). (This means the reading and parsing of info files should be
428 abstracted into a function.)
429
431 Mark Smith <junior@danga.com> - most of the implementation and
432 maintenance.
433 Brad Fitzpatrick <brad@danga.com> - concepts and rough draft.
435 Robin H. Johnson <robbat2@orbis-terrarum.net> - locate function.
437 Copyright (c) 2002-2004 Danga Interactive. All rights reserved.
439perl v5.12.1 2010-04-02 MOGTOOL(1)