ids2ngram(1)

1IDS2NGRAM(1)          User Contributed Perl Documentation         IDS2NGRAM(1)
2
3
4

NAME

6       ids2ngram - generate n-gram data file from ids file
7

SYNOPSIS

9       ids2ngram [option]... ids_file...
10

DESCRIPTION

12       ids2ngram generates idngram file, which is a sorted [id1,..,idN,freq]
13       array, from binary id stream files. Here, the id stream files are
14       always generated by mmseg or slmseg. Basically, it finds all occurrence
15       of n-words tuples (i.e. the tuple of (id1,..,idN)), and sorts these
16       tuples by the lexicographic order of the ids make up the tuples, then
17       write them to specified output file.
18

INPUT

20       The input file is presented as a binary id stream, which looks like:
21           [id0,...,idX]
22

OPTIONS

24       All the following options are mandatory.
25
26       -n,--NMax N
27           Generates N-gram result. ids2ngram does only support uni-gram, bi-
28           gram, and trigram, so any number not in the range of 1..3 is not
29           valid.
30
31       -s,--swap swap-file
32           Specify the temporary intermediate file.
33
34       -o, --out output-file
35           Specify the result idngram file, e.g. the array of [id1, ..., idN,
36           freq]
37
38       -p, --para N
39           Specify the maximum n-gram items per paragraph. ids2ngram writes to
40           the temporary file on a per-paragraph basis. Every time it writes a
41           paragraph out, it frees the corresponding memory allocated for it.
42           When your computer system permits, a higher N is suggested. This
43           can speed up the processing speed because of less I/O.
44

EXAMPLE

46       Following example will use three input idstream file idsfile[1,2,3] to
47       generate the idngram file all.id3gram. Each para (internal map size or
48       hash size) would be 1024000, using swap file for temp result. All temp
49       para result would eventually be merged to got the final result.
50
51       ids2ngram -n 3 -s /tmp/swap -o all.id3gram -p 1024000 idsfile1 idsfile2
52       idsfile3
53

AUTHOR

55       Originally written by Phill.Zhang <phill.zhang@sun.com>.  Currently
56       maintained by Kov.Chai <tchaikov@gmail.com>.
57