1RASTREAM(1) General Commands Manual RASTREAM(1)
2
6 rastream - stream block processor for argus(8) data.
7
9 rastream [[-M splitmode] [splitmode options]] [-f file processing pro‐
10 gram -B secs] [raoptions] [-- filter-expression]
11
13 Rastream reads argus data from an argus-data source, and splits the
14 resulting output into consecutive sections of records based on size,
15 count time, or flow event, writing the output into a set of output-
16 files.
17 Rastream provides the option to run a program against the output files,
19 seconds after the file is understood to be finished. The program must
20 be specified in a manner so that rastream can find it, either within
21 the system $PATH, or provided as a full pathname.
22 By default, rastream splits the stream by output file record count,
24 putting 10,000 records of input into each argus output file, or stan‐
25 dard out, as needed. The behavior is similar to the unix split.1 com‐
26 mand.
27 The output files' name consists of a prefix, which is specified using
29 the -w ra option, and a suffix, which is created for each resulting
30 file. If no prefix is provided, then rastream will use 'x' as the
31 default prefix. The suffix that is used is determined by the mode of
32 operation. When rastream is using the default count mode or the size
33 mode, the suffix is a group of letters 'aa', ´ab´, and so on, such that
34 concatenating the output files in sorted order by file name produces
35 the original input file. If rastream will need to create more output
36 files than are allowed by the default suffix strategy, more letters
37 will be added, in order to accomodate the needed files. When the mode
38 is time mode, the default output filename suffix is
39 '%Y.%m.%d.%h.%m.%s', which is used by strftime() to create an output
40 filename that is time oriented. This default is overrided by adding a
41 '%' extension to the name provided on the commandline using the -w
42 option.
43 When standard out is specified, using -w -, rastream will output a sin‐
45 gle argus-stream with START and STOP argus management records inserted
46 appropriately to indicate where the output is split. See argus(8) for
47 more information on output stream formats.
48 When rastream is spliting on output record count (the default), the
50 number of records is specified as an ordinal counter, the default is
51 10,000 records. When rastream is spliting based on the maximum output
52 file size, the size is specified as bytes. The scale of the bytes can
53 be specified by appending 'b', 'k' and 'm' to the number provided.
54 When rastream is spliting based on time, the time period is specified
56 with the option, and can be any period based in seconds (s), minutes
57 (m), hours (h), days (d), weeks (w), months (M) or years (y). Rastream
58 will create and modify records as required to split on prescribed time
59 boundaries. If any record spans a time boundary, the record is split
60 and the metrics are adjusted using a uniform distribution model to dis‐
61 tribute the statistics between the two records. Care is taken to avoid
62 records with zero packet and byte counts, that could result from round‐
63 off error.
64 When rastream is spliting based on flow event, the flow that acts as
66 the event marker is specified using a standard ra filter expression,
67 that is bounded by quotes ("). Records that preceed the first flow
68 event in the data stream are written to the specified output file, and
69 then new files are generated with the flow event record being the first
70 record of the new file. This method will allow you to use wire events
71 as triggers for spliting data.
72
75 Rastream, like all ra based clients, supports a number of ra options
76 including remote data access, reading from multiple files and filtering
77 of input argus records through a terminating filter expression. ras‐
78 tream(1) specific options are:
79 -a suffix length
81 Starting append suffix length. The default is 2 characters.
82 -B duration
84 Buffer hold time before processing. This value is usually in the
85 5-15 second range and provides time for rastream to sort records
86 and schedule outputfile processing. The number is derived from the
87 larges FAR status interval of all the argus data sources encoun‐
88 tered.
89 -f program
92 Post processing program. rastream, will execute this program just
93 after closing the output file, passing the full path to the closed
94 output file as a parameter, using this convention:
95 program -r /full/path/to/closed/file
97 This allows you to post-process the output file in an automated
99 fashion.
100 Generally, this program can do anything you like, such as aggregat‐
102 ing and correcting flow records, labeling records for semantic
103 enhancement, indexing the files, using programs like rasqltimein‐
104 dex(), and compressing the files. Traditionally, the program has
105 been a shell-script, perl program, or php script, so that it can be
106 easily modified, on the fly, but it can be any executable that can
107 handle the "-r filename" parameter convention. The program should
108 provides its own accountability and error logging, so that you know
109 that things are working as you expect.
110 rastream must have a path to the program, the program must be exe‐
112 cutable, and rastream must have permission to run the program for
113 this strategy to be successful.
114 An example rastream.sh is provided in the ./support/Config direc‐
116 tory.
117 -M splitmode
120 Supported spliting modes are:
121 count <num>
122 size <size>
123 time <period>
124 flow "filter-expression"
125 -w filename
127 Rastream supports an extended -w option that allows for output
128 record contents to be inserted into the output filename. Specified
129 using '$' (dollar) notation, any printable field can be used. Care
130 should be taken to honor any shell escape requirements when speci‐
131 fying on the command line. See ra(1) for the list of printable
132 fields.
133 Another extended feature, when using time mode, rastream will
135 process the supplied filename using strftime(3), so that time
136 fields can be inserted into the resulting output filename.
137
140 This invocation reads argus(8) data from inputfile and splits the
141 argus(8) data stream based on output file size of no greater than 1
142 Megabyte. The resulting output files have a prefix of argus. and suf‐
143 fix that starts with 'aa'. The single trailing '.' is significant.
144 rastream -r inputfile -M size 1m -w argus.
146 This invocation splits inputfile based on hard 10 minute time bound‐
149 aries. The resulting output files are created with a prefix of /ar‐
150 chive/%Y/%m/%d/argus. and the suffix is %H.%M.%S. The values will be
151 supplied based on the time in the record being written out.
152 rastream -r * -M time 10m -w "/archive/%Y/%m/%d/argus.%H.%M.%S"
154 This invocation splits inputfile based on the argus source identifier.
157 The resulting output files are created with a prefix of /archive/Source
158 Identifier/argus. and the default suffix starting with "aa". The
159 source identifier will be supplied based on the contents of the record
160 being exported.
161 rastream -r * -M time 10m -w "/archive/$srcid/argus."
163 This invocation splits inputfile based on a flow event marker. The
165 resulting output files are created with a prefix of 'outfile.' and the
166 default suffix starting with "aa". Whenever a ping to a specific host
167 is seen in the stream, a new output file is generated.
168 rastream -r * -M flow "echo and host 1.2.3.4" -w outfile.
170
173 Copyright (c) 2000-2016 QoSient. All rights reserved.
174
177 ra(1), rarc(5), argus(8),
178
181 Carter Bullard (carter@qosient.com).
182rastream 3.0.8 12 August 2003 RASTREAM(1)