1RASQLINSERT(1) General Commands Manual RASQLINSERT(1)
2
6 rasqlinsert - write argus(8) data into mysql database tables.
7
9 rasqlinsert [raoptions] [-- filter-expression]
10
13 Rasqlinsert writes argus data into a mysql database.
14 The principal function of rasqlinsert is to insert and update flow
16 data attributes, into a MySQL database table. Using the same syntax and
17 strategies for all other ra* programs, rasqlinsert creates databases
18 and database tables, based on the print specification on the either the
19 command-line or the .rarc file.
20 The concept is that where a ra* program would print fields to standard
22 out in ascii, rasqlinsert will insert those fields into the database as
23 attributes. The flow key, as defined by the "-m fields" option, pro‐
24 vides the definition of any keys that would be used in the schema. A
25 "-m none" option, will remove the use of any DBMS keys for inserted
26 data, and is the method to use when inserting streaming, unprocessed,
27 primitive argus data into a database table.
28 The schema is important for database utility and performance. You can
30 use MySQL querys against the attributes that you insert into the
31 tables, such searching and sorting on IP addresses, time, packet
32 counts, etc.... While rasqlinsert does not limit you to the number of
33 attributes (columns) per record you provide, the RDBMS performance will
34 quide you as to how many fields are useful.
35 Rasqlinsert by default, includes the actual binary argus 'record' in
37 the schema, and inserts and updates the binary record when needed.
38 This enables a large number of fucnctions that extend beyond simple
39 RDBMS schema's that are useful. Adding the 'record' is expensive, and
40 some will elect to not use this feature. This can be controlled using
41 the option '-s -record' as a print field option in the standard ra.1
42 command line. When the 'record' attribute is present, rasql.1 can read
43 the records directly from the database, to provide additional process‐
44 ing on the database table contents.
45 When keys are used, the database will enforce that any insertions meet
47 the relaitional requirements, i.e. that the keys be unique. This
48 requirement demands a sense of caching and key tracking, which rasqlin‐
49 sert is specifically designed to provide.
50 Rasqlinsert by default, will append data to existing tables, without
52 checking the schema for consistency. If your schema has keys, and you
53 attempt to append new records to an existing table, there is a high
54 likelyhood for error, as rasqlinsert will attempt to insert a record
55 that collides with an existing flow key. Use the "-M cache" option to
56 cause rasqlinsert to reference the table contents prior to aggregation
57 and insertion.
58 The binary data rasqlinsert inserts by default, is read using rasql.
60
63 Rasqlinsert, like all ra based clients, supports a number of ra options
64 including filtering of input argus records through a terminating filter
65 expression. Rasqlinsert(1) specific options are:
66 -M cache
68 This causes rasqlinsert to use the database table as its persistent
69 cache store. This mechanism is used to control memory use when
70 dealing with large amounts of data and flow keys.
71 -M drop
73 This causes rasqlinsert to drop any pre-existing database table
74 that has the same name as the target table name, on startup.
75
78 This invocation writes aggregated argus(8) data from the file into a
79 database table. The standard 5-tuple fields, 'saddr daddr proto sport
80 dport' are used as keys for each entry. rasqlinsert will aggregate all
81 the data prior to inserting the data into the database:
82 rasqlinsert -r file -w mysql://user@localhost/db/table
84 Because aggregation can require a lot of memory, rasqlinsert provides
86 an option '-M cache' to have rasqlinsert use the database table as the
87 persistent cache store for the aggregation. With this example, the
88 standard 5-tuple fields, rasqlinsert will aggregate data over short
89 spans of time as it reads the data from the file, and then commit the
90 data to the database. If additional data arrives that matches that
91 unique flow, rasqlinsert will fetch the entry from the database, aggre‐
92 gate, and then update the data entry in the database.
93 rasqlinsert -M cache -r file -w mysql://user@localhost/db/table
95 rasqlinsert can provide the same function for streaming data read
97 directly from an argus data source. This allows rasqlinsert to
98 reassemble all status records for an individual flow, such that the
99 resulting table has only a single entry for each communciation rela‐
100 tionship seen.
101 rasqlinsert -M cache -S argus -w mysql://user@localhost/db/table
103 This invocation writes argus(8) data from the file into a database ta‐
105 ble, without aggregation, by specifying no relational key in the data.
106 rasqlinsert -m none -r file -w mysql://user@localhost/db/table
108 This invocation writes argus(8) data from the stream into a database
110 table, without modification.
111 rasqlinsert -m none -S argus -w mysql://user@localhost/db/table
113 This invocation writes argus(8) data from the stream into a daily data‐
115 base table, without modification. rasqlinsert will generate table names
116 based on time and insert its data relative to the timestamps found in
117 the flow records it processes. In this specific example, "-M time 1d"
118 specifies daily tables.
119 rasqlinsert -m none -S argus -w mysql://user@localhost/db/table_%Y_%m_%d -M time 1d
121
124 Copyright (c) 2000-2016 QoSient. All rights reserved.
125
128 rasql(1), ra(1), rarc(5), argus(8),
129
132 Carter Bullard (carter@qosient.com).
133rasqlinsert 3.0.8 12 August 2009 RASQLINSERT(1)