1funfilters(n)                 SAORD Documentation                funfilters(n)
2
3
4

NAME

6       Funfilters: Filtering Rows in a Table
7

SYNOPSIS

9       This document contains a summary of the user interface for filtering
10       rows in binary tables.
11

DESCRIPTION

13       Table filtering allows a program to select rows from an table (e.g.,
14       X-ray event list) by checking each row against one or more expressions
15       involving the columns in the table. When a table is filtered, only
16       valid rows satisfying these expressions are passed through for process‐
17       ing.
18
19       A filter expression is specified using bracket notation appended to the
20       filename of the data being processed:
21
22         foo.fits[pha==1&&pi==2]
23
24       It is also possible to put region specification inside a file and then
25       pass the filename in bracket notation:
26
27         foo.fits[@my.reg]
28
29       Filters must be placed after the extension and image section informa‐
30       tion, when such information is present. The correct order is:
31
32       ·   file[fileinfo,sectioninfo][filters]
33
34       ·   file[fileinfo,sectioninfo,filters]
35
36       where:
37
38       ·   file is the Funtools file name
39
40       ·   fileinfo is an ARRAY, EVENT, FITS extension, or FITS index
41
42       ·   sectioninfo is the image section to extract
43
44       ·   filters are spatial region and table (row) filters to apply
45
46       See Funtools Files for more information on file and image section spec‐
47       ifications.
48
49       Filter Expressions
50
51       Table filtering can be performed on columns of data in a FITS binary
52       table or a raw event file.  Table filtering is accomplished by means of
53       table filter specifications.  An table filter specification consists of
54       one or more filter expressions Filter specifications also can contain
55       comments and local/global processing directives.
56
57       More specifically, a filter specification consist of one or more lines
58       containing:
59
60         # comment until end of line
61         # include the following file in the table descriptor
62         @file
63         # each row expression can contain filters separated by operators
64         [filter_expression] BOOLOP [filter_expression2], ...
65         # each row expression can contain filters separated by the comma operator
66         [filter_expression1], [filter_expression2], ...
67         # the special row# keyword allows a range of rows to be processed
68         row#=m:n
69         # or a single row
70         row#=m
71         # regions are supported -- but are described elsewhere
72         [spatial_region_expression]
73
74       A single filter expression consists of an arithmetic, logical, or other
75       operations involving one or more column values from a table. Columns
76       can be compared to other columns, to header values, or to numeric con‐
77       stants. Standard math functions can be applied to columns. Separate
78       filter expressions can be combined using boolean operators.  Standard C
79       semantics can be used when constructing expressions, with the usual
80       precedence and associativity rules holding sway:
81
82         Operator                                Associativity
83         --------                                -------------
84         ()                                      left to right
85         !! (logical not)                        right to left
86         !  (bitwise not) - (unary minus)        right to left
87         *  /                                    left to right
88         +  -                                    left to right
89         < <= > >=                               left to right
90         == !=                                   left to right
91         &  (bitwise and)                        left to right
92         ^  (bitwise exclusive or)               left to right
93         ⎪  (bitwise inclusive or)               left to right
94         && (logical and)                        left to right
95         ⎪⎪ (logical or)                         left to right
96         =                                       right to left
97
98       For example, if energy and pha are columns in a table, then the follow‐
99       ing are valid expressions:
100
101         pha>1
102         energy == pha
103         (pha>1) && (energy<=2)
104         max(pha,energy)>=2.5
105
106       Comparison values can be integers or floats. Integer comparison values
107       can be specified in decimal, octal (using '0' as prefix), hex (using
108       '0x' as prefix) or binary (using '0b' as prefix). Thus, the following
109       all specify the same comparison test of a status mask:
110
111         (status & 15) == 8           # decimal
112         (status & 017) == 010        # octal
113         (status & 0xf) == 0x8        # hex
114         (status & 0b1111) == 0b1000  # binary
115
116       The special keyword row# allows you to process a range of rows.  When
117       row# is specified, the filter code skips to the designated row  and
118       only processes the specified number of rows. The "*" character can be
119       utilized as the high limit value to denote processing of the remaining
120       rows. Thus:
121
122         row#=100:109
123
124       processes 10 rows, starting with row 100 (counting from 1), while:
125
126         row#=100:*
127
128       specifies that all but the first 99 rows are to be processed.
129
130       Spatial region filtering allows a program to select regions of an image
131       or rows of a table (e.g., X-ray events) using simple geometric shapes
132       and boolean combinations of shapes.  For a complete description of
133       regions, see Spatial Region Filtering.
134
135       Separators Also Are Operators
136
137       As mentioned previously, multiple filter expressions can be specified
138       in a filter descriptor, separated by commas or new-lines.  When such a
139       comma or new-line separator is used, the boolean AND operator is auto‐
140       matically generated in its place. Thus and expression such as:
141
142         pha==1,pi=2:4
143
144       is equivalent to:
145
146         (pha==1) && (pi>=2&&pi<=4)
147
148       [Note that the behavior of separators is different for filter expres‐
149       sions and spatial region expressions.  The former uses AND as the oper‐
150       ator, while the latter user OR. See Combining Region and Table Filters
151       for more information about these conventions and how they are treated
152       when combined.]
153
154       Range Lists
155
156       Aside from the standard C syntax, filter expressions can make use of
157       IRAF-style range lists which specify a range of values. The syntax
158       requires that the column name be followed by an '=' sign, which is fol‐
159       lowed by one or more comma-delimited range expressions of the form:
160
161         col = vv              # col == vv in range
162         col = :vv             # col <= vv in range
163         col = vv:             # col >= vv in range
164         col = vv1:vv2         # vv1 <= col <= vv2 in range
165
166       The vv's above must be numeric constants; the right hand side of a
167       range list cannot contain a column name or header value.
168
169       Note that, unlike an ordinary comma separator, the comma separator used
170       between two or more range expressions denotes OR.  Thus, when two or
171       more range expressions are combined with a comma separator, the result‐
172       ing expression is a shortcut for more complicated boolean logic. For
173       example:
174
175         col = :3,6:8,10:
176
177       is equivalent to:
178
179         (col=6 && col =10)
180
181       Note also that the single-valued rangelist:
182
183         col = val
184
185       is equivalent to the C-based filter expression:
186
187         col == val
188
189       assuming, of course, that val is a numeric constant.
190
191       Math Operations and Functions
192
193       It is permissible to specify C math functions as part of the filter
194       syntax.  When the filter parser recognizes a function call, it automat‐
195       ically includes the math.h and links in the C math library.  Thus, it
196       is possible to filter rows by expressions such as these:
197
198       ·   (pi+pha)>(2+log(pi)-pha)
199
200       ·   min(pi,pha)*14>x
201
202       ·   max(pi,pha)==(pi+1)
203
204       ·   feq(pi,pha)
205
206       ·   div(pi,pha)>0
207
208       The function feq(a,b) returns true (1) if the difference between a and
209       b (taken as double precision values) is less than approximately 10E-15.
210       The function div(a,b) divides a by b, but returns NaN (not a number) if
211       b is 0. It is a safe way to avoid floating point errors when dividing
212       one column by another.
213
214       Include Files
215
216       The special @filename directive specifies an include file containing
217       filter expressions. This file is processed as part of the overall fil‐
218       ter descriptor:
219
220         foo.fits[pha==1,@foo]
221
222       Header Parameters
223
224       The filter syntax supports comparison between a column value and a
225       header parameter value of a FITS binary tables (raw event files have no
226       such header).  The header parameters can be taken from the binary table
227       header or the primary header.  For example, assuming there is a header
228       value MEAN_PHA in one of these headers, you can select photons having
229       exactly this value using:
230
231       ·   pha==MEAN_PHA
232
233       Table filtering is more easily described by means of examples.  Con‐
234       sider data containing the following table structure:
235
236       ·   double TIME
237
238       ·   int X
239
240       ·   int Y
241
242       ·   short PI
243
244       ·   short PHA
245
246       ·   int DX
247
248       ·   int DY
249
250       Tables can be filtered on these columns using IRAF/QPOE range syntax or
251       any valid C syntax.  The following examples illustrate the possibili‐
252       ties:
253
254       ·   pha=10
255
256       ·   pha==10
257
258           select rows whose pha value is exactly 10
259
260       ·   pha=10:50
261
262           select rows whose pha value is in the range of 10 to 50
263
264       ·   pha=10:50,100
265
266           select rows whose pha value is in the range of 10 to 50 or is equal
267           to 100
268
269       ·   pha>=10 && pha<=50
270
271           select rows whose pha value is in the range of 10 to 50
272
273       ·   pi=1,2&&pha>3
274
275           select rows whose pha value is 1 or 2 and whose pi value is 3
276
277       ·   pi=1,2 ⎪⎪ pha>3
278
279           select rows whose pha value is 1 or 2 or whose pi value is 3
280
281       ·   pha==pi+1
282
283           select rows whose pha value is 1 less than the pi value
284
285       ·   (pha==pi+1) && (time>50000.0)
286
287           select rows whose pha value is 1 less than the pi value and whose
288           time value is greater than 50000
289
290       ·   (pi+pha)>20
291
292           select rows in which the sum of the pi and pha values is greater
293           than 20
294
295       ·   pi%2==1
296
297           select rows in which the pi value is odd
298
299       Currently, integer range list limits cannot be specified in binary
300       notation (use decimal, hex, or octal instead). Please contact us if
301       this is a problem.
302

SEE ALSO

304       See funtools(n) for a list of Funtools help pages
305
306
307
308version 1.4.0                   August 15, 2007                  funfilters(n)
Impressum