1funfilters(n) SAORD Documentation funfilters(n)
2
6 Funfilters: Filtering Rows in a Table
7
9 This document contains a summary of the user interface for filtering
10 rows in binary tables.
11
13 Table filtering allows a program to select rows from an table (e.g.,
14 X-ray event list) by checking each row against one or more expressions
15 involving the columns in the table. When a table is filtered, only
16 valid rows satisfying these expressions are passed through for process‐
17 ing.
18 A filter expression is specified using bracket notation appended to the
20 filename of the data being processed:
21 foo.fits[pha==1&&pi==2]
23 It is also possible to put region specification inside a file and then
25 pass the filename in bracket notation:
26 foo.fits[@my.reg]
28 Filters must be placed after the extension and image section informa‐
30 tion, when such information is present. The correct order is:
31 · file[fileinfo,sectioninfo][filters]
33 · file[fileinfo,sectioninfo,filters]
35 where:
37 · file is the Funtools file name
39 · fileinfo is an ARRAY, EVENT, FITS extension, or FITS index
41 · sectioninfo is the image section to extract
43 · filters are spatial region and table (row) filters to apply
45 See Funtools Files for more information on file and image section spec‐
47 ifications.
48 Filter Expressions
50 Table filtering can be performed on columns of data in a FITS binary
52 table or a raw event file. Table filtering is accomplished by means of
53 table filter specifications. An table filter specification consists of
54 one or more filter expressions Filter specifications also can contain
55 comments and local/global processing directives.
56 More specifically, a filter specification consist of one or more lines
58 containing:
59 # comment until end of line
61 # include the following file in the table descriptor
62 @file
63 # each row expression can contain filters separated by operators
64 [filter_expression] BOOLOP [filter_expression2], ...
65 # each row expression can contain filters separated by the comma operator
66 [filter_expression1], [filter_expression2], ...
67 # the special row# keyword allows a range of rows to be processed
68 row#=m:n
69 # or a single row
70 row#=m
71 # regions are supported -- but are described elsewhere
72 [spatial_region_expression]
73 A single filter expression consists of an arithmetic, logical, or other
75 operations involving one or more column values from a table. Columns
76 can be compared to other columns, to header values, or to numeric con‐
77 stants. Standard math functions can be applied to columns. Separate
78 filter expressions can be combined using boolean operators. Standard C
79 semantics can be used when constructing expressions, with the usual
80 precedence and associativity rules holding sway:
81 Operator Associativity
83 -------- -------------
84 () left to right
85 !! (logical not) right to left
86 ! (bitwise not) - (unary minus) right to left
87 * / left to right
88 + - left to right
89 < <= > >= left to right
90 == != left to right
91 & (bitwise and) left to right
92 ^ (bitwise exclusive or) left to right
93 ⎪ (bitwise inclusive or) left to right
94 && (logical and) left to right
95 ⎪⎪ (logical or) left to right
96 = right to left
97 For example, if energy and pha are columns in a table, then the follow‐
99 ing are valid expressions:
100 pha>1
102 energy == pha
103 (pha>1) && (energy<=2)
104 max(pha,energy)>=2.5
105 Comparison values can be integers or floats. Integer comparison values
107 can be specified in decimal, octal (using '0' as prefix), hex (using
108 '0x' as prefix) or binary (using '0b' as prefix). Thus, the following
109 all specify the same comparison test of a status mask:
110 (status & 15) == 8 # decimal
112 (status & 017) == 010 # octal
113 (status & 0xf) == 0x8 # hex
114 (status & 0b1111) == 0b1000 # binary
115 The special keyword row# allows you to process a range of rows. When
117 row# is specified, the filter code skips to the designated row and
118 only processes the specified number of rows. The "*" character can be
119 utilized as the high limit value to denote processing of the remaining
120 rows. Thus:
121 row#=100:109
123 processes 10 rows, starting with row 100 (counting from 1), while:
125 row#=100:*
127 specifies that all but the first 99 rows are to be processed.
129 Spatial region filtering allows a program to select regions of an image
131 or rows of a table (e.g., X-ray events) using simple geometric shapes
132 and boolean combinations of shapes. For a complete description of
133 regions, see Spatial Region Filtering.
134 Separators Also Are Operators
136 As mentioned previously, multiple filter expressions can be specified
138 in a filter descriptor, separated by commas or new-lines. When such a
139 comma or new-line separator is used, the boolean AND operator is auto‐
140 matically generated in its place. Thus and expression such as:
141 pha==1,pi=2:4
143 is equivalent to:
145 (pha==1) && (pi>=2&&pi<=4)
147 [Note that the behavior of separators is different for filter expres‐
149 sions and spatial region expressions. The former uses AND as the oper‐
150 ator, while the latter user OR. See Combining Region and Table Filters
151 for more information about these conventions and how they are treated
152 when combined.]
153 Range Lists
155 Aside from the standard C syntax, filter expressions can make use of
157 IRAF-style range lists which specify a range of values. The syntax
158 requires that the column name be followed by an '=' sign, which is fol‐
159 lowed by one or more comma-delimited range expressions of the form:
160 col = vv # col == vv in range
162 col = :vv # col <= vv in range
163 col = vv: # col >= vv in range
164 col = vv1:vv2 # vv1 <= col <= vv2 in range
165 The vv's above must be numeric constants; the right hand side of a
167 range list cannot contain a column name or header value.
168 Note that, unlike an ordinary comma separator, the comma separator used
170 between two or more range expressions denotes OR. Thus, when two or
171 more range expressions are combined with a comma separator, the result‐
172 ing expression is a shortcut for more complicated boolean logic. For
173 example:
174 col = :3,6:8,10:
176 is equivalent to:
178 (col=6 && col =10)
180 Note also that the single-valued rangelist:
182 col = val
184 is equivalent to the C-based filter expression:
186 col == val
188 assuming, of course, that val is a numeric constant.
190 Math Operations and Functions
192 It is permissible to specify C math functions as part of the filter
194 syntax. When the filter parser recognizes a function call, it automat‐
195 ically includes the math.h and links in the C math library. Thus, it
196 is possible to filter rows by expressions such as these:
197 · (pi+pha)>(2+log(pi)-pha)
199 · min(pi,pha)*14>x
201 · max(pi,pha)==(pi+1)
203 · feq(pi,pha)
205 · div(pi,pha)>0
207 The function feq(a,b) returns true (1) if the difference between a and
209 b (taken as double precision values) is less than approximately 10E-15.
210 The function div(a,b) divides a by b, but returns NaN (not a number) if
211 b is 0. It is a safe way to avoid floating point errors when dividing
212 one column by another.
213 Include Files
215 The special @filename directive specifies an include file containing
217 filter expressions. This file is processed as part of the overall fil‐
218 ter descriptor:
219 foo.fits[pha==1,@foo]
221 Header Parameters
223 The filter syntax supports comparison between a column value and a
225 header parameter value of a FITS binary tables (raw event files have no
226 such header). The header parameters can be taken from the binary table
227 header or the primary header. For example, assuming there is a header
228 value MEAN_PHA in one of these headers, you can select photons having
229 exactly this value using:
230 · pha==MEAN_PHA
232 Table filtering is more easily described by means of examples. Con‐
234 sider data containing the following table structure:
235 · double TIME
237 · int X
239 · int Y
241 · short PI
243 · short PHA
245 · int DX
247 · int DY
249 Tables can be filtered on these columns using IRAF/QPOE range syntax or
251 any valid C syntax. The following examples illustrate the possibili‐
252 ties:
253 · pha=10
255 · pha==10
257 select rows whose pha value is exactly 10
259 · pha=10:50
261 select rows whose pha value is in the range of 10 to 50
263 · pha=10:50,100
265 select rows whose pha value is in the range of 10 to 50 or is equal
267 to 100
268 · pha>=10 && pha<=50
270 select rows whose pha value is in the range of 10 to 50
272 · pi=1,2&&pha>3
274 select rows whose pha value is 1 or 2 and whose pi value is 3
276 · pi=1,2 ⎪⎪ pha>3
278 select rows whose pha value is 1 or 2 or whose pi value is 3
280 · pha==pi+1
282 select rows whose pha value is 1 less than the pi value
284 · (pha==pi+1) && (time>50000.0)
286 select rows whose pha value is 1 less than the pi value and whose
288 time value is greater than 50000
289 · (pi+pha)>20
291 select rows in which the sum of the pi and pha values is greater
293 than 20
294 · pi%2==1
296 select rows in which the pi value is odd
298 Currently, integer range list limits cannot be specified in binary
300 notation (use decimal, hex, or octal instead). Please contact us if
301 this is a problem.
302
304 See funtools(n) for a list of Funtools help pages
305version 1.4.0 August 15, 2007 funfilters(n)