1funfiles(n) SAORD Documentation funfiles(n)
2
6 FunFiles: Funtools Data Files
7
9 This document describes the data file formats (FITS, array, raw events)
10 as well as the file types (gzip, socket, etc.) supported by Funtools.
11
13 Funtools supports FITS images and binary tables, and binary files con‐
14 taining array (homogeneous) data or event (heterogeneous) data. IRAF-
15 style brackets are appended to the filename to specify various kinds of
16 information needed to characterize these data:
17 file[ext⎪ind⎪ARRAY()⎪EVENTS(),section][filters]
19 or
20 file[ext⎪ind⎪ARRAY()⎪EVENTS(),section,filters]
21 where:
23 · file is the Funtools file name
25 · ext is the FITS extension name
27 · ind is the FITS extension number
29 · ARRAY() is an array specification
31 · EVENTS() is an event specification
33 · section is the image section specification
35 · filters are spatial region and table (row) filters
37 Supported Data Formats
39 Funtools programs (and the underlying libraries) support the following
41 data file formats:
42 · FITS images (and image extensions)
44 · FITS binary tables
46 · binary files containing an array of homogeneous data
48 · binary files containing events, i.e. records of heterogeneous data
50 · column-based text files, which are documented here
52 · non-disk files and lists of files
54 Information needed to identify and characterize the event or image data
56 can be specified on the command line using IRAF-style bracket notation
57 appended to the filename:
58 foo.fits # open FITS default extension
60 image.fits[3] # open FITS extension #3
61 events.fits[EVENTS] # open EVENTS extension
62 array.file[ARRAY(s1024)] # open 1024x1024 short array
63 events.file[EVENTS(x:1024,y:1024...)] # open non-FITS event list
64 Note that in many Unix shells (e.g., csh and tcsh), filenames must be
66 enclosed in quotes to protect the brackets from shell processing.
67 FITS Images and Binary Tables
69 When FunOpen() opens a FITS file without a bracket specifier, the
71 default behavior is to look for a valid image in the primary HDU. In
72 the absence of a primary image, Funtools will try to open an extension
73 named either EVENTS or STDEVT, if one of these exists. This default
74 behavior supports both FITS image processing and standard X-ray event
75 list processing (which, after all, is what we at SAO/HEAD do).
76 In order to open a FITS binary table or image extension explicitly, it
78 is necessary to specify either the extension name or the extension num‐
79 ber in brackets:
80 foo.fits[1] # open extension #1: the primary HDU
82 foo.fits[3] # open extension #3 of a FITS file
83 foo.fits[GTI] # open GTI extension of a FITS file
84 The ext argument specifies the name of the FITS extension (i.e. the
86 value of the EXTENSION header parameter in a FITS extension), while the
87 index specifies the value of the FITS EXTVER header parameter. Follow‐
88 ing FITS conventions, extension numbers start at 1.
89 When a FITS data file is opened for reading using FunOpen(), the speci‐
91 fied extension is automatically located and is used to initialize the
92 Funtools internal data structures.
93 Non-FITS Raw Event Files
95 In addition to FITS tables, Funtools programs and libraries can operate
97 on non-FITS files containing heterogeneous event records. To specify
98 such an event file, use:
99 · file[EVENTS(event-spec)]
101 · file[EVENTS()]
103 where event-spec is a string that specified the names, data types, and
105 optional image dimensions for each element of the event record:
106 · [name]:[n][type]:[(lodim:)hidim]
108 Data types follow standard conventions for FITS binary tables, but
110 include two extra unsigned types ('U' and 'V'):
111 · B -- unsigned 8-bit char
113 · I -- signed 16-bit int
115 · J -- signed 32-bit int
117 · K -- signed 64-bit int
119 · E -- 32-bit float
121 · D -- 64-bit float
123 · U -- unsigned 16-bit int
125 · V -- unsigned 32-bit int
127 An optional integer value n can be prefixed to the type to indicate
129 that the element is an array of n values. For example:
130 foo.fits[EVENTS(x:I,y:I,status:4J)]
132 defines x and y as 16-bit ints and status as an array of 4 32-bit ints.
134 Furthermore, image dimensions can be attached to the event specifica‐
136 tion in order to tell Funtools how to bin the events into an image.
137 They follow the conventions for the FITS TLMIN/TLMAX keywords. If the
138 low image dimension is not specified, it defaults to 1. Thus:
139 · RAWX:J:1:100
141 · RAWX:J:100
143 both specify that the dimension of this column runs from 1 to 100.
145 NB: it is required that all padding be specified in the record defini‐
147 tion. Thus, when writing out whole C structs instead of individual
148 record elements, great care must be taken to include the compiler-added
149 padding in the event definition.
150 For example, suppose a FITS binary table has the following set of col‐
152 umn definitions:
153 TTYPE1 = 'X ' / Label for field
155 TFORM1 = '1I ' / Data type for field
156 TLMIN1 = 1 / Min. axis value
157 TLMAX1 = 10 / Max. axis value
158 TTYPE2 = 'Y ' / Label for field
159 TFORM2 = '1I ' / Data type for field
160 TLMIN2 = 2 / Min. axis value
161 TLMAX2 = 11 / Max. axis value
162 TTYPE3 = 'PHA ' / Label for field
163 TFORM3 = '1I ' / Data type for field
164 TTYPE4 = 'PI ' / Label for field
165 TFORM4 = '1J ' / Data type for field
166 TTYPE5 = 'TIME ' / Label for field
167 TFORM5 = '1D ' / Data type for field
168 TTYPE6 = 'DX ' / Label for field
169 TFORM6 = '1E ' / Data type for field
170 TLMIN6 = 1 / Min. axis value
171 TLMAX6 = 10 / Max. axis value
172 TTYPE7 = 'DY ' / Label for field
173 TFORM7 = '1E ' / Data type for field
174 TLMIN7 = 3 / Min. axis value
175 TLMAX7 = 12 / Max. axis value
176 An raw event file containing these same data would have the event spec‐
178 ification:
179 EVENTS(X:I:10,Y:I:2:11,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:3:12)
181 If no event specification string is included within the EVENTS() opera‐
183 tor, then the event specification is taken from the EVENTS environment
184 variable:
185 setenv EVENTS "X:I:10,Y:I:10,PHA:I,PI:J,TIME:D,DX:E:10,DY:E:10"
187 In addition to knowing the data structure, it is necessary to know the
189 endian ordering of the data, i.e., whether or not the data is in bigen‐
190 dian format, so that we can convert to the native format for this plat‐
191 form. This issue does not arise for FITS Binary Tables because all FITS
192 files use big-endian ordering, regardless of platform. But for non-FITS
193 data, big-endian data produced on a Sun workstation but read on a Linux
194 PC needs to be byte-swapped, since PCs use little-endian ordering. To
195 specify an ordering, use the bigendian= or endian= keywords on the com‐
196 mand-line or the EVENTS_BIGENDIAN or EVENTS_ENDIAN environment vari‐
197 ables. The value of the bigendian variables should be "true" or
198 "false", while the value of the endian variables should be "little" or
199 "big".
200 For example, a PC can access data produced by a Sun using:
202 hrc.nepr[EVENTS(),bigendian=true]
204 or
205 hrc.nepr[EVENTS(),endian=big]
206 or
207 setenv EVENTS_BIGENDIAN true
208 or
209 setenv EVENTS_ENDIAN big
210 If none of these are specified, the data are assumed to follow the for‐
212 mat for that platform and no byte-swapping is performed.
213 Non-FITS Array Files
215 In addition to FITS images, Funtools programs and libraries can operate
217 on non-FITS files containing arrays of homogeneous data. To specify an
218 array file, use:
219 · file[ARRAY(array-spec)]
221 · file[ARRAY()]
223 where array-spec is of the form:
225 · [type][dim1][.dim2][:skip][endian]
227 and where [type] is:
229 · b (8-bit unsigned char)
231 · s (16-bit short int)
233 · u (16-bit unsigned short int)
235 · i (32-bit int)
237 · r,f (32-bit float)
239 · d (64-bit float)
241 The dim1 specification is required, but dim2 is optional and defaults
243 to dim1. The skip specification is optional and defaults to 0. The
244 optional endian specification can be 'l' or 'b' and defaults to the
245 endian type for the current machine.
246 If no array specification is included within the ARRAY() operator, then
248 the array specification is taken from the ARRAY environment variable.
249 For example:
250 foo.arr[ARRAY(r512)] # bitpix=-32 dim1=512 dim2=512
252 foo.arr[ARRAY(r512.400)] # bitpix=-32 dim1=512 dim2=400
253 foo.arr[ARRAY(r512.400]) # bitpix=-32 dim1=512 dim2=400
254 foo.arr[ARRAY(r512.400:2880)] # bitpix=-32 dim1=512 dim2=400 skip=2880
255 foo.arr[ARRAY(r512l)] # bitpix=-32 dim1=512 dim2=512 endian=little
256 setenv ARRAY "r512.400:2880"
257 foo.arr[ARRAY()] # bitpix=-32 dim1=512 dim2=400 skip=2880
258 Specifying Image Sections
260 Once a data file (and possibly, a FITS extension) has been specified,
262 the next (optional) part of a bracket specification can be used to
263 select image section information, i.e., to specify the x,y limits of an
264 image section, as well as the blocking factor to apply to that section.
265 This information can be added to any file specification but only is
266 used by Funtools image processing routines.
267 The format of the image section specification is one of the following:
269 · file[xy0:xy1,block]
271 · file[x0:x1,y0:y1,block]
273 · file[x0:x1,*,block]
275 · file[*,y0:y1,block]
277 · file[*,block]
279 where the limit values can be ints or "*" for default. A single "*" can
281 be used instead of val:val, as shown. Note that blocking is applied to
282 the section after it is extracted.
283 In addition to image sections specified by the lo and hi x,y limits,
285 image sections using center positions can be specified:
286 · file[dim1@xcen,dim2@ycen]
288 · file[xdim2@xcen@ycen]
290 · file[dim1@xcen,dim2@ycen,block]
292 · file[dim@xcen@ycen,block]
294 Note that the (float) values for dim, dim1, dim2, xcen, ycen must be
296 specified or else the expression does not make sense!
297 In all cases, block is optional and defaults to 1. An 's' or 'a' can be
299 appended to signify "sum" or "average" blocking (default is "sum").
300 Section specifications are given in image coordinates by default. If
301 you wish to specify physical coordinates, add a 'p' as the last charac‐
302 ter of the section specification, before the closing bracket. For
303 example:
304 · file[-8:-7,-8:-7p]
306 · file[-8:-7,-8:-7,2p]
308 A section can be specified in any Funtools file name. If the operation
310 to be applied to that file is an imaging operation, then the specifica‐
311 tion will be utilized. If the operation is purely a table operation,
312 then the section specification is ignored.
313 Do not be confused by:
315 foo.fits[2]
317 foo.fits[*,2]
318 The former specifies opening the second extension of the FITS file.
320 The latter specifies application of block 2 to the image section.
321 Note that the section specification must come after any of FITS ext
323 name or ind number, but all sensible defaults are supported:
324 · file[ext]
326 · file[ext,index]
328 · file[index]
330 · file[ext,section]
332 · file[ext,index,section]
334 · file[index,section]
336 · file[section]
338 Binning FITS Binary Tables and Non-FITS Event Files
340 If a FITS binary table or a non-FITS raw event file is to be binned
342 into a 2D image (e.g., using the funimage program), it is necessary to
343 specify the two columns to be used for the binning, as well as the
344 dimensions of the image. Funtools first looks for a specifier of the
345 form:
346 bincols=([xnam[:tlmin[:tlmax:[binsiz]]]],[ynam[:tlmin[:tlmax[:binsiz]]]])
348 in bracket syntax, and uses the column names thus specified. The tlmin,
350 tlmax, and binsiz specifiers determine the image binning dimensions
351 using:
352 dim = (tlmax - tlmin)/binsiz (floating point data)
354 dim = (tlmax - tlmin)/binsiz + 1 (integer data)
355 These tlmin, tlmax, and binsiz specifiers can be omitted if TLMIN,
357 TLMAX, and TDBIN header parameters are present in the FITS binary table
358 header, respectively. If only one parameter is specified, it is assumed
359 to be tlmax, and tlmin defaults to 1. If two parameters are specified,
360 they are assumed to be tlmin and tlmax.
361 For example, to bin an HRC event list columns "VPOS" and "UPOS", use:
363 hrc.nepr[bincols=(VPOS,UPOS)]
365 or
367 hrc.nepr[bincols=(VPOS:49152,UPOS:4096)]
369 Note that you can optionally specify the dimensions of these columns to
371 cover cases where neither TLMAX keywords are defined in the header. If
372 either dimension is specified, then both must be specified.
373 You can set the FITS_BINCOLS or EVENTS_BINCOLS environment variable as
375 an alternative to adding the "bincols=" specifier to each file name for
376 FITS binary tables and raw event files, respectively. If no binning
377 keywords or environment variables are specified, or if the specified
378 columns are not in the binary table, the Chandra parameters CPREF (or
379 PREFX) are searched for in the FITS binary table header. Failing this,
380 columns named "X" and "Y" are sought. If these are not found, the code
381 looks for columns containing the characters "X" and "Y". Thus, you can
382 bin on "DETX" and "DETX" columns without specifying them, if these are
383 the only column names containing the "X" and "Y" characters.
384 Ordinarily, each event or row contributes one count to an image pixel
386 during the 2D binning process. Thus, if five events all have the same
387 (x,y) position, the image pixel value for that position will have a
388 value of five. It is possible to specify a variable contribution for
389 each event by using the vcol=[colname] filter spec:
390 vcol=[colname]
392 The vcol colname is a column containing a numeric value in each event
394 row that will be used as the contribution of the given event to its
395 image pixel. For example, consider an event file that has the following
396 content:
397 x:e:4 y:e:4 v:e
399 ------ ------ ----
400 1 1 1.0
401 2 2 2.0
402 3 3 3.0
403 4 4 0.0
404 1 1 1.0
405 2 2 2.0
406 3 3 3.0
407 4 4 4.0
408 There are two events with x,y value of (1,1) so ordinarily a 2D image
410 will have a value of 2 in the (1,1) pixel. If the v column is specified
411 as the value column:
412 foo.fits'[vcol=v]'
414 then each pixel will contain the additive sum of the associated (x,y)
416 column values from the v column. For example, image pixel (1,1) will
417 contain 1. + 1. = 2, image pixel (2,2) will contain (2 + 2) = 4, etc.
418 An important variation on the use of a value column to specify the con‐
420 tribution an event makes to an image pixel is when the value column
421 contains the reciprocal of the event contribution. For this case, the
422 column name should be prefixed with a / (divide sign) thus:
423 foo.fits'[vcol=/v]'
425 Each image pixel value will then be the sum of the reciprocals of the
427 value column. A zero in the value column results in NaN (not a number).
428 Thus, in the above example, image pixel (1.1) will contain 1/1 + 1/1 =
429 2, image pixel (2,2) will contain (1/2 + 1/2) = 1, etc. Image pixel
430 (4,4) will contain (1/0 + 1/4) = NaN.
431 You can set the FITS_VCOL or EVENTS_VCOL environment variable as an
433 alternative to adding the "vcol=" specifier to each file name for FITS
434 binary tables and raw event files, respectively.
435 Finally, when binning events, the data type of the resulting 2D image
437 must be specified. This can be done with the "bitpix=[n]" keyword in
438 the bracket specification. For example:
439 events.fits[bincols=(VPOS,UPOS),bitpix=-32]
441 will create a floating point image binned on columns VPOS and UPOS. If
443 no bitpix keyword is specified, bitpix=32 is assumed. As with bincols
444 values, you also can use the FITS_BITPIX and EVENTS_BITPIX environment
445 variables to set this value for FITS binary tables and raw event files,
446 respectively.
447 The funimage program also allows you to create a 1D image projection
449 along any column of a table by using the bincols=[column] filter speci‐
450 fication and specifying a single column. For example, the following
451 command projects a 1D image along the chipx column of a table:
452 funimage ev.fits'[bincols=chipx]' im.fits
454 See funimage for more information about creating 1D and 2D images.
456 Finally, please note that Funtools supports most FITS standards. We
458 will add missing support as required by the community. In general, how‐
459 ever, we do not support non-standard extensions. For example, we sense
460 the presence of the binary table 'variable length array' proposed
461 extension and we pass it along when copying and filtering files, but we
462 do not process it. We will add support for new standards as they become
463 official.
464 Table and Spatial Region Filters
466 Note that, in addition extensions and image sections, Funtools bracket
468 notation can be used to specify table and spatial region filters.
469 These filters are always placed after the image section information.
470 They can be specified in the same bracket or in a separate bracket
471 immediately following:
472 · file[ext⎪ind⎪ARRAY()⎪EVENTS(),section][filters]
474 · file[ext⎪ind⎪ARRAY()⎪EVENTS(),section,filters]
476 where:
478 · file is the Funtools file name
480 · ARRAY() is an array specification
482 · EVENTS() is an event list specification
484 · ext is the FITS extension name
486 · ind is the FITS extension number
488 · section is the image section to extract
490 · filters are spatial region and table (row) filters to apply
492 The topics of table and region filtering are covered in detail in:
494 · Table Filtering
496 · Spatial Region Filtering
498 Disk Files and Other Supported File Types
500 The specified file usually is an ordinary disk file. In addition,
502 gzip'ed files are supported in Funtools: gzip'ed input files are auto‐
503 matically uncompressed as they are read, and gzip'ed output files are
504 compressed as they are written. NB: if a FITS binary table is written
505 in gzip format, the number of rows in the table will be set to -1. Such
506 a file will work with Funtools programs but will not work with other
507 FITS programs such as ds9.
508 The special keywords "stdin" and "stdout" designate Unix standard input
510 and standard output, respectively. The string "-" (hyphen) will be
511 taken to mean "stdin" if the file is opened for reading and "stdout" if
512 the file is opened for writing.
513 A file also can be an INET socket on the same or another machine using
515 the syntax:
516 machine:port
518 Thus, for example:
520 karapet:1428
522 specifies that I/O should be performed to/from port 1428 on the machine
524 karapet. If no machine name is specified, the default is to use the
525 current machine:
526 :1428
528 This means to open port 1428 on the current machine. Socket support
530 allows you to generate a distributed pipe:
531 on karapet: funtask1 in.fits bynars:1428
533 on bynars: funtask2 :1428 out.fits
534 The socket mechanism thus supports simple parallel processing using
536 process decomposition. Note that parallel processing using data decom‐
537 position is supported via the section specifier (see below), and the
538 row# specifier, which is part of Table Filtering.
539 A file also can be a pointer to shared memory using the syntax:
541 shm:[id⎪@key][:size]
543 A shared memory segment is specified with a shm: prefix, followed by
545 either the shared memory id or the shared memory key (where the latter
546 is prefixed by the '@' character). The size (in bytes) of the shared
547 memory segment can then be appended (preceded by the ':' character). If
548 the size specification is absent, the code will attempt to determine
549 the length automatically.
550 If the open mode contains the string "w+", then the memory segment will
552 be created if it does not exist. (It also will be released and deleted
553 when the file is closed.) In the case where a memory segment is being
554 created, the length of the segment is required.
555 A file also can be Unix piped command (i.e. a program to run) using the
557 syntax:
558 "pipe: command arg1 ... argn"
560 The output from the command must be a valid FITS file. It is important
562 to use quotes to protect spaces so that command arguments are passed
563 correctly. A silly example is:
564 fundisp "pipe: funtable 'foo.fits[cir 512 512 .1]' stdout"
566 This seemed like a good idea at the time ...
568 Lists of Files
570 Funtools also will process a list of files as a single file using the
572 syntax:
573 "list: file1 file2 ... filen"
575 The files in the list are separated by whitespace. Any of the above
577 file types can be used. For example, if two files, foo1.fits and
578 foo2.fits, are part of the same observation, they can be processed as a
579 single file (using their own filters):
580 fundisp "list: foo1.fits[cir(512,512,10)] foo2.fits[cir(511,511,10)]"
582 X Y PHA PI TIME DX DY
583 -------- -------- -------- -------- --------------------- -------- --------
584 512 512 6 7 79493997.45854475 578 574
585 512 512 8 9 79494575.58943175 579 573
586 512 512 5 6 79493631.03866175 578 575
587 512 512 5 5 79493290.86521725 578 575
588 512 512 8 9 79493432.00990875 579 573
589 511 511 5 5 79488631.09462625 580 575
590 511 511 10 11 79488780.60006675 580 573
591 511 511 4 4 79494562.35474326 580 575
592 511 511 6 6 79488203.01561825 580 575
593 511 511 6 6 79488017.99730176 580 575
594 511 511 4 4 79494332.45355175 580 575
595 511 511 9 10 79492685.94014275 581 574
596 511 511 5 5 79487708.71298325 580 575
597 511 511 8 9 79493719.00160225 581 573
598 Again, note that it is important to avoid spaces in the filters because
600 the list separator also is whitespace. To protect whitespace in a fil‐
601 ter, enclose the file specification in quotes:
602 fundisp "list: 'foo1.fits[cir 512 512 .1]' foo2.fits[cir(511,511,.1)]"
604
606 See funtools(n) for a list of Funtools help pages
607version 1.4.0 August 15, 2007 funfiles(n)