1Fsdb(3) User Contributed Perl Documentation Fsdb(3)
2
6 Fsdb - a flat-text database for shell scripting
7
9 Fsdb, the flatfile streaming database is package of commands for
10 manipulating flat-ASCII databases from shell scripts. Fsdb is useful
11 to process medium amounts of data (with very little data you'd do it by
12 hand, with megabytes you might want a real database). Fsdb was known
13 as as Jdb from 1991 to Oct. 2008.
14 Fsdb is very good at doing things like:
16 • extracting measurements from experimental output
18 • examining data to address different hypotheses
20 • joining data from different experiments
22 • eliminating/detecting outliers
24 • computing statistics on data (mean, confidence intervals,
26 correlations, histograms)
27 • reformatting data for graphing programs
29 Fsdb is built around the idea of a flat text file as a database. Fsdb
31 files (by convention, with the extension .fsdb), have a header
32 documenting the schema (what the columns mean), and then each line
33 represents a database record (or row).
34 For example:
36 #fsdb experiment duration
38 ufs_mab_sys 37.2
39 ufs_mab_sys 37.3
40 ufs_rcp_real 264.5
41 ufs_rcp_real 277.9
42 Is a simple file with four experiments (the rows), each with a
44 description, size parameter, and run time in the first, second, and
45 third columns.
46 Rather than hand-code scripts to do each special case, Fsdb provides
48 higher-level functions. Although it's often easy throw together a
49 custom script to do any single task, I believe that there are several
50 advantages to using Fsdb:
51 • these programs provide a higher level interface than plain Perl, so
53 ** Fewer lines of simpler code:
55 dbrow '_experiment eq "ufs_mab_sys"' | dbcolstats duration
57 Picks out just one type of experiment and computes statistics
59 on it, rather than:
60 while (<>) { split; $sum+=$F[1]; $ss+=$F[1]**2; $n++; }
62 $mean = $sum / $n; $std_dev = ...
63 in dozens of places.
65 • the library uses names for columns, so
67 ** No more $F[1], use "_duration".
69 ** New or different order columns? No changes to your scripts!
71 Thus if your experiment gets more complicated with a size
73 parameter, so your log changes to:
74 #fsdb experiment size duration
76 ufs_mab_sys 1024 37.2
77 ufs_mab_sys 1024 37.3
78 ufs_rcp_real 1024 264.5
79 ufs_rcp_real 1024 277.9
80 ufs_mab_sys 2048 45.3
81 ufs_mab_sys 2048 44.2
82 Then the previous scripts still work, even though duration is now
84 the third column, not the second.
85 • A series of actions are self-documenting (the provenance of
87 processsing done to produce each output is recorded in comments).
88 ** No more wondering what hacks were used to compute the final
90 data, just look at the comments at the end of the output.
91 For example, the commands
93 dbrow '_experiment eq "ufs_mab_sys"' | dbcolstats duration
95 add to the end of the output the lines
97 # | dbrow _experiment eq "ufs_mab_sys"
98 # | dbcolstats duration
99 • The library is mature, supporting large datasets (more than 100GB),
101 parallelism, corner cases, error handling, backed by an automated
102 test suite.
103 ** No more puzzling about bad output because your custom script
105 skimped on error checking.
106 ** No more memory thrashing when you try to sort ten million
108 records.
109 ** Makes use of multiple cores in your computer when it can,
111 because each pipeline component runs in parallel, and because
112 key tools (dbsort, dbmapreduce) run in parlallel when possible.
113 • Fsdb-2.x supports Perl scripting (in addition to shell scripting),
115 with libraries to do Fsdb input and output, and easy support for
116 pipelines. The shell script
117 dbcol name test1 | dbroweval '_test1 += 5;'
119 can be written in perl as:
121 dbpipeline(dbcol(qw(name test1)), dbroweval('_test1 += 5;'));
123 (The disadvantage is that you need to learn what functions Fsdb
125 provides.)
126 Fsdb is built on flat-ASCII databases. By storing data in simple text
128 files and processing it with pipelines it is easy to experiment (in the
129 shell) and look at the output. To the best of my knowledge, the
130 original implementation of this idea was "/rdb", a commercial product
131 described in the book UNIX relational database management: application
132 development in the UNIX environment by Rod Manis, Evan Schaffer, and
133 Robert Jorgensen (1988 by Prentice Hall, and also at the web page
134 <http://www.rdb.com/>). Fsdb is an incompatible re-implementation of
135 their idea without any accelerated indexing or forms support. (But
136 it's free, and probably has better statistics!).
137 Fsdb-2.x will exploit multiple processors or cores, and provides Perl-
139 level support for input, output, and threaded-pipelines. (As of
140 Fsdb-2.44 it no longer uses Perl threading, just processes, since they
141 are faster.)
142 Installation instructions follow at the end of this document. Fsdb-2.x
144 requires Perl 5.8 to run. All commands have manual pages and provide
145 usage with the "--help" option. All commands are backed by an
146 automated test suite.
147 The most recent version of Fsdb is available on the web at
149 <http://www.isi.edu/~johnh/SOFTWARE/FSDB/index.html>.
150
152 3.1, 2022-11-22 A post-3.0 cleanup release with minor fixes.
153 ENHANCEMENT
154 Type specifications in a few more programs that I missed:
155 dbrowuniq, dbcolpercentile.
156 ENHANCEMENT
158 Minor documentation improvements.
159
161 executive summary
162 what's new
163 README CONTENTS
164 installation
165 basic data format
166 basic data manipulation
167 list of commands
168 another example
169 a gradebook example
170 a password example
171 history
172 related work
173 release notes
174 copyright
175 comments
176
178 Fsdb now uses the standard Perl build and installation from
179 ExtUtil::MakeMaker(3), so the quick answer to installation is to type:
180 perl Makefile.PL
182 make
183 make test
184 make install
185 Or, if you want to install it somewhere else, change the first line to
187 perl Makefile.PL PREFIX=$HOME
189 and it will go in your home directory's bin, etc. (See
191 ExtUtil::MakeMaker(3) for more details.)
192 Fsdb requires perl 5.8 or later.
194 A test-suite is available, run it with
196 make test
198 In the past, the ports existed for FreeBSD and MacOS. If someone
200 running one of those OSes wants to contribute a new port, please let me
201 know.
202
204 These programs are based on the idea storing data in simple ASCII
205 files. A database is a file with one header line and then data or
206 comment lines. For example:
207 #fsdb account passwd uid gid fullname homedir shell
209 johnh * 2274 134 John_Heidemann /home/johnh /bin/bash
210 greg * 2275 134 Greg_Johnson /home/greg /bin/bash
211 root * 0 0 Root /root /bin/bash
212 # this is a simple database
213 The header line must be first and begins with "#fsdb". There are rows
215 (records) and columns (fields), just like in a normal database.
216 Comment lines begin with "#". Column names are any string not
217 containing spaces or single quote (although it is prudent to keep them
218 alphanumeric with underscore).
219 Columns can optionally include type anntations by following name with
221 :t where t is some type. (Types are not used in Perl, but are relevant
222 in Python and Go Fsdb bindings.) Types use a subset of perl pack
223 specifiers: c, s, l, q are signed 8, 16, 32, and 64-bit integers, f is
224 a float, d is double float, a is utf-8 string, and > and < can
225 force big or little endianness.
226 By default, columns are delimited by whitespace. With this default
228 configuration, the contents of a field cannot contain whitespace.
229 However, this limitation can be relaxed by changing the field separator
230 as described below.
231 The big advantage of simple flat-text databases is that it is usually
233 easy to massage data into this format, and it's reasonably easy to take
234 data out of this format into other (text-based) programs, like gnuplot,
235 jgraph, and LaTeX. Think Unix. Think pipes. (Or even output to Excel
236 and HTML if you prefer.)
237 Since no-whitespace in columns was a problem for some applications,
239 there's an option which relaxes this rule. You can specify the field
240 separator in the table header with "-F x" where "x" is a code for the
241 new field separator. A full list of codes is at dbfilealter(1), but
242 two common special values are "-F t" which is a separator of a single
243 tab character, and "-F S", a separator of two spaces. Both allowing
244 (single) spaces in fields. An example:
245 #fsdb -F S account passwd uid gid fullname homedir shell
247 johnh * 2274 134 John Heidemann /home/johnh /bin/bash
248 greg * 2275 134 Greg Johnson /home/greg /bin/bash
249 root * 0 0 Root /root /bin/bash
250 # this is a simple database
251 See dbfilealter(1) for more details. Regardless of what the column
253 separator is for the body of the data, it's always whitespace in the
254 header.
255 There's also a third format: a "list". Because it's often hard to see
257 what's columns past the first two, in list format each "column" is on a
258 separate line. The programs dblistize and dbcolize convert to and from
259 this format, and all programs work with either formats. The command
260 dbfilealter -R C < DATA/passwd.fsdb
262 outputs:
264 #fsdb -R C account passwd uid gid fullname homedir shell
266 account: johnh
267 passwd: *
268 uid: 2274
269 gid: 134
270 fullname: John_Heidemann
271 homedir: /home/johnh
272 shell: /bin/bash
273 account: greg
275 passwd: *
276 uid: 2275
277 gid: 134
278 fullname: Greg_Johnson
279 homedir: /home/greg
280 shell: /bin/bash
281 account: root
283 passwd: *
284 uid: 0
285 gid: 0
286 fullname: Root
287 homedir: /root
288 shell: /bin/bash
289 # this is a simple database
291 # | dblistize
292 See dbfilealter(1) for more details.
294
296 A number of programs exist to manipulate databases. Complex functions
297 can be made by stringing together commands with shell pipelines. For
298 example, to print the home directories of everyone with ``john'' in
299 their names, you would do:
300 cat DATA/passwd | dbrow '_fullname =~ /John/' | dbcol homedir
302 The output might be:
304 #fsdb homedir
306 /home/johnh
307 /home/greg
308 # this is a simple database
309 # | dbrow _fullname =~ /John/
310 # | dbcol homedir
311 (Notice that comments are appended to the output listing each command,
313 providing an automatic audit log.)
314 In addition to typical database functions (select, join, etc.) there
316 are also a number of statistical functions.
317 The real power of Fsdb is that one can apply arbitrary code to rows to
319 do powerful things.
320 cat DATA/passwd | dbroweval '_fullname =~ s/(\w+)_(\w+)/$2,_$1/'
322 converts "John_Heidemann" into "Heidemann,_John". Not too much more
324 work could split fullname into firstname and lastname fields.
325 (Or:
327 cat DATA/passwd | dbcolcreate sort | dbroweval -b 'use Fsdb::Support'
329 '_sort = _fullname; _sort =~ s/_/ /g; _sort = fullname_to_sort(_sort);'
330
332 An advantage of Fsdb is that you can talk about columns by name
333 (symbolically) rather than simply by their positions. So in the above
334 example, "dbcol homedir" pulled out the home directory column, and
335 "dbrow '_fullname =~ /John/'" matched against column fullname.
336 In general, you can use the name of the column listed on the "#fsdb"
338 line to identify it in most programs, and _name to identify it in code.
339 Some alternatives for flexibility:
341 • Numeric values identify columns positionally, numbering from 0. So
343 0 or _0 is the first column, 1 is the second, etc.
344 • In code, _last_columnname gets the value from columname's previous
346 row.
347 See dbroweval(1) for more details about writing code.
349
351 Enough said. I'll summarize the commands, and then you can experiment.
352 For a detailed description of each command, see a summary by running it
353 with the argument "--help" (or "-?" if you prefer.) Full manual pages
354 can be found by running the command with the argument "--man", or
355 running the Unix command "man dbcol" or whatever program you want.
356 TABLE CREATION
358 dbcolcreate
359 add columns to a database
360 dbcoldefine
362 set the column headings for a non-Fsdb file
363 TABLE MANIPULATION
365 dbcol
366 select columns from a table
367 dbrow
369 select rows from a table
370 dbsort
372 sort rows based on a set of columns
373 dbjoin
375 compute the natural join of two tables
376 dbcolrename
378 rename a column
379 dbcolmerge
381 merge two columns into one
382 dbcolsplittocols
384 split one column into two or more columns
385 dbcolsplittorows
387 split one column into multiple rows
388 dbfilepivot
390 "pivots" a file, converting multiple rows corresponding to the same
391 entity into a single row with multiple columns.
392 dbfilevalidate
394 check that db file doesn't have some common errors
395 COMPUTATION AND STATISTICS
397 dbcolstats
398 compute statistics over a column (mean,etc.,optionally median)
399 dbmultistats
401 group rows by some key value, then compute stats (mean, etc.) over
402 each group (equivalent to dbmapreduce with dbcolstats as the
403 reducer)
404 dbmapreduce
406 group rows (map) and then apply an arbitrary function to each group
407 (reduce)
408 dbrvstatdiff
410 compare two samples distributions (mean/conf interval/T-test)
411 dbcolmovingstats
413 computing moving statistics over a column of data
414 dbcolstatscores
416 compute Z-scores and T-scores over one column of data
417 dbcolpercentile
419 compute the rank or percentile of a column
420 dbcolhisto
422 compute histograms over a column of data
423 dbcolscorrelate
425 compute the coefficient of correlation over several columns
426 dbcolsregression
428 compute linear regression and correlation for two columns
429 dbrowaccumulate
431 compute a running sum over a column of data
432 dbrowcount
434 count the number of rows (a subset of dbstats)
435 dbrowdiff
437 compute differences between a columns in each row of a table
438 dbrowenumerate
440 number each row
441 dbroweval
443 run arbitrary Perl code on each row
444 dbrowuniq
446 count/eliminate identical rows (like Unix uniq(1))
447 dbfilediff
449 compare fields on rows of a file (something like Unix diff(1))
450 OUTPUT CONTROL
452 dbcolneaten
453 pretty-print columns
454 dbfilealter
456 convert between column or list format, or change the column
457 separator
458 dbfilestripcomments
460 remove comments from a table
461 dbformmail
463 generate a script that sends form mail based on each row
464 CONVERSIONS
466 (These programs convert data into fsdb. See their web pages for
467 details.)
468 cgi_to_db
470 <http://stein.cshl.org/boulder/>
471 combined_log_format_to_db
473 <http://httpd.apache.org/docs/2.0/logs.html>
474 html_table_to_db
476 HTML tables to fsdb (assuming they're reasonably formatted).
477 kitrace_to_db
479 <http://ficus-www.cs.ucla.edu/ficus-members/geoff/kitrace.html>
480 ns_to_db
482 <http://mash-www.cs.berkeley.edu/ns/>
483 sqlselect_to_db
485 the output of SQL SELECT tables to db
486 tabdelim_to_db
488 spreadsheet tab-delimited files to db
489 tcpdump_to_db
491 (see man tcpdump(8) on any reasonable system)
492 xml_to_db
494 XML input to fsdb, assuming they're very regular
495 (And out of fsdb:)
497 db_to_csv
499 Comma-separated-value format from fsdb.
500 db_to_html_table
502 simple conversion of Fsdb to html tables
503 STANDARD OPTIONS
505 Many programs have common options:
506 -? or --help
508 Show basic usage.
509 -N on --new-name
511 When a command creates a new column like dbrowaccumulate's "accum",
512 this option lets one override the default name of that new column.
513 -T TmpDir
515 where to put tmp files. Also uses environment variable TMPDIR, if
516 -T is not specified. Default is /tmp.
517 Show basic usage.
519 -c FRACTION or --confidence FRACTION
521 Specify confidence interval FRACTION (dbcolstats, dbmultistats,
522 etc.)
523 -C S or "--element-separator S"
525 Specify column separator S (dbcolsplittocols, dbcolmerge).
526 -d or --debug
528 Enable debugging (may be repeated for greater effect in some
529 cases).
530 -a or --include-non-numeric
532 Compute stats over all data (treating non-numbers as zeros). (By
533 default, things that can't be treated as numbers are ignored for
534 stats purposes)
535 -S or --pre-sorted
537 Assume the data is pre-sorted. May be repeated to disable
538 verification (saving a small amount of work).
539 -e E or --empty E
541 give value E as the value for empty (null) records
542 -i I or --input I
544 Input data from file I.
545 -o O or --output O
547 Write data out to file O.
548 --header H
550 Use H as the full Fsdb header, rather than reading a header from
551 then input. This option is particularly useful when using Fsdb
552 under Hadoop, where split files don't have heades.
553 --nolog.
555 Skip logging the program in a trailing comment.
556 When giving Perl code (in dbrow and dbroweval) column names can be
558 embedded if preceded by underscores. Look at dbrow(1) or dbroweval(1)
559 for examples.)
560 Most programs run in constant memory and use temporary files if
562 necessary. Exceptions are dbcolneaten, dbcolpercentile, dbmapreduce,
563 dbmultistats, dbrowsplituniq.
564 STANDARD SORTING OPTIONS
566 A number of programs do sorting, or depend on defining an ordering of
567 rows. Such programs use these standard sorting options:
568 -r or --descending
570 sort in reverse order (high to low)
571 -R or --ascending
573 sort in normal order (low to high)
574 -t or --type-inferred-sorting
576 sort fields by type (numeric or leicographic), automatically
577 -n or --numeric
579 sort numerically
580 -N or --lexical
582 sort lexicographically
583
585 Take the raw data in "DATA/http_bandwidth", put a header on it
586 ("dbcoldefine size bw"), took statistics of each category
587 ("dbmultistats -k size bw"), pick out the relevant fields ("dbcol size
588 mean stddev pct_rsd"), and you get:
589 #fsdb size mean stddev pct_rsd
591 1024 1.4962e+06 2.8497e+05 19.047
592 10240 5.0286e+06 6.0103e+05 11.952
593 102400 4.9216e+06 3.0939e+05 6.2863
594 # | dbcoldefine size bw
595 # | /home/johnh/BIN/DB/dbmultistats -k size bw
596 # | /home/johnh/BIN/DB/dbcol size mean stddev pct_rsd
597 (The whole command was:
599 cat DATA/http_bandwidth |
601 dbcoldefine size |
602 dbmultistats -k size bw |
603 dbcol size mean stddev pct_rsd
604 all on one line.)
606 Then post-process them to get rid of the exponential notation by adding
608 this to the end of the pipeline:
609 dbroweval '_mean = sprintf("%8.0f", _mean); _stddev = sprintf("%8.0f", _stddev);'
611 (Actually, this step is no longer required since dbcolstats now uses a
613 different default format.)
614 giving:
616 #fsdb size mean stddev pct_rsd
618 1024 1496200 284970 19.047
619 10240 5028600 601030 11.952
620 102400 4921600 309390 6.2863
621 # | dbcoldefine size bw
622 # | dbmultistats -k size bw
623 # | dbcol size mean stddev pct_rsd
624 # | dbroweval { _mean = sprintf("%8.0f", _mean); _stddev = sprintf("%8.0f", _stddev); }
625 In a few lines, raw data is transformed to processed output.
627 Suppose you expect there is an odd distribution of results of one
629 datapoint. Fsdb can easily produce a CDF (cumulative distribution
630 function) of the data, suitable for graphing:
631 cat DB/DATA/http_bandwidth | \
633 dbcoldefine size bw | \
634 dbrow '_size == 102400' | \
635 dbcol bw | \
636 dbsort -n bw | \
637 dbrowenumerate | \
638 dbcolpercentile count | \
639 dbcol bw percentile | \
640 xgraph
641 The steps, roughly: 1. get the raw input data and turn it into fsdb
643 format, 2. pick out just the relevant column (for efficiency) and sort
644 it, 3. for each data point, assign a CDF percentage to it, 4. pick out
645 the two columns to graph and show them
646
648 The first commercial program I wrote was a gradebook, so here's how to
649 do it with Fsdb.
650 Format your data like DATA/grades.
652 #fsdb name email id test1
654 a a@ucla.example.edu 1 80
655 b b@usc.example.edu 2 70
656 c c@isi.example.edu 3 65
657 d d@lmu.example.edu 4 90
658 e e@caltech.example.edu 5 70
659 f f@oxy.example.edu 6 90
660 Or if your students have spaces in their names, use "-F S" and two
662 spaces to separate each column:
663 #fsdb -F S name email id test1
665 alfred aho a@ucla.example.edu 1 80
666 butler lampson b@usc.example.edu 2 70
667 david clark c@isi.example.edu 3 65
668 constantine drovolis d@lmu.example.edu 4 90
669 debrorah estrin e@caltech.example.edu 5 70
670 sally floyd f@oxy.example.edu 6 90
671 To compute statistics on an exam, do
673 cat DATA/grades | dbstats test1 |dblistize
675 giving
677 #fsdb -R C ...
679 mean: 77.5
680 stddev: 10.84
681 pct_rsd: 13.987
682 conf_range: 11.377
683 conf_low: 66.123
684 conf_high: 88.877
685 conf_pct: 0.95
686 sum: 465
687 sum_squared: 36625
688 min: 65
689 max: 90
690 n: 6
691 ...
692 To do a histogram:
694 cat DATA/grades | dbcolhisto -n 5 -g test1
696 giving
698 #fsdb low histogram
700 65 *
701 70 **
702 75
703 80 *
704 85
705 90 **
706 # | /home/johnh/BIN/DB/dbhistogram -n 5 -g test1
707 Now you want to send out grades to the students by e-mail. Create a
709 form-letter (in the file test1.txt):
710 To: _email (_name)
712 From: J. Random Professor <jrp@usc.example.edu>
713 Subject: test1 scores
714 _name, your score on test1 was _test1.
716 86+ A
717 75-85 B
718 70-74 C
719 0-69 F
720 Generate the shell script that will send the mail out:
722 cat DATA/grades | dbformmail test1.txt > test1.sh
724 And run it:
726 sh <test1.sh
728 The last two steps can be combined:
730 cat DATA/grades | dbformmail test1.txt | sh
732 but I like to keep a copy of exactly what I send.
734 At the end of the semester you'll want to compute grade totals and
736 assign letter grades. Both fall out of dbroweval. For example, to
737 compute weighted total grades with a 40% midterm/60% final where the
738 midterm is 84 possible points and the final 100:
739 dbcol -rv total |
741 dbcolcreate total - |
742 dbroweval '
743 _total = .40 * _midterm/84.0 + .60 * _final/100.0;
744 _total = sprintf("%4.2f", _total);
745 if (_final eq "-" || ( _name =~ /^_/)) { _total = "-"; };' |
746 dbcolneaten
747 If you got the data originally from a spreadsheet, save it in "tab-
749 delimited" format and convert it with tabdelim_to_db (run
750 tabdelim_to_db -? for examples).
751
753 To convert the Unix password file to db:
754 cat /etc/passwd | sed 's/:/ /g'| \
756 dbcoldefine -F S login password uid gid gecos home shell \
757 >passwd.fsdb
758 To convert the group file
760 cat /etc/group | sed 's/:/ /g' | \
762 dbcoldefine -F S group password gid members \
763 >group.fsdb
764 To show the names of the groups that div7-members are in (assuming DIV7
766 is in the gecos field):
767 cat passwd.fsdb | dbrow '_gecos =~ /DIV7/' | dbcol login gid | \
769 dbjoin -i - -i group.fsdb gid | dbcol login group
770
772 Which Fsdb programs are the most complicated (based on number of test
773 cases)?
774 ls TEST/*.cmd | \
776 dbcoldefine test | \
777 dbroweval '_test =~ s@^TEST/([^_]+).*$@$1@' | \
778 dbrowuniq -c | \
779 dbsort -nr count | \
780 dbcolneaten
781 (Answer: dbmapreduce, then dbcolstats, dbfilealter and dbjoin.)
783 Stats on an exam (in $FILE, where $COLUMN is the name of the exam)?
785 cat $FILE | dbcolstats -q 4 $COLUMN <$FILE | dblistize | dbstripcomments
787 cat $FILE | dbcolhisto -g -n 20 $COLUMN | dbcolneaten | dbstripcomments
789 Merging a the hw1 column from file hw1.fsdb into grades.fsdb assuming
791 there's a common student id in column "id":
792 dbcol id hw1 <hw1.fsdb >t.fsdb
794 dbjoin -a -e - grades.fsdb t.fsdb id | \
796 dbsort name | \
797 dbcolneaten >new_grades.fsdb
798 Merging two fsdb files with the same rows:
800 cat file1.fsdb file2.fsdb >output.fsdb
802 or if you want to clean things up a bit
804 cat file1.fsdb file2.fsdb | dbstripextraheaders >output.fsdb
806 or if you want to know where the data came from
808 for i in 1 2
810 do
811 dbcolcreate source $i < file$i.fsdb
812 done >output.fsdb
813 (assumes you're using a Bourne-shell compatible shell, not csh).
815
817 As with any tool, one should (which means must) understand the limits
818 of the tool.
819 All Fsdb tools should run in constant memory. In some cases (such as
821 dbcolstats with quartiles, where the whole input must be re-read),
822 programs will spool data to disk if necessary.
823 Most tools buffer one or a few lines of data, so memory will scale with
825 the size of each line. (So lines with many columns, or when columns
826 have lots data, may cause large memory consumption.)
827 All Fsdb tools should run in constant or at worst "n log n" time.
829 All Fsdb tools use normal Perl math routines for computation. Although
831 I make every attempt to choose numerically stable algorithms (although
832 I also welcome feedback and suggestions for improvement), normal
833 rounding due to computer floating point approximations can result in
834 inaccuracies when data spans a large range of precision. (See for
835 example the dbcolstats_extrema test cases.)
836 Any requirements and limitations of each Fsdb tool is documented on its
838 manual page.
839 If any Fsdb program violates these assumptions, that is a bug that
841 should be documented on the tool's manual page or ideally fixed.
842 Fsdb does depend on Perl's correctness, and Perl (and Fsdb) have some
844 bugs. Fsdb should work on perl from version 5.10 onward.
845
847 There have been four major versions of Fsdb: fsdb-0.x was begun in 1991
848 for my personal use. Fsdb 1.0 is a complete re-write of the pre-1995
849 versions, and was distributed from 1995 to 2007. Fsdb 2.0 is a
850 significant re-write of the 1.x versions to systematically use a
851 library and threads (although threads were abandoned in 2.44). Fsdb
852 3.0 in 2022 adds type specifiers to the schema, mostly to support use
853 in languages with stronger typing (like Python, Go, and C).
854 Fsdb (in its various forms) has been used extensively by its author
856 since 1991. Since 1995 it's been used by two other researchers at UCLA
857 and several at ISI. In February 1998 it was announced to the Internet.
858 Since then it has found a few users, some outside where I work.
859 Major changes:
861 1.0 1997-07-22: first public release.
863 2.0 2008-01-25: rewrite to use a common library, and starting to use
864 threads.
865 2.12 2008-10-16: completion of the rewrite, and first RPM package.
866 2.44 2013-10-02: abandoning threads for improved performance
867 3.0 2022-04-04: adding type specifiers to the schema
868 Fsdb 2.0 Rationale
870 I've thought about fsdb-2.0 for many years, but it was started in
871 earnest in 2007. Fsdb-2.0 has the following goals:
872 in-one-process processing
874 While fsdb is great on the Unix command line as a pipeline between
875 programs, it should also be possible to set it up to run in a
876 single process. And if it does so, it should be able to avoid
877 serializing and deserializing (converting to and from text) data
878 between each module. (Accomplished in fsdb-2.0: see dbpipeline,
879 although still needs tuning.)
880 clean IO API
882 Fsdb's roots go back to perl4 and 1991, so the fsdb-1.x library is
883 very, very crufty. More than just being ugly (but it was that
884 too), this made things reading from one format file and writing to
885 another the application's job, when it should be the library's.
886 (Accomplished in fsdb-1.15 and improved in 2.0: see Fsdb::IO.)
887 normalized module APIs
889 Because fsdb modules were added as needed over 10 years, sometimes
890 the module APIs became inconsistent. (For example, the 1.x
891 "dbcolcreate" required an empty value following the name of the new
892 column, but other programs specify empty values with the "-e"
893 argument.) We should smooth over these inconsistencies.
894 (Accomplished as each module was ported in 2.0 through 2.7.)
895 everyone handles all input formats
897 Given a clean IO API, the distinction between "colized" and
898 "listized" fsdb files should go away. Any program should be able
899 to read and write files in any format. (Accomplished in fsdb-2.1.)
900 Fsdb-2.0 preserves backwards compatibility where possible, but breaks
902 it where necessary to accomplish the above goals. In August 2008,
903 Fsdb-2.7 was declared preferred over the 1.x versions. Benchmarking in
904 2013 showed that threading performed much worse than just using pipes,
905 so Fsdb-2.44 uses threading "style", but implemented with processes
906 (via my "Freds" library).
907 Contributors
909 Fsdb includes code ported from Geoff Kuenning
910 ("Fsdb::Support::TDistribution").
911 Fsdb contributors: Ashvin Goel goel@cse.oge.edu, Geoff Kuenning
913 geoff@fmg.cs.ucla.edu, Vikram Visweswariah visweswa@isi.edu, Kannan
914 Varadahan kannan@isi.edu, Lars Eggert larse@isi.edu, Arkadi Gelfond
915 arkadig@dyna.com, David Graff graff@ldc.upenn.edu, Haobo Yu
916 haoboy@packetdesign.com, Pavlin Radoslavov pavlin@catarina.usc.edu,
917 Graham Phillips, Yuri Pradkin, Alefiya Hussain, Ya Xu, Michael
918 Schwendt, Fabio Silva fabio@isi.edu, Jerry Zhao zhaoy@isi.edu, Ning Xu
919 nxu@aludra.usc.edu, Martin Lukac mlukac@lecs.cs.ucla.edu, Xue Cai,
920 Michael McQuaid, Christopher Meng, Calvin Ardi, H. Merijn Brand, Lan
921 Wei, Hang Guo, Wes Hardaker.
922 Fsdb includes datasets contributed from NIST (DATA/nist_zarr13.fsdb),
924 from
925 <http://www.itl.nist.gov/div898/handbook/eda/section4/eda4281.htm>, the
926 NIST/SEMATECH e-Handbook of Statistical Methods, section 1.4.2.8.1.
927 Background and Data. The source is public domain, and reproduced with
928 permission.
929
931 As stated in the introduction, Fsdb is an incompatible reimplementation
932 of the ideas found in "/rdb". By storing data in simple text files and
933 processing it with pipelines it is easy to experiment (in the shell)
934 and look at the output. The original implementation of this idea was
935 /rdb, a commercial product described in the book UNIX relational
936 database management: application development in the UNIX environment by
937 Rod Manis, Evan Schaffer, and Robert Jorgensen (and also at the web
938 page <http://www.rdb.com/>).
939 While Fsdb is inspired by Rdb, it includes no code from it, and Fsdb
941 makes several different design choices. In particular: rdb attempts to
942 be closer to a "real" database, with provision for locking, file
943 indexing. Fsdb focuses on single user use and so eschews these
944 choices. Rdb also has some support for interactive editing. Fsdb
945 leaves editing to text editors like emacs or vi.
946 In August, 2002 I found out Carlo Strozzi extended RDB with his package
948 NoSQL <http://www.linux.it/~carlos/nosql/>. According to Mr. Strozzi,
949 he implemented NoSQL in awk to avoid the Perl start-up of RDB.
950 Although I haven't found Perl startup overhead to be a big problem on
951 my platforms (from old Sparcstation IPCs to 2GHz Pentium-4s), you may
952 want to evaluate his system. The Linux Journal has a description of
953 NoSQL at <http://www.linuxjournal.com/article/3294>. It seems quite
954 similar to Fsdb. Like /rdb, NoSQL supports indexing (not present in
955 Fsdb). Fsdb appears to have richer support for statistics, and, as of
956 Fsdb-2.x, its support for Perl threading may support faster performance
957 (one-process, less serialization and deserialization).
958
960 Versions prior to 1.0 were released informally on my web page but were
961 not announced.
962 0.0 1991
964 started for my own research use
965 0.1 26-May-94
967 first check-in to RCS
968 0.2 15-Mar-95
970 parts now require perl5
971 1.0, 22-Jul-97
973 adds autoconf support and a test script.
974 1.1, 20-Jan-98
976 support for double space field separators, better tests
977 1.2, 11-Feb-98
979 minor changes and release on comp.lang.perl.announce
980 1.3, 17-Mar-98
982 • adds median and quartile options to dbstats
983 • adds dmalloc_to_db converter
985 • fixes some warnings
987 • dbjoin now can run on unsorted input
989 • fixes a dbjoin bug
991 • some more tests in the test suite
993 1.4, 27-Mar-98
995 • improves error messages (all should now report the program that
996 makes the error)
997 • fixed a bug in dbstats output when the mean is zero
999 1.5, 25-Jun-98
1001 BUG FIX dbcolhisto, dbcolpercentile now handles non-numeric values like
1002 dbstats
1003 NEW dbcolstats computes zscores and tscores over a column
1004 NEW dbcolscorrelate computes correlation coefficients between two
1005 columns
1006 INTERNAL ficus_getopt.pl has been replaced by DbGetopt.pm
1007 BUG FIX all tests are now ``portable'' (previously some tests ran only
1008 on my system)
1009 BUG FIX you no longer need to have the db programs in your path (fix
1010 arose from a discussion with Arkadi Gelfond)
1011 BUG FIX installation no longer uses cp -f (to work on SunOS 4)
1012 1.6, 24-May-99
1014 NEW dbsort, dbstats, dbmultistats now run in constant memory (using tmp
1015 files if necessary)
1016 NEW dbcolmovingstats does moving means over a series of data
1017 NEW dbcol has a -v option to get all columns except those listed
1018 NEW dbmultistats does quartiles and medians
1019 NEW dbstripextraheaders now also cleans up bogus comments before the
1020 fist header
1021 BUG FIX dbcolneaten works better with double-space-separated data
1022 1.7, 5-Jan-00
1024 NEW dbcolize now detects and rejects lines that contain embedded copies
1025 of the field separator
1026 NEW configure tries harder to prevent people from improperly
1027 configuring/installing fsdb
1028 NEW tcpdump_to_db converter (incomplete)
1029 NEW tabdelim_to_db converter: from spreadsheet tab-delimited files to
1030 db
1031 NEW mailing lists for fsdb are "fsdb-announce@heidemann.la.ca.us"
1032 and "fsdb-talk@heidemann.la.ca.us"
1033 To subscribe to either, send mail
1034 to "fsdb-announce-request@heidemann.la.ca.us" or
1035 "fsdb-talk-request@heidemann.la.ca.us" with "subscribe" in the
1036 BODY of the message.
1037 BUG FIX dbjoin used to produce incorrect output if there were extra,
1039 unmatched values in the 2nd table. Thanks to Graham Phillips for
1040 providing a test case.
1041 BUG FIX the sample commands in the usage strings now all should
1042 explicitly include the source of data (typically from "cat foo.fsdb
1043 |"). Thanks to Ya Xu for pointing out this documentation deficiency.
1044 BUG FIX (DOCUMENTATION) dbcolmovingstats had incorrect sample output.
1045 1.8, 28-Jun-00
1047 BUG FIX header options are now preserved when writing with dblistize
1048 NEW dbrowuniq now optionally checks for uniqueness only on certain
1049 fields
1050 NEW dbrowsplituniq makes one pass through a file and splits it into
1051 separate files based on the given fields
1052 NEW converter for "crl" format network traces
1053 NEW anywhere you use arbitrary code (like dbroweval), _last_foo now
1054 maps to the last row's value for field _foo.
1055 OPTIMIZATION comment processing slightly changed so that dbmultistats
1056 now is much faster on files with lots of comments (for example, ~100k
1057 lines of comments and 700 lines of data!) (Thanks to Graham Phillips
1058 for pointing out this performance problem.)
1059 BUG FIX dbstats with median/quartiles now correctly handles singleton
1060 data points.
1061 1.9, 6-Nov-00
1063 NEW dbfilesplit, split a single input file into multiple output files
1064 (based on code contributed by Pavlin Radoslavov).
1065 BUG FIX dbsort now works with perl-5.6
1066 1.10, 10-Apr-01
1068 BUG FIX dbstats now handles the case where there are more n-tiles than
1069 data
1070 NEW dbstats now includes a -S option to optimize work on pre-sorted
1071 data (inspired by code contributed by Haobo Yu)
1072 BUG FIX dbsort now has a better estimate of memory usage when run on
1073 data with very short records (problem detected by Haobo Yu)
1074 BUG FIX cleanup of temporary files is slightly better
1075 1.11, 2-Nov-01
1077 BUG FIX dbcolneaten now runs in constant memory
1078 NEW dbcolneaten now supports "field specifiers" that allow some control
1079 over how wide columns should be
1080 OPTIMIZATION dbsort now tries hard to be filesystem cache-friendly
1081 (inspired by "Information and Control in Gray-box Systems" by the
1082 Arpaci-Dusseau's at SOSP 2001)
1083 INTERNAL t_distr now ported to perl5 module DbTDistr
1084 1.12, 30-Oct-02
1086 BUG FIX dbmultistats documentation typo fixed
1087 NEW dbcolmultiscale
1088 NEW dbcol has -r option for "relaxed error checking"
1089 NEW dbcolneaten has new -e option to strip end-of-line spaces
1090 NEW dbrow finally has a -v option to negate the test
1091 BUG FIX math bug in dbcoldiff fixed by Ashvin Goel (need to check
1092 Scheaffer test cases)
1093 BUG FIX some patches to run with Perl 5.8. Note: some programs
1094 (dbcolmultiscale, dbmultistats, dbrowsplituniq) generate warnings like:
1095 "Use of uninitialized value in concatenation (.)" or "string at
1096 /usr/lib/perl5/5.8.0/FileCache.pm line 98, <STDIN> line 2". Please
1097 ignore this until I figure out how to suppress it. (Thanks to Jerry
1098 Zhao for noticing perl-5.8 problems.)
1099 BUG FIX fixed an autoconf problem where configure would fail to find a
1100 reasonable prefix (thanks to Fabio Silva for reporting the problem)
1101 NEW db_to_html_table: simple conversion to html tables (NO fancy stuff)
1102 NEW dblib now has a function dblib_text2html() that will do simple
1103 conversion of iso-8859-1 to HTML
1104 1.13, 4-Feb-04
1106 NEW fsdb added to the freebsd ports tree
1107 <http://www.freshports.org/databases/fsdb/>. Maintainer:
1108 "larse@isi.edu"
1109 BUG FIX properly handle trailing spaces when data must be numeric (ex.
1110 dbstats with -FS, see test dbstats_trailing_spaces). Fix from Ning Xu
1111 "nxu@aludra.usc.edu".
1112 NEW dbcolize error message improved (bug report from Terrence Brannon),
1113 and list format documented in the README.
1114 NEW cgi_to_db converts CGI.pm-format storage to fsdb list format
1115 BUG FIX handle numeric synonyms for column names in dbcol properly
1116 ENHANCEMENT "talking about columns" section added to README. Lack of
1117 documentation pointed out by Lars Eggert.
1118 CHANGE dbformmail now defaults to using Mail ("Berkeley Mail") to send
1119 mail, rather than sendmail (sendmail is still an option, but mail
1120 doesn't require running as root)
1121 NEW on platforms that support it (i.e., with perl 5.8), fsdb works fine
1122 with unicode
1123 NEW dbfilevalidate: check a db file for some common errors
1124 1.14, 24-Aug-06
1126 ENHANCEMENT README cleanup
1127 INCOMPATIBLE CHANGE dbcolsplit renamed dbcolsplittocols
1128 NEW dbcolsplittorows split one column into multiple rows
1129 NEW dbcolsregression compute linear regression and correlation for two
1130 columns
1131 ENHANCEMENT cvs_to_db: better error handling, normalize field names,
1132 skip blank lines
1133 ENHANCEMENT dbjoin now detects (and fails) if non-joined files have
1134 duplicate names
1135 BUG FIX minor bug fixed in calculation of Student t-distributions
1136 (doesn't change any test output, but may have caused small errors)
1137 1.15, 12-Nov-07
1139 NEW fsdb-1.14 added to the MacOS Fink system
1140 <http://pdb.finkproject.org/pdb/package.php/fsdb>. (Thanks to Lars
1141 Eggert for maintaining this port.)
1142 NEW Fsdb::IO::Reader and Fsdb::IO::Writer now provide reasonably clean
1143 OO I/O interfaces to Fsdb files. Highly recommended if you use fsdb
1144 directly from perl. In the fullness of time I expect to reimplement
1145 the entire thing using these APIs to replace the current dblib.pl which
1146 is still hobbled by its roots in perl4.
1147 NEW dbmapreduce now implements a Google-style map/reduce abstraction,
1148 generalizing dbmultistats.
1149 ENHANCEMENT fsdb now uses the Perl build system (Makefile.PL, etc.),
1150 instead of autoconf. This change paves the way to better perl-5-style
1151 modularization, proper manual pages, input of both listize and colize
1152 format for every program, and world peace.
1153 ENHANCEMENT dblib.pl is now moved to Fsdb::Old.pm.
1154 BUG FIX dbmultistats now propagates its format argument (-f). Bug and
1155 fix from Martin Lukac (thanks!).
1156 ENHANCEMENT dbformmail documentation now is clearer that it doesn't
1157 send the mail, you have to run the shell script it writes. (Problem
1158 observed by Unkyu Park.)
1159 ENHANCEMENT adapted to autoconf-2.61 (and then these changes were
1160 discarded in favor of The Perl Way.
1161 BUG FIX dbmultistats memory usage corrected (O(# tags), not O(1))
1162 ENHANCEMENT dbmultistats can now optionally run with pre-grouped input
1163 in O(1) memory
1164 ENHANCEMENT dbroweval -N was finally implemented (eat comments)
1165 2.0, 25-Jan-08
1167 2.0, 25-Jan-08 --- a quiet 2.0 release (gearing up towards complete)
1168 ENHANCEMENT: shifting old programs to Perl modules, with the front-end
1170 program as just a wrapper. In the short-term, this change just means
1171 programs have real man pages. In the long-run, it will mean that one
1172 can run a pipeline in a single Perl program. So far: dbcol, dbroweval,
1173 the new dbrowcount. dbsort the new dbmerge, the old "dbstats" (renamed
1174 dbcolstats), dbcolrename, dbcolcreate,
1175 NEW: Fsdb::Filter::dbpipeline is an internal-only module that lets one
1176 use fsdb commands from within perl (via threads).
1177 It also provides perl function aliases for the internal modules, so
1178 a string of fsdb commands in perl are nearly as terse as in the
1179 shell:
1180 use Fsdb::Filter::dbpipeline qw(:all);
1182 dbpipeline(
1183 dbrow(qw(name test1)),
1184 dbroweval('_test1 += 5;')
1185 );
1186 INCOMPATIBLE CHANGE: The old dbcolstats has been renamed
1188 dbcolstatscores. The new dbcolstats does the same thing as the old
1189 dbstats. This incompatibility is unfortunate but normalizes program
1190 names.
1191 CHANGE: The new dbcolstats program always outputs "-" (the default
1192 empty value) for statistics it cannot compute (for example, standard
1193 deviation if there is only one row), instead of the old mix of "-" and
1194 "na".
1195 INCOMPATIBLE CHANGE: The old dbcolstats program, now called
1196 dbcolstatscores, also has different arguments. The "-t mean,stddev"
1197 option is now "--tmean mean --tstddev stddev". See dbcolstatscores for
1198 details.
1199 INCOMPATIBLE CHANGE: dbcolcreate now assumes all new columns get the
1200 default value rather than requiring each column to have an initial
1201 constant value. To change the initial value, sue the new "-e" option.
1202 NEW: dbrowcount counts rows, an almost-subset of dbcolstats's "n"
1203 output (except without differentiating numeric/non-numeric input), or
1204 the equivalent of "dbstripcomments | wc -l".
1205 NEW: dbmerge merges two sorted files. This functionality was previously
1206 embedded in dbsort.
1207 INCOMPATIBLE CHANGE: dbjoin's "-i" option to include non-matches is now
1208 renamed "-a", so as to not conflict with the new standard option "-i"
1209 for input file.
1210 2.1, 6-Apr-08
1212 2.1, 6-Apr-08 --- another alpha 2.0, but now all converted programs
1213 understand both listize and colize format
1214 ENHANCEMENT: shifting more old programs to Perl modules. New in 2.1:
1216 dbcolneaten, dbcoldefine, dbcolhisto, dblistize, dbcolize, dbrecolize
1217 ENHANCEMENT dbmerge now handles an arbitrary number of input files, not
1218 just exactly two.
1219 NEW dbmerge2 is an internal routine that handles merging exactly two
1220 files.
1221 INCOMPATIBLE CHANGE dbjoin now specifies inputs like dbmerge2, rather
1222 than assuming the first two arguments were tables (as in fsdb-1).
1223 The old dbjoin argument "-i" is now "-a" or <--type=outer>.
1224 A minor change: comments in the source files for dbjoin are now
1226 intermixed with output rather than being delayed until the end.
1227 ENHANCEMENT dbsort now no longer produces warnings when null values are
1229 passed to numeric comparisons.
1230 BUG FIX dbroweval now once again works with code that lacks a trailing
1231 semicolon. (This bug fixes a regression from 1.15.)
1232 INCOMPATIBLE CHANGE dbcolneaten's old "-e" option (to avoid end-of-line
1233 spaces) is now "-E" to avoid conflicts with the standard empty field
1234 argument.
1235 INCOMPATIBLE CHANGE dbcolhisto's old "-e" option is now "-E" to avoid
1236 conflicts. And its "-n", "-s", and "-w" are now "-N", "-S", and "-W" to
1237 correspond.
1238 NEW dbfilealter replaces dbrecolize, dblistize, and dbcolize, but with
1239 different options.
1240 ENHANCEMENT The library routines "Fsdb::IO" now understand both list-
1241 format and column-format data, so all converted programs can now
1242 automatically read either format. This capability was one of the
1243 milestone goals for 2.0, so yea!
1244 2.2, 23-May-08
1246 Release 2.2 is another 2.x alpha release. Now most of the commands are
1247 ported, but a few remain, and I plan one last incompatible change (to
1248 the file header) before 2.x final.
1249 ENHANCEMENT
1251 shifting more old programs to Perl modules. New in 2.2:
1252 dbrowaccumulate, dbformmail. dbcolmovingstats. dbrowuniq.
1253 dbrowdiff. dbcolmerge. dbcolsplittocols. dbcolsplittorows.
1254 dbmapreduce. dbmultistats. dbrvstatdiff. Also dbrowenumerate
1255 exists only as a front-end (command-line) program.
1256 INCOMPATIBLE CHANGE
1258 The following programs have been dropped from fsdb-2.x:
1259 dbcoltighten, dbfilesplit, dbstripextraheaders,
1260 dbstripleadingspace.
1261 NEW combined_log_format_to_db to convert Apache logfiles
1263 INCOMPATIBLE CHANGE
1265 Options to dbrowdiff are now -B and -I, not -a and -i.
1266 INCOMPATIBLE CHANGE
1268 dbstripcomments is now dbfilestripcomments.
1269 BUG FIXES
1271 dbcolneaten better handles empty columns; dbcolhisto warning
1272 suppressed (actually a bug in high-bucket handling).
1273 INCOMPATIBLE CHANGE
1275 dbmultistats now requires a "-k" option in front of the key (tag)
1276 field, or if none is given, it will group by the first field (both
1277 like dbmapreduce).
1278 KNOWN BUG
1280 dbmultistats with quantile option doesn't work currently.
1281 INCOMPATIBLE CHANGE
1283 dbcoldiff is renamed dbrvstatdiff.
1284 BUG FIXES
1286 dbformmail was leaving its log message as a command, not a
1287 comment. Oops. No longer.
1288 2.3, 27-May-08 (alpha)
1290 Another alpha release, this one just to fix the critical dbjoin bug
1291 listed below (that happens to have blocked my MP3 jukebox :-).
1292 BUG FIX
1294 Dbsort no longer hangs if given an input file with no rows.
1295 BUG FIX
1297 Dbjoin now works with unsorted input coming from a pipeline (like
1298 stdin). Perl-5.8.8 has a bug (?) that was making this case
1299 fail---opening stdin in one thread, reading some, then reading more
1300 in a different thread caused an lseek which works on files, but
1301 fails on pipes like stdin. Go figure.
1302 BUG FIX / KNOWN BUG
1304 The dbjoin fix also fixed dbmultistats -q (it now gives the right
1305 answer). Although a new bug appeared, messages like:
1306 Attempt to free unreferenced scalar: SV 0xa9dd0c4, Perl
1307 interpreter: 0xa8350b8 during global destruction. So the
1308 dbmultistats_quartile test is still disabled.
1309 2.4, 18-Jun-08
1311 Another alpha release, mostly to fix minor usability problems in
1312 dbmapreduce and client functions.
1313 ENHANCEMENT
1315 dbrow now defaults to running user supplied code without warnings
1316 (as with fsdb-1.x). Use "--warnings" or "-w" to turn them back on.
1317 ENHANCEMENT
1319 dbroweval can now write different format output than the input,
1320 using the "-m" option.
1321 KNOWN BUG
1323 dbmapreduce emits warnings on perl 5.10.0 about "Unbalanced string
1324 table refcount" and "Scalars leaked" when run with an external
1325 program as a reducer.
1326 dbmultistats emits the warning "Attempt to free unreferenced
1328 scalar" when run with quartiles.
1329 In each case the output is correct. I believe these can be
1331 ignored.
1332 CHANGE
1334 dbmapreduce no longer logs a line for each reducer that is invoked.
1335 2.5, 24-Jun-08
1337 Another alpha release, fixing more minor bugs in "dbmapreduce" and
1338 lossage in "Fsdb::IO".
1339 ENHANCEMENT
1341 dbmapreduce can now tolerate non-map-aware reducers that pass back
1342 the key column in put. It also passes the current key as the last
1343 argument to external reducers.
1344 BUG FIX
1346 Fsdb::IO::Reader, correctly handle "-header" option again. (Broken
1347 since fsdb-2.3.)
1348 2.6, 11-Jul-08
1350 Another alpha release, needed to fix DaGronk. One new port, small bug
1351 fixes, and important fix to dbmapreduce.
1352 ENHANCEMENT
1354 shifting more old programs to Perl modules. New in 2.2:
1355 dbcolpercentile.
1356 INCOMPATIBLE CHANGE and ENHANCEMENTS dbcolpercentile arguments changed,
1358 use "--rank" to require ranking instead of "-r". Also, "--ascending"
1359 and "--descending" can now be specified separately, both for
1360 "--percentile" and "--rank".
1361 BUG FIX
1362 Sigh, the sense of the --warnings option in dbrow was inverted. No
1363 longer.
1364 BUG FIX
1366 I found and fixed the string leaks (errors like "Unbalanced string
1367 table refcount" and "Scalars leaked") in dbmapreduce and
1368 dbmultistats. (All "IO::Handle"s in threads must be manually
1369 destroyed.)
1370 BUG FIX
1372 The "-C" option to specify the column separator in dbcolsplittorows
1373 now works again (broken since it was ported).
1374 2.7, 30-Jul-08 beta
1376 The beta release of fsdb-2.x. Finally, all programs are ported. As
1378 statistics, the number of lines of non-library code doubled from 7.5k
1379 to 15.5k. The libraries are much more complete, going from 866 to 5164
1380 lines. The overall number of programs is about the same, although 19
1381 were dropped and 11 were added. The number of test cases has grown
1382 from 116 to 175. All programs are now in perl-5, no more shell scripts
1383 or perl-4. All programs now have manual pages.
1384 Although this is a major step forward, I still expect to rename "jdb"
1386 to "fsdb".
1387 ENHANCEMENT
1389 shifting more old programs to Perl modules. New in 2.7:
1390 dbcolscorellate. dbcolsregression. cgi_to_db. dbfilevalidate.
1391 db_to_csv. csv_to_db, db_to_html_table, kitrace_to_db,
1392 tcpdump_to_db, tabdelim_to_db, ns_to_db.
1393 INCOMPATIBLE CHANGE
1395 The following programs have been dropped from fsdb-2.x: db2dcliff,
1396 dbcolmultiscale, crl_to_db. ipchain_logs_to_db. They may come
1397 back, but seemed overly specialized. The following program
1398 dbrowsplituniq was dropped because it is superseded by dbmapreduce.
1399 dmalloc_to_db was dropped pending a test cases and examples.
1400 ENHANCEMENT
1402 dbfilevalidate now has a "-c" option to correct errors.
1403 NEW html_table_to_db provides the inverse of db_to_html_table.
1405 2.8, 5-Aug-08
1407 Change header format, preserving forwards compatibility.
1408 BUG FIX
1410 Complete editing pass over the manual, making sure it aligns with
1411 fsdb-2.x.
1412 SEMI-COMPATIBLE CHANGE
1414 The header of fsdb files has changed, it is now #fsdb, not #h (or
1415 #L) and parsing of -F and -R are also different. See dbfilealter
1416 for the new specification. The v1 file format will be read,
1417 compatibly, but not written.
1418 BUG FIX
1420 dbmapreduce now tolerates comments that precede the first key,
1421 instead of failing with an error message.
1422 2.9, 6-Aug-08
1424 Still in beta; just a quick bug-fix for dbmapreduce.
1425 ENHANCEMENT
1427 dbmapreduce now generates plausible output when given no rows of
1428 input.
1429 2.10, 23-Sep-08
1431 Still in beta, but picking up some bug fixes.
1432 ENHANCEMENT
1434 dbmapreduce now generates plausible output when given no rows of
1435 input.
1436 ENHANCEMENT
1438 dbroweval the warnings option was backwards; now corrected. As a
1439 result, warnings in user code now default off (like in fsdb-1.x).
1440 BUG FIX
1442 dbcolpercentile now defaults to assuming the target column is
1443 numeric. The new option "-N" allows selection of a non-numeric
1444 target.
1445 BUG FIX
1447 dbcolscorrelate now includes "--sample" and "--nosample" options to
1448 compute the sample or full population correlation coefficients.
1449 Thanks to Xue Cai for finding this bug.
1450 2.11, 14-Oct-08
1452 Still in beta, but picking up some bug fixes.
1453 ENHANCEMENT
1455 html_table_to_db is now more aggressive about filling in empty
1456 cells with the official empty value, rather than leaving them blank
1457 or as whitespace.
1458 ENHANCEMENT
1460 dbpipeline now catches failures during pipeline element setup and
1461 exits reasonably gracefully.
1462 BUG FIX
1464 dbsubprocess now reaps child processes, thus avoiding running out
1465 of processes when used a lot.
1466 2.12, 16-Oct-08
1468 Finally, a full (non-beta) 2.x release!
1469 INCOMPATIBLE CHANGE
1471 Jdb has been renamed Fsdb, the flatfile-streaming database. This
1472 change affects all internal Perl APIs, but no shell command-level
1473 APIs. While Jdb served well for more than ten years, it is easily
1474 confused with the Java debugger (even though Jdb was there first!).
1475 It also is too generic to work well in web search engines.
1476 Finally, Jdb stands for ``John's database'', and we're a bit beyond
1477 that. (However, some call me the ``file-system guy'', so one could
1478 argue it retains that meeting.)
1479 If you just used the shell commands, this change should not affect
1481 you. If you used the Perl-level libraries directly in your code,
1482 you should be able to rename "Jdb" to "Fsdb" to move to 2.12.
1483 The jdb-announce list not yet been renamed, but it will be shortly.
1485 With this release I've accomplished everything I wanted to in
1487 fsdb-2.x. I therefore expect to return to boring, bugfix releases.
1488 2.13, 30-Oct-08
1490 BUG FIX
1491 dbrowaccumulate now treats non-numeric data as zero by default.
1492 BUG FIX
1494 Fixed a perl-5.10ism in dbmapreduce that breaks that program under
1495 5.8. Thanks to Martin Lukac for reporting the bug.
1496 2.14, 26-Nov-08
1498 BUG FIX
1499 Improved documentation for dbmapreduce's "-f" option.
1500 ENHANCEMENT
1502 dbcolmovingstats how computes a moving standard deviation in
1503 addition to a moving mean.
1504 2.15, 13-Apr-09
1506 BUG FIX
1507 Fix a make install bug reported by Shalindra Fernando.
1508 2.16, 14-Apr-09
1510 BUG FIX
1511 Another minor release bug: on some systems programize_module looses
1512 executable permissions. Again reported by Shalindra Fernando.
1513 2.17, 25-Jun-09
1515 TYPO FIXES
1516 Typo in the dbroweval manual fixed.
1517 IMPROVEMENT
1519 There is no longer a comment line to label columns in dbcolneaten,
1520 instead the header line is tweaked to line up. This change
1521 restores the Jdb-1.x behavior, and means that repeated runs of
1522 dbcolneaten no longer add comment lines each time.
1523 BUG FIX
1525 It turns out dbcolneaten was not correctly handling trailing
1526 spaces when given the "-E" option to suppress them. This
1527 regression is now fixed.
1528 EXTENSION
1530 dbroweval(1) can now handle direct references to the last row via
1531 $lfref, a dubious but now documented feature.
1532 BUG FIXES
1534 Separators set with "-C" in dbcolmerge and dbcolsplittocols were
1535 not properly setting the heading, and null fields were not
1536 recognized. The first bug was reported by Martin Lukac.
1537 2.18, 1-Jul-09 A minor release
1539 IMPROVEMENT
1540 Documentation for Fsdb::IO::Reader has been improved.
1541 IMPROVEMENT
1543 The package should now be PGP-signed.
1544 2.19, 10-Jul-09
1546 BUG FIX
1547 Internal improvements to debugging output and robustness of
1548 dbmapreduce and dbpipeline. TEST/dbpipeline_first_fails.cmd re-
1549 enabled.
1550 2.20, 30-Nov-09 (A collection of minor bugfixes, plus a build against
1552 Fedora 12.)
1553 BUG FIX
1554 Loging for dbmapreduce with code refs is now stable (it no longer
1555 includes a hex pointer to the code reference).
1556 BUG FIX
1558 Better handling of mixed blank lines in Fsdb::IO::Reader (see test
1559 case dbcolize_blank_lines.cmd).
1560 BUG FIX
1562 html_table_to_db now handles multi-line input better, and handles
1563 tables with COLSPAN.
1564 BUG FIX
1566 dbpipeline now cleans up threads in an "eval" to prevent "cannot
1567 detach a joined thread" errors that popped up in perl-5.10.
1568 Hopefully this prevents a race condition that causes the test
1569 suites to hang about 20% of the time (in dbpipeline_first_fails).
1570 IMPROVEMENT
1572 dbmapreduce now detects and correctly fails when the input and
1573 reducer have incompatible field separators.
1574 IMPROVEMENT
1576 dbcolstats, dbcolhisto, dbcolscorrelate, dbcolsregression, and
1577 dbrowcount now all take an "-F" option to let one specify the
1578 output field separator (so they work better with dbmapreduce).
1579 BUG FIX
1581 An omitted "-k" from the manual page of dbmultistats is now there.
1582 Bug reported by Unkyu Park.
1583 2.21, 17-Apr-10 bug fix release
1585 BUG FIX
1586 Fsdb::IO::Writer now no longer fails with -outputheader => never
1587 (an obscure bug).
1588 IMPROVEMENT
1590 Fsdb (in the warnings section) and dbcolstats now more carefully
1591 document how they handle (and do not handle) numerical precision
1592 problems, and other general limits. Thanks to Yuri Pradkin for
1593 prompting this documentation.
1594 IMPROVEMENT
1596 "Fsdb::Support::fullname_to_sortkey" is now restored from "Jdb".
1597 IMPROVEMENT
1599 Documention for multiple styles of input approaches (including
1600 performance description) added to Fsdb::IO.
1601 2.22, 2010-10-31 One new tool dbcolcopylast and several bug fixes for Perl
1603 5.10.
1604 BUG FIX
1605 dbmerge now correctly handles n-way merges. Bug reported by Yuri
1606 Pradkin.
1607 INCOMPARABLE CHANGE
1609 dbcolneaten now defaults to not padding the last column.
1610 ADDITION
1612 dbrowenumerate now takes -N NewColumn to give the new column a name
1613 other than "count". Feature requested by Mike Rouch in January
1614 2005.
1615 ADDITION
1617 New program dbcolcopylast copies the last value of a column into a
1618 new column copylast_column of the next row. New program requested
1619 by Fabio Silva; useful for converting dbmultistats output into
1620 dbrvstatdiff input.
1621 BUG FIX
1623 Several tools (particularly dbmapreduce and dbmultistats) would
1624 report errors like "Unbalanced string table refcount: (1) for
1625 "STDOUT" during global destruction" on exit, at least on certain
1626 versions of Perl (for me on 5.10.1), but similar errors have been
1627 off-and-on for several Perl releases. Although I think my code
1628 looked OK, I worked around this problem with a different way of
1629 handling standard IO redirection.
1630 2.23, 2011-03-10 Several small portability bugfixes; improved dbcolstats
1632 for large datasets
1633 IMPROVEMENT
1634 Documentation to dbrvstatdiff was changed to use "sd" to refer to
1635 standard deviation, not "ss" (which might be confused with sum-of-
1636 squares).
1637 BUG FIX
1639 This documentation about dbmultistats was missing the -k option in
1640 some cases.
1641 BUG FIX
1643 dbmapreduce was failing on MacOS-10.6.3 for some tests with the
1644 error
1645 dbmapreduce: cannot run external dbmapreduce reduce program (perl TEST/dbmapreduce_external_with_key.pl)
1647 The problem seemed to be only in the error, not in operation. On
1649 MacOS, the error is now suppressed. Thanks to Alefiya Hussain for
1650 providing access to a Mac system that allowed debugging of this
1651 problem.
1652 IMPROVEMENT
1654 The csv_to_db command requires an external Perl library
1655 (Text::CSV_XS). On computers that lack this optional library,
1656 previously Fsdb would configure with a warning and then test cases
1657 would fail. Now those test cases are skipped with an additional
1658 warning.
1659 BUG FIX
1661 The test suite now supports alternative valid output, as a hack to
1662 account for last-digit floating point differences. (Not very
1663 satisfying :-(
1664 BUG FIX
1666 dbcolstats output for confidence intervals on very large datasets
1667 has changed. Previously it failed for more than 2^31-1 records,
1668 and handling of T-Distributions with thousands of rows was a bit
1669 dubious. Now datasets with more than 10000 are considered
1670 infinitely large and hopefully correctly handled.
1671 2.24, 2011-04-15 Improvements to fix an old bug in dbmapreduce with
1673 different field separators
1674 IMPROVEMENT
1675 The dbfilealter command had a "--correct" option to work-around
1676 from incompatible field-separators, but it did nothing. Now it
1677 does the correct but sad, data-loosing thing.
1678 IMPROVEMENT
1680 The dbmultistats command previously failed with an error message
1681 when invoked on input with a non-default field separator. The root
1682 cause was the underlying dbmapreduce that did not handle the case
1683 of reducers that generated output with a different field separator
1684 than the input. We now detect and repair incompatible field
1685 separators. This change corrects a problem originally documented
1686 and detected in Fsdb-2.20. Bug re-reported by Unkyu Park.
1687 2.25, 2011-08-07 Two new tools, xml_to_db and dbfilepivot, and a bugfix for
1689 two people.
1690 IMPROVEMENT
1691 kitrace_to_db now supports a --utc option, which also fixes this
1692 test case for users outside of the Pacific time zone. Bug reported
1693 by David Graff, and also by Peter Desnoyers (within a week of each
1694 other :-)
1695 NEW xml_to_db can convert simple, very regular XML files into Fsdb.
1697 NEW dbfilepivot "pivots" a file, converting multiple rows corresponding
1699 to the same entity into a single row with multiple columns.
1700 2.26, 2011-12-12 Bug fixes, particularly for perl-5.14.2.
1702 BUG FIX
1703 Bugs fixed in Fsdb::IO::Reader(3) manual page.
1704 BUG FIX
1706 Fixed problems where dbcolstats was truncating floating point
1707 numbers when sorting. This strange behavior happens as of
1708 perl-5.14.2 and it seems like a Perl bug. I've worked around it
1709 for the test suites, but I'm a bit nervous.
1710 2.27, 2012-11-15 Accumulated bug fixes.
1712 IMPROVEMENT
1713 csv_to_db now reports errors in CVS input with real diagnostics.
1714 IMPROVEMENT
1716 dbcolmovingstats can now compute median, when given the "-m"
1717 option.
1718 BUG FIX
1720 dbcolmovingstats non-numeric handling (the "-a" option) now works
1721 properly.
1722 DOCUMENTATION
1724 The internal t/test_command.t test framework is now documented.
1725 BUG FIX
1727 dbrowuniq now correctly handles the case where there is no input
1728 (previously it output a blank line, which is a malformed fsdb
1729 file). Thanks to Yuri Pradkin for reporting this bug.
1730 2.28, 2012-11-15 A quick release to fix most rpmlint errors.
1732 BUG FIX
1733 Fixed a number of minor release problems (wrong permissions, old
1734 FSF address, etc.) found by rpmlint.
1735 2.29, 2012-11-20 a quick release for CPAN testing
1737 IMPROVEMENT
1738 Tweaked the RPM spec.
1739 IMPROVEMENT
1741 Modified Makefile.PL to fail gracefully on Perl installations that
1742 lack threads. (Without this fix, I get massive failures in the
1743 non-ithreads test system.)
1744 2.30, 2012-11-25 improvements to perl portability
1746 BUG FIX
1747 Removed unicode character in documention of dbcolscorrelated so pod
1748 tests will pass. (Sigh, that should work :-( )
1749 BUG FIX
1751 Fixed test suite failures on 5 tests (dbcolcreate_double_creation
1752 was the first) due to Carp's addition of a period. This problem
1753 was breaking Fsdb on perl-5.17. Thanks to Michael McQuaid for
1754 helping diagnose this problem.
1755 IMPROVEMENT
1757 The test suite now prints out the names of tests it tries.
1758 2.31, 2012-11-28 A release with actual improvements to dbfilepivot and
1760 dbrowuniq.
1761 BUG FIX
1762 Documentation fixes: typos in dbcolscorrelated, bugs in
1763 dbfilepivot, clarification for comment handling in
1764 Fsdb::IO::Reader.
1765 IMPROVEMENT
1767 Previously dbfilepivot assumed the input was grouped by keys and
1768 didn't very that pre-condition. Now there is no pre-condition (it
1769 will sort the input by default), and it checks if the invariant is
1770 violated.
1771 BUG FIX
1773 Previously dbfilepivot failed if the input had comments (oops :-);
1774 no longer.
1775 IMPROVEMENT
1777 Now dbrowuniq has the "-L" option to preserve the last unique row
1778 (instead of the first), a common idiom.
1779 2.32, 2012-12-21 Test suites should now be more numerically robust.
1781 NEW New dbfilediff does fsdb-aware file differencing. It does not do
1782 smart intuition of add/removes like Unix diff(1), but it does know
1783 about columns, and with "-E", it does numeric-aware differences.
1784 IMPROVEMENT
1786 Test suites that are numeric now use dbfilediff to do numeric-aware
1787 comparisons, so the test suite should now be robust to slightly
1788 different computers and operating systems and compilers than
1789 exactly what I use.
1790 2.33, 2012-12-23 Minor fixes to some test cases.
1792 IMPROVEMENT
1793 dbfilediff and dbrowuniq now supports the "-N" option to give the
1794 new column a different name. (And a test cases where this
1795 duplication mattered have been fixed.)
1796 IMPROVEMENT
1798 dbrvstatdiff now show the t-test breakpoint with a reasonable
1799 number of floating point digits.
1800 BUG FIX
1802 Fixed a numerical stability problem in the dbroweval_last test
1803 case.
1804
1806 2.34, 2013-02-10 Parallelism in dbmerge.
1807 IMPROVEMENT
1808 Documention for dbjoin now includes resource requirements.
1809 IMPROVEMENT
1811 Default memory usage for dbsort is now about 256MB. (The world
1812 keeps moving forward.)
1813 IMPROVEMENT
1815 dbmerge now does merging in parallel. As a side-effect, dbsort
1816 should be faster when input overflows memory. The level of
1817 parallelism can be limited with the "--parallelism" option. (There
1818 is more work to do here, but we're off to a start.)
1819 2.35, 2013-02-23 Improvements to dbmerge parallelism
1821 BUG FIX
1822 Fsdb temporary files are now created more securely (with
1823 File::Temp).
1824 IMPROVEMENT
1826 Programs that sort or merge on fields (dbmerge2, dbmerge, dbsort,
1827 dbjoin) now report an error if no fields on which to join or merge
1828 are given.
1829 IMPROVEMENT
1831 Parallelism in dbmerge is should now be more consistent, with less
1832 starting and stopping.
1833 IMPROVEMENT In dbmerge, the "--xargs" option lets one give input
1835 filenames on standard input, rather than the command line. This feature
1836 paves the way for faster dbsort for large inputs (by pipelining sorting
1837 and merging), expected in the next release.
1838 2.36, 2013-02-25 dbsort pipelines with dbmerge
1840 IMPROVEMENT For large inputs, dbsort now pipelines sorting and merging,
1841 allowing earlier processing.
1842 BUG FIX Since 2.35, dbmerge delayed cleanup of intermediate files,
1843 thereby requiring extra disk space.
1844 2.37, 2013-02-26 quick bugfix to support parallel sort and merge from
1846 recent releases
1847 BUG FIX Since 2.35, dbmerge delayed removal of input files given by
1848 "--xargs". This problem is now fixed.
1849 2.38, 2013-04-29 minor bug fixes
1851 CLARIFICATION
1852 Configure now rejects Windows since tests seem to hang on some
1853 versions of Windows. (I would love help from a Windows developer
1854 to get this problem fixed, but I cannot do it.) See
1855 https://rt.cpan.org/Ticket/Display.html?id=84201.
1856 IMPROVEMENT
1858 All programs that use temporary files (dbcolpercentile,
1859 dbcolscorrelate, dbcolstats, dbcolstatscores) now take the "-T"
1860 option and set the temporary directory consistently.
1861 In addition, error messages are better when the temporary directory
1863 has problems. Problem reported by Liang Zhu.
1864 BUG FIX
1866 dbmapreduce was failing with external, map-reduce aware reducers
1867 (when invoked with -M and an external program). (Sigh, did this
1868 case ever work?) This case should now work. Thanks to Yuri
1869 Pradkin for reporting this bug (in 2011).
1870 BUG FIX
1872 Fixed perl-5.10 problem with dbmerge. Thanks to Yuri Pradkin for
1873 reporting this bug (in 2013).
1874 2.39, date 2013-05-31 quick release for the dbrowuniq extension
1876 BUG FIX
1877 Actually in 2.38, the Fedora .spec got cleaner dependencies.
1878 Suggestion from Christopher Meng via
1879 <https://bugzilla.redhat.com/show_bug.cgi?id=877096>.
1880 ENHANCEMENT
1882 Fsdb files are now explicitly set into UTF-8 encoding, unless one
1883 specifies "-encoding" to "Fsdb::IO".
1884 ENHANCEMENT
1886 dbrowuniq now supports "-I" for incremental counting.
1887 2.40, 2013-07-13 small bug fixes
1889 BUG FIX
1890 dbsort now has more respect for a user-given temporary directory;
1891 it no longer is ignored for merging.
1892 IMPROVEMENT
1894 dbrowuniq now has options to output the first, last, and both first
1895 and last rows of a run ("-F", "-L", and "-B").
1896 BUG FIX
1898 dbrowuniq now correctly handles "-N". Sigh, it didn't work before.
1899 2.41, 2013-07-29 small bug and packaging fixes
1901 ENHANCEMENT
1902 Documentation to dbrvstatdiff improved (inspired by questions from
1903 Qian Kun).
1904 BUG FIX
1906 dbrowuniq no longer duplicates singleton unique lines when
1907 outputting both (with "-B").
1908 BUG FIX
1910 Add missing "XML::Simple" dependency to Makefile.PL.
1911 ENHANCEMENT
1913 Tests now show the diff of the failing output if run with "make
1914 test TEST_VERBOSE=1".
1915 ENHANCEMENT
1917 dbroweval now includes documentation for how to output extra rows.
1918 Suggestion from Yuri Pradkin.
1919 BUG FIX
1921 Several improvements to the Fedora package from Michael Schwendt
1922 via <https://bugzilla.redhat.com/show_bug.cgi?id=877096>, and from
1923 the harsh master that is rpmlint. (I am stymied at teaching it
1924 that "outliers" is spelled correctly. Maybe I should send it
1925 Schneier's book. And an unresolvable invalid-spec-name lurks in
1926 the SRPM.)
1927 2.42, 2013-07-31 A bug fix and packaging release.
1929 ENHANCEMENT
1930 Documentation to dbjoin improved to better memory usage. (Based on
1931 problem report by Lin Quan.)
1932 BUG FIX
1934 The .spec is now perl-Fsdb.spec to satisfy rpmlint. Thanks to
1935 Christopher Meng for a specific bug report.
1936 BUG FIX
1938 Test dbroweval_last.cmd no longer has a column that caused failures
1939 because of numerical instability.
1940 BUG FIX
1942 Some tests now better handle bugs in old versions of perl (5.10,
1943 5.12). Thanks to Calvin Ardi for help debugging this on a Mac with
1944 perl-5.12, but the fix should affect other platforms.
1945 2.43, 2013-08-27 Adds in-file compression.
1947 BUG FIX
1948 Changed the sort on TEST/dbsort_merge.cmd to strings (from
1949 numerics) so we're less susceptible to false test-failures due to
1950 floating point IO differences.
1951 EXPERIMENTAL ENHANCEMENT
1953 Yet more parallelism in dbmerge: new "endgame-mode" builds a merge
1954 tree of processes at the end of large merge tasks to get maximally
1955 parallelism. Currently this feature is off by default because it
1956 can hang for some inputs. Enable this experimental feature with
1957 "--endgame".
1958 ENHANCEMENT
1960 "Fsdb::IO" now handles being given "IO::Pipe" objects (as exercised
1961 by dbmerge).
1962 BUG FIX
1964 Handling of NamedTmpfiles now supports concurrency. This fix will
1965 hopefully fix occasional "Use of uninitialized value $_ in string
1966 ne at ...NamedTmpfile.pm line 93." errors.
1967 BUG FIX
1969 Fsdb now requires perl 5.10. This is a bug fix because some test
1970 cases used to require it, but this fact was not properly
1971 documented. (Back-porting to 5.008 would require removing all "//"
1972 operators.)
1973 ENHANCEMENT
1975 Fsdb now handles automatic compression of file contents. Enable
1976 compression with "dbfilealter -Z xz" (or "gz" or "bz2"). All
1977 programs should operate on compressed files and leave the output
1978 with the same level of compression. "xz" is recommended as fastest
1979 and most efficient. "gz" is produces unrepeatable output (and so
1980 has no output test), it seems to insist on adding a timestamp.
1981 2.44, 2013-10-02 A major change--all threads are gone.
1983 ENHANCEMENT
1984 Fsdb is now thread free and only uses processes for parallelism.
1985 This change is a big change--the entire motivation for Fsdb-2 was
1986 to exploit parallelism via threading. Parallelism--good, but perl
1987 threading--bad for performance. Horribly bad for performance.
1988 About 20x worse than pipes on my box. (See perl bug #119445 for
1989 the discussion.)
1990 NEW "Fsdb::Support::Freds" provides a thread-like abstraction over
1992 forking, with some nice support for callbacks in the parent upon
1993 child termination.
1994 ENHANCEMENT
1996 Details about removing threads: "dbpipeline" is thread free, and
1997 new tests to verify each of its parts. The easy cases are
1998 "dbcolpercentile", "dbcolstats", "dbfilepivot", "dbjoin", and
1999 "dbcolstatscores", each of which use it in simple ways
2000 (2013-09-09). "dbmerge" is now thread free (2013-09-13), but was a
2001 significant rewrite, which brought "dbsort" along. "dbmapreduce"
2002 is partly thread free (2013-09-21), again as a rewrite, and it
2003 brings "dbmultistats" along. Full "dbmapreduce" support took much
2004 longer (2013-10-02).
2005 BUG FIX
2007 When running with user-only output ("-n"), dbroweval now resets the
2008 output vector $ofref after it has been output.
2009 NEW dbcolcreate will create all columns at the head of each row with
2011 the "--first" option.
2012 NEW dbfilecat will concatenate two files, verifying that they have the
2014 same schema.
2015 ENHANCEMENT
2017 dbmapreduce now passes comments through, rather than eating them as
2018 before.
2019 Also, dbmapreduce now supports a "--" option to prevent
2021 misinterpreting sub-program parameters as for dbmapreduce.
2022 INCOMPATIBLE CHANGE
2024 dbmapreduce no longer figures out if it needs to add the key to the
2025 output. For multi-key-aware reducers, it never does (and cannot).
2026 For non-multi-key-aware reducers, it defaults to add the key and
2027 will now fail if the reducer adds the key (with error "dbcolcreate:
2028 attempt to create pre-existing column..."). In such cases, one
2029 must disable adding the key with the new option "--no-prepend-key".
2030 INCOMPATIBLE CHANGE
2032 dbmapreduce no longer copies the input field separator by default.
2033 For multi-key-aware reducers, it never does (and cannot). For non-
2034 multi-key-aware reducers, it defaults to not copying the field
2035 separator, but it will copy it (the old default) with the
2036 "--copy-fs" option
2037 2.45, 2013-10-07 cleanup from de-thread-ification
2039 BUG FIX
2040 Corrected a fast busy-wait in dbmerge.
2041 ENHANCEMENT
2043 Endgame mode enabled in dbmerge; it (and also large cases of
2044 dbsort) should now exploit greater parallelism.
2045 BUG FIX
2047 Test case with "Fsdb::BoundedQueue" (gone since 2.44) now removed.
2048 2.46, 2013-10-08 continuing cleanup of our no-threads version
2050 BUG FIX
2051 Fixed some packaging details. (Really, threads are no longer
2052 required, missing tests in the MANIFEST.)
2053 IMPROVEMENT
2055 dbsort now better communicates with the merge process to avoid
2056 bursty parallelism.
2057 Fsdb::IO::Writer now can take "-autoflush =" 1> for line-buffered
2059 IO.
2060 2.47, 2013-10-12 test suite cleanup for non-threaded perls
2062 BUG FIX
2063 Removed some stray "use threads" in some test cases. We didn't
2064 need them, and these were breaking non-threaded perls.
2065 BUG FIX
2067 Better handling of Fred cleanup; should fix intermittent
2068 dbmapreduce failures on BSD.
2069 ENHANCEMENT
2071 Improved test framework to show output when tests fail. (This
2072 time, for real.)
2073 2.48, 2014-01-03 small bugfixes and improved release engineering
2075 ENHANCEMENT
2076 Test suites now skip tests for libraries that are missing. (Patch
2077 for missing "IO::Compresss:Xz" contributed by Calvin Ardi.)
2078 ENHANCEMENT
2080 Removed references to Jdb in the package specification. Since the
2081 name was changed in 2008, there's no longer a huge need for
2082 backwards compatibility. (Suggestion form Petr Šabata.)
2083 ENHANCEMENT
2085 Test suites now invoke the perl using the path from
2086 $Config{perlpath}. Hopefully this helps testing in environments
2087 where there are multiple installed perls and the default perl is
2088 not the same as the perl-under-test (as happens in
2089 cpantesters.org).
2090 BUG FIX
2092 Added specific encoding to this manpage to account for Unicode.
2093 Required to build correctly against perl-5.18.
2094 2.49, 2014-01-04 bugfix to unicode handling in Fsdb IO (plus minor
2096 packaging fixes)
2097 BUG FIX
2098 Restored a line in the .spec to chmod g-s.
2099 BUG FIX
2101 Unicode decoding is now handled correctly for programs that read
2102 from standard input. (Also: New test scripts cover unicode input
2103 and output.)
2104 BUG FIX
2106 Fix to Fsdb documentation encoding line. Addresses test failure in
2107 perl-5.16 and earlier. (Who knew "encoding" had to be followed by
2108 a blank line.)
2109
2111 2.50, 2014-05-27 a quick release for spec tweaks
2112 ENHANCEMENT
2113 In dbroweval, the "-N" (no output, even comments) option now
2114 implies "-n", and it now suppresses the header and trailer.
2115 BUG FIX
2117 A few more tweaks to the perl-Fsdb.spec from Petr Šabata.
2118 BUG FIX
2120 Fixed 3 uses of "use v5.10" in test suites that were causing test
2121 failures (due to warnings, not real failures) on some platforms.
2122 2.51, 2014-09-05 Feature enhancements to dbcolmovingstats, dbcolcreate,
2124 dbmapreduce, and new sqlselect_to_db
2125 ENHANCEMENT
2126 dbcolcreate now has a "--no-recreate-fatal" that causes it to
2127 ignore creation of existing columns (instead of failing).
2128 ENHANCEMENT
2130 dbmapreduce once again is robust to reducers that output the key;
2131 "--no-prepend-key" is no longer mandatory.
2132 ENHANCEMENT
2134 dbcolsplittorows can now enumerate the output rows with "-E".
2135 BUG FIX
2137 dbcolmovingstats is more mathematically robust. Previously for
2138 some inputs and some platforms, floating point rounding could
2139 sometimes cause squareroots of negative numbers.
2140 NEW sqlselect_to_db converts the output of the MySQL or MarinaDB select
2142 comment into fsdb format.
2143 INCOMPATIBLE CHANGE
2145 dbfilediff now outputs the second row when doing sloppy numeric
2146 comparisons, to better support test suites.
2147 2.52, 2014-11-03 Fixing the test suite for line number changes.
2149 ENHANCEMENT
2150 Test suites changes to be robust to exact line numbers of failures,
2151 since different Perl releases fail on different lines.
2152 <https://bugzilla.redhat.com/show_bug.cgi?id=1158380>
2153 2.53, 2014-11-26 bug fixes and stability improvements to dbmapreduce
2155 ENHANCEMENT
2156 The dbfilediff how supports a "--quiet" option.
2157 ENHANCEMENT
2159 Better documention of dbpipeline_filter.
2160 BUGFIX
2162 Added groff-base and perl-podlators to the Fedora package spec.
2163 Fixes <https://bugzilla.redhat.com/show_bug.cgi?id=1163149>. (Also
2164 in package 2.52-2.)
2165 BUGFIX
2167 An important stability improvement to dbmapreduce. It, plus
2168 dbmultistats, and dbcolstats now support controlled parallelism
2169 with the "--pararallelism=N" option. They default to run with the
2170 number of available CPUs. dbmapreduce also moderates its level of
2171 parallelism. Previously it would create reducers as needed,
2172 causing CPU thrashing if reducers ran much slower than data
2173 production.
2174 BUGFIX
2176 The combination of dbmapreduce with dbrowenumerate now works as it
2177 should. (The obscure bug was an interaction with dbcolcreate with
2178 non-multi-key reducers that output their own key. dbmapreduce has
2179 too many useful corner cases.)
2180 2.54, 2014-11-28 fix for the test suite to correct failing tests on not-my-
2182 platform
2183 BUGFIX
2184 Sigh, the test suite now has a test suite. Because, yes, I broke
2185 it, causing many incorrect failures at cpantesters. Now fixed.
2186 2.55, 2015-01-05 many spelling fixes and dbcolmovingstats tests are more
2188 robust to different numeric precision
2189 ENHANCEMENT
2190 dbfilediff now can be extra quiet, as I continue to try to track
2191 down a numeric difference on FreeBSD AMD boxes.
2192 ENHANCEMENT
2194 dbcolmovingstats gave different test output (just reflecting
2195 rounding error) when stddev approaches zero. We now detect hand
2196 handle this case. See
2197 <https://rt.cpan.org/Public/Bug/Display.html?id=101220> and thanks
2198 to H. Merijn Brand for the bug report.
2199 BUG FIX
2201 Many, many spelling bugs found by H. Merijn Brand; thanks for the
2202 bug report.
2203 INCOMPATBLE CHANGE
2205 A number of programs had misspelled "separator" in
2206 "--fieldseparator" and "--columnseparator" options as "seperator".
2207 These are now correctly spelled.
2208 2.56, 2015-02-03 fix against Getopt::Long-2.43's stricter error checkign
2210 BUG FIX
2211 Internal argument parsing uses Getopt::Long, but mixed pass-through
2212 and <>. Bug reported by Petr Pisar at
2213 <https://bugzilla.redhat.com/show_bug.cgi?id=1188538>.a
2214 BUG FIX
2216 Added missing BuildRequires for "XML::Simple".
2217 2.57, 2015-04-29 Minor changes, with better performance from dbmulitstats.
2219 BUG FIX
2220 dbfilecat now honors "--remove-inputs" (previously it didn't).
2221 This omission meant that dbmapreduce (and dbmultistats) would
2222 accumulate files in /tmp when running. Bad news for inputs with 4M
2223 keys.
2224 ENHANCMENT
2226 dbmultistats should be faster with lots of small keys. dbcolstats
2227 now supports "-k" to get some of the functionality of dbmultistats
2228 (if data is pre-sorted and median/quartiles are not required).
2229 dbfilecat now honors "--remove-inputs" (previously it didn't).
2231 This omission meant that dbmapreduce (and dbmultistats) would
2232 accumulate files in /tmp when running. Bad news for inputs with 4M
2233 keys.
2234 2.58, 2015-04-30 Bugfix in dbmerge
2236 BUG FIX
2237 Fixed a case where dbmerge suffered mojobake in endgame mode. This
2238 bug surfaced when dbsort was applied to large files (big enough to
2239 require merging) with unicode in them; the symptom was soemthing
2240 like:
2241 Wide character in print at /usr/lib64/perl5/IO/Handle.pm line
2242 420, <GEN12> line 111.
2243 2.59, 2016-09-01 Collect a few small bug fixes and documentation
2245 improvements.
2246 BUG FIX
2247 More IO is explicitly marked UTF-8 to avoid Perl's tendency to
2248 mojibake on otherwise valid unicode input. This change helps
2249 html_table_to_db.
2250 ENHANCEMENT
2252 dbcolscorrelate now crossreferences dbcolsregression.
2253 ENHANCEMENT
2255 Documentation for dbrowdiff now clarifies that the default is
2256 baseline mode.
2257 BUG FIX
2259 dbjoin now propagates "-T" into the sorting process (if it is
2260 required). Thanks to Lan Wei for reporting this bug.
2261 2.60, 2016-09-04 Adds support for hash joins.
2263 ENHANCEMENT
2264 dbjoin now supports hash joins with "-t lefthash" and "-t
2265 righthash". Hash joins cache a table in memory, but do not require
2266 that the other table be sorted. They are ideal when joining a
2267 large table against a small one.
2268 2.61, 2016-09-05 Support left and right outer joins.
2270 ENHANCEMENT
2271 dbjoin now handles left and right outer joins with "-t left" and
2272 "-t right".
2273 ENHANCEMENT
2275 dbjoin hash joins are now selected with "-m lefthash" and "-m
2276 righthash" (not the shortlived "-t righthash" option).
2277 (Technically this change is incompatible with Fsdd-2.60, but no one
2278 but me ever used that version.)
2279 2.62, 2016-11-29 A new yaml_to_db and other minor improvements.
2281 ENHANCEMENT
2282 Documentation for xml_to_db now includes sample output.
2283 NEW yaml_to_db converts a specific form of YAML to fsdb.
2285 BUG FIX
2287 The test suite now uses "diff -c -b" rather than "diff -cb" to make
2288 OpenBSD-5.9 happier, I hope.
2289 ENHANCEMENT
2291 Comments that log operations at the end of each file now do simple
2292 quoting of spaces. (It is not guaranteed to be fully shell-
2293 compliant.)
2294 ENHANCEMENT
2296 There is a new standard option, "--header", allowing one to specify
2297 an Fsdb header for inputs that lack it. Currently it is supported
2298 by dbcoldefine, dbrowuniq, dbmapreduce, dbmultistats, dbsort,
2299 dbpipeline.
2300 ENHANCEMENT
2302 dbfilepivot now allows the --possible-pivots option, and if it is
2303 provided processes the data in one pass.
2304 ENHANCEMENT
2306 dbroweval logs are now quoted.
2307 2.63, 2017-02-03 Re-add some features supposedly in 2.62 but not, and add
2309 more --header options.
2310 ENHANCEMENT
2311 The option -j is now a synonym for --parallelism. (And several
2312 documention bugs about this option are fixed.)
2313 ENHANCEMENT
2315 Additional support for "--header" in dbcolmerge, dbcol, dbrow, and
2316 dbroweval.
2317 BUG FIX
2319 Version 2.62 was supposed to have this improvement, but did not
2320 (and now does): dbfilepivot now allows the --possible-pivots
2321 option, and if it is provided processes the data in one pass.
2322 BUG FIX
2324 Version 2.62 was supposed to have this improvement, but did not
2325 (and now does): dbroweval logs are now quoted.
2326 2.64, 2017-11-20 several small bugfixes and enhancements
2328 BUG FIX
2329 In dbroweval, the "next row" option previously did not correctly
2330 set up "_last_fieldname". It now does.
2331 ENHANCEMENT
2333 The csv_to_db converter now has an optional "-F x" option to set
2334 the field separator.
2335 ENHANCEMENT
2337 Finally dbcolsplittocols has a "--header" option, and a new "-N"
2338 option to give the list of resulting output columns.
2339 INCOMPATIBLE CHANGE
2341 Now dbcolstats and dbmultistats produce no output (but a schema)
2342 when given no input but a schema. Previously they gave a null row
2343 of output. The "--output-on-no-input" and
2344 "--no-output-on-no-input" options can control this behavior.
2345 2.65, 2018-02-16 Minor release, bug fix and -F option.
2347 ENHANCEMENT
2348 dbmultistats and dbmapreduce now both take a "-F x" option to set
2349 the field separator.
2350 BUG FIX
2352 Fixed missing "use Carp" in dbcolstats. Also went back and cleaned
2353 up all uses of croak(). Thanks to Zefram for the bug report.
2354 2.66, 2018-12-20 Critical bug fix in dbjoin.
2356 BUG FIX
2357 Removed old tests from MANIFEST. (Thanks to Hang Guo for reporting
2358 this bug.)
2359 IMPROVEMENT
2361 Errors for non-existing input files now include the bad filename
2362 (before: "cannot setup filehandle", now: "cannot open input: cannot
2363 open TEST/bad_filename").
2364 BUG FIX
2366 Hash joins with three identical rows were failing with the
2367 assertion failure "internal error: confused about overflow" due to
2368 a now-fixed bug.
2369 2.67, 2019-07-10 add support for reading and writing hdfs
2371 IMPROVEMENT
2372 dbformmail now has an "mh" mechanism that writes messages to
2373 individual files (an mh-style mailbox).
2374 BUG FIX
2376 dbrow failed to include the Carp library, leading to fails on
2377 croak.
2378 BUG FIX
2380 Fixed dbjoin error message for an unsorted right stream was
2381 incorrect (it said left).
2382 IMPROVEMENT
2384 All Fsdb programs can now read from and write to HDFS, when files
2385 that start with "hdfs:" are given to -i and -o options.
2386 2.68, 2019-09-19 All programs now support automatic decompression based on
2388 file extension.
2389 IMPROVEMENT
2390 The omitted-possible-error test case for dbfilepivot now has an
2391 altnerative output that I saw on some BSD-running systems (thanks
2392 to CPAN).
2393 IMPROVEMENT
2395 dbmerge and dbmerge2 now support "--header". dbmerge2 now gives
2396 better error messages when presented the wrong number of inputs.
2397 BUG FIX
2399 dbsort now works with "--header" even when the file is big (due to
2400 fixes to dbmerge).
2401 IMPROVEMENT
2403 cvs_to_db now processes data with the "binary" option, allowing it
2404 to handle newlines embedded in quoted fields.
2405 IMPROVEMENT
2407 All programs now will transparently decompress input files, if they
2408 are listed as a filename as an input argument that extends with a
2409 standard extension (.gz, .bz2, and .xz).
2410 2.69, 2019-11-22 a small bugfix in dbcolstats
2412 BUG FIX
2413 Filled in the the test case for autodecompress, which was missing
2414 for the 2.68 release.
2415 ENHANCEMENT
2417 The groff program is required for build, and the "Makefile.PL"
2418 fails if groff is missing at build time. Thanks to Chris Williams
2419 for suggesting this check, and the CPAN auto-building system for
2420 trying many platforms.
2421 BUG FIX
2423 The dbcolstats program had numerical instability that sometimes
2424 results in failing with a square-root of a negative number when
2425 many values varied right at the edge of floating-point precision.
2426 We now detect and report that case as 0 stddev. Thanks to Hang Guo
2427 for providing a test case.
2428 2.70, 2020-11-12 Some small quality-of-life enhancements and corner-case
2430 bugfixes.
2431 ENHANCEMENT
2432 dbcol can now take an option "-a" to include all columns, allowing
2433 reordering of certain columns while passing the rest through.
2434 ENHANCEMENT
2436 dbrowuniq and dbmerge now buffer comments in a way that the last
2437 row of data output is no longer in the last block of comments.
2438 (The data is identical, but for humans looking at output, this
2439 change makes it less likely to lose the last row.)
2440 BUG FIX
2442 dbmultistats and dbpipeline documentation now indicates that they
2443 support "--header" (something they did since version 2.62 in
2444 2016-11-29, but now documented.
2445 ENHANCEMENT
2447 dbcolcreate now supports "--header".
2448 BUG FIX
2450 Fixed several spelling errors in deprecated programs and removed
2451 information about the no-longer existing FreeBSD and MacOS ports.
2452 Thanks to Calvin Ardi for the patch.
2453 BUG FIX
2455 dbmerge now handles --xargs when only one file is provided (and
2456 passes the file through unchanged). It also throws a clean error
2457 with --xargs if zero files are provided. (To support dbmerge,
2458 dbcol now has an internal "--saveoutput" option.) Thanks to Yuri
2459 Pradkin for reporting the unhandled corner-case.
2460 2.71, 2020-11-16 Fix a race condition breaking test suites.
2462 BUG FIX
2463 Suppress a race condition in dbcolmerge was sometimes throwing the
2464 error "Fsdb::Support::Freds: ending, but running process:
2465 dbmerge:xargs" in the dbmerge_0_xargs test case, on exit.
2466 2.72, 2020-12-01 A small bug and a packaging improvement.
2468 BUG FIX
2469 dbcolhisto now handles the degenerate case where everything has the
2470 same value (previously it would throw "illegal division by zero").
2471 ENHANCEMENT
2473 The spec for Fedora now includes "make" as BuildRequires, something
2474 required for Fedora 34.
2475 2.73, 2021-05-18 Updates dbcolpercentile with "--weighted", and with more
2477 ipv6.
2478 ENHANCEMENT
2479 dbcolpercentile now has a "--weighted" option.
2480 ENHANCEMENT
2482 The new Fsdb::Support::IPv6 package includes ipv6_normalize,
2483 ipv6_zeroize to rewrite ipv6 print addresses in IPv6 normal form,
2484 with a 0 in each 4-nybble field.
2485 2.74, 2021-06-23 More ipv6.
2487 ENHANCEMENT
2488 Fsdb::Support::IPv6 package includes ipv6_fullhex to rewrite ipv6
2489 print addresses as full, 128-bit hex values.
2490 2.75, 2022-04-02 New type specifications in the schema to better support
2492 type conversions in python.
2493 ENHANCEMENT
2494 Add optional type specifications to the schema. Types are not used
2495 in Perl, but are relevant in Python and Go Fsdb bindings. Types
2496 use a subset of perl pack specifiers: c, s, l, q are signed 8, 16,
2497 32, and 64-bit integers, f is a float, d is double float, a is
2498 utf-8 string, and > and < can force big or little endianness.
2499 The default type for everything is "a", that is, utf-8 strings.
2500 Thanks to Wes Hardaker for pushing to get this long-desired feature
2501 out the door; his Python bindings need types.
2502 ENHANCEMENT
2504 dbcol, dbcolcreate, dbcolcopylast, and dbcolrename now understand
2505 and propagate schema types. dbsort, dbjoin, dbmerge, dbmerge2 and
2506 dbfilepivot all take a new option "-t" to sort by type-inferred
2507 comparision, if a type is given.
2508 ENHANCEMENT
2510 dbcolstat, dbmultistats, and dbcolmovingstats now include type
2511 information in their output schema. (They assumes input variables
2512 are floats, not integers.)
2513 ENHANCEMENT
2515 Even more IPv6: the functions in Fsdb::Support::IPv6 package now
2516 support strings of hex digits as an alternate encoding for IP
2517 address (and they are already the output of ipv6_fullhex), and
2518 "ip_fullhex_to_normal" converts full hex-encoded IPv4 or IPv6
2519 addresses to their "normal" form (dotted-quad or IPv6 printable
2520 format).
2521 3.0, 2022-04-04 Complete type support and accordingly bump major version.
2523 NEW The major version number is now 3.0 to correspond to the addition
2524 of types (although they were actually added in 2.75). Old fsdb
2525 files are supported (Fsdb-3.0 is backwards compatible with
2526 databases), but older versions will confuse types in new files (new
2527 Fsdb files are not forward compatible with old versions).
2528 ENHANCEMENT
2530 Type specifications in a few more programs: dbcolhisto,
2531 dbcolscorrelate, dbcolsregression, dbcolstatscores,
2532 dbrowaccumulate, dbrowcount, dbrowdiff, dbrvstatdiff.
2533 ENHANCEMENT
2535 dbcolhisto now puts an empty value on any empty rows.
2536 NEW dbcoltype redefines column types, or clears them with the "-v"
2538 option.
2539
2541 John Heidemann, "johnh@isi.edu"
2542 See "Contributors" for the many people who have contributed bug reports
2544 and fixes.
2545
2547 Fsdb is Copyright (C) 1991-2022 by John Heidemann <johnh@isi.edu>.
2548 This program is free software; you can redistribute it and/or modify it
2550 under the terms of version 2 of the GNU General Public License as
2551 published by the Free Software Foundation.
2552 This program is distributed in the hope that it will be useful, but
2554 WITHOUT ANY WARRANTY; without even the implied warranty of
2555 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
2556 General Public License for more details.
2557 You should have received a copy of the GNU General Public License along
2559 with this program; if not, write to the Free Software Foundation, Inc.,
2560 675 Mass Ave, Cambridge, MA 02139, USA.
2561 A copy of the GNU General Public License can be found in the file
2563 ``COPYING''.
2564
2566 Any comments about these programs should be sent to John Heidemann
2567 "johnh@isi.edu".
2568perl v5.36.0 2023-01-20 Fsdb(3)