1CSV_XS(3) User Contributed Perl Documentation CSV_XS(3)
2
6 Text::CSV_XS - comma-separated values manipulation routines
7
9 use Text::CSV_XS;
10 $csv = Text::CSV_XS->new (); # create a new object
12 $csv = Text::CSV_XS->new (\%attr); # create a new object
13 $status = $csv->combine (@columns); # combine columns into a string
15 $line = $csv->string (); # get the combined string
16 $status = $csv->parse ($line); # parse a CSV string into fields
18 @columns = $csv->fields (); # get the parsed fields
19 $status = $csv->status (); # get the most recent status
21 $bad_argument = $csv->error_input (); # get the most recent bad argument
22 $status = $csv->print ($io, $colref); # Write an array of fields
24 # immediately to a file $io
25 $colref = $csv->getline ($io); # Read a line from file $io,
26 # parse it and return an array
27 # ref of fields
28 $eof = $csv->eof (); # Indicate if last parse or
29 # getline () hit End Of File
30 $csv->types (\@t_array); # Set column types
32
34 Text::CSV_XS provides facilities for the composition and decomposition
35 of comma-separated values. An instance of the Text::CSV_XS class can
36 combine fields into a CSV string and parse a CSV string into fields.
37 The module accepts either strings or files as input and can utilize any
39 user-specified characters as delimiters, separators, and escapes so it
40 is perhaps better called ASV (anything separated values) rather than
41 just CSV.
42 Embedded newlines
44 Important Note: The default behavior is to only accept ascii charac‐
46 ters. This means that fields can not contain newlines. If your data
47 contains newlines embedded in fields, or characters above 0x7e (tilde),
48 or binary data, you *must* set "binary => 1" in the call to "new ()".
49 To cover the widest range of parsing options, you will always want to
50 set binary.
51 But you still have the problem that you have to pass a correct line to
53 the "parse ()" method, which is more complicated from the usual point
54 of usage:
55 my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/ });
57 while (<>) { # WRONG!
58 $csv->parse ($_);
59 my @fields = $csv->fields ();
60 will break, as the while might read broken lines, as that doesn't care
62 about the quoting. If you need to support embedded newlines, the way to
63 go is either
64 use IO::Handle;
66 my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/ });
67 while (my $row = $csv->getline (*ARGV)) {
68 my @fields = @$row;
69 or, more safely in perl 5.6 and up
71 my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/ });
73 open my $io, "<", $file or die "$file: $!";
74 while (my $row = $csv->getline ($io)) {
75 my @fields = @$row;
76
78 While no formal specification for CSV exists, RFC 4180 1) describes a
79 common format and establishes "text/csv" as the MIME type registered
80 with the IANA.
81 Many informal documents exist that describe the CSV format. How To: The
83 Comma Separated Value (CSV) File Format 2) provides an overview of the
84 CSV format in the most widely used applications and explains how it can
85 best be used and supported.
86 1) http://tools.ietf.org/html/rfc4180
88 2) http://www.creativyst.com/Doc/Articles/CSV/CSV01.htm
89 The basic rules are as follows:
91 CSV is a delimited data format that has fields/columns separated by the
93 comma character and records/rows separated by newlines. Fields that
94 contain a special character (comma, newline, or double quote), must be
95 enclosed in double quotes. However, if a line contains a single entry
96 which is the empty string, it may be enclosed in double quotes. If a
97 field's value contains a double quote character it is escaped by plac‐
98 ing another double quote character next to it. The CSV file format does
99 not require a specific character encoding, byte order, or line termina‐
100 tor format.
101 · Each record is one line terminated by a line feed (ASCII/LF=0x0A) or
103 a carriage return and line feed pair (ASCII/CRLF=0x0D 0x0A), however,
104 line-breaks can be embedded.
105 · Fields are separated by commas.
107 · Allowable characters within a CSV field include 0x09 (tab) and the
109 inclusive range of 0x20 (space) through 0x7E (tilde). In binary mode
110 all characters are accepted, at least in quoted fields.
111 · A field within CSV must be surrounded by double-quotes to contain a
113 the separator character (comma).
114 Though this is the most clear and restrictive definition, Text::CSV_XS
116 is way more liberal than this, and allows extension:
117 · Line termination by a single carriage return is accepted by default
119 · The separation-, escape-, and escape- characters can be any ASCII
121 character in the range from 0x20 (space) to 0x7E (tilde). Characters
122 outside this range may or may not work as expected. Multibyte charac‐
123 ters, like U+060c (ARABIC COMMA), U+FF0C (FULLWIDTH COMMA), U+241B
124 (SYMBOL FOR ESCAPE), U+2424 (SYMBOL FOR NEWLINE), U+FF02 (FULLWIDTH
125 QUOTATION MARK), and U+201C (LEFT DOUBLE QUOTATION MARK) (to give
126 some examples of what might look promising) are therefor not allowed.
127 · A field within CSV must be surrounded by double-quotes to contain an
129 embedded double-quote, represented by a pair of consecutive dou‐
130 ble-quotes. In binary mode you may additionally use the sequence ""0"
131 for representation of a NULL byte.
132 · Several violations of the above specification may be allowed by pass‐
134 ing options to the object creator.
135
137 version ()
138 (Class method) Returns the current module version.
139 new (\%attr)
141 (Class method) Returns a new instance of Text::CSV_XS. The objects
142 attributes are described by the (optional) hash ref "\%attr". Cur‐
143 rently the following attributes are available:
144 eol An end-of-line string to add to rows, usually "undef" (nothing,
146 default), "\012" (Line Feed) or "\015\012" (Carriage Return,
147 Line Feed). Cannot be longer than 7 (ASCII) characters.
148 If both $/ and "eol" equal "\015", parsing lines that end on
150 only a Carriage Return without Line Feed, will be "parse"d cor‐
151 rect. Line endings, whether in $/ or "eol", other than
152 "undef", "\n", "\r\n", or "\r" are not (yet) supported for
153 parsing.
154 sep_char
156 The char used for separating fields, by default a comma. (",").
157 Limited to a single-byte character, usually in the range from
158 0x20 (space) to 0x7e (tilde).
159 The separation character can not be equal to the quote charac‐
161 ter. The separation character can not be equal to the escape
162 character.
163 allow_whitespace
165 When this option is set to true, whitespace (TAB's and SPACE's)
166 surrounding the separation character is removed when parsing.
167 So lines like:
168 1 , "foo" , bar , 3 , zapp
170 are now correctly parsed, even though it violates the CSV
172 specs. Note that all whitespace is stripped from start and end
173 of each field. That would make is more a feature than a way to
174 be able to parse bad CSV lines, as
175 1, 2.0, 3, ape , monkey
177 will now be parsed as
179 ("1", "2.0", "3", "ape", "monkey")
181 even if the original line was perfectly sane CSV.
183 quote_char
185 The char used for quoting fields containing blanks, by default
186 the double quote character ("""). A value of undef suppresses
187 quote chars. (For simple cases only). Limited to a single-byte
188 character, usually in the range from 0x20 (space) to 0x7e
189 (tilde).
190 The quote character can not be equal to the separation charac‐
192 ter.
193 allow_loose_quotes
195 By default, parsing fields that have "quote_char" characters
196 inside an unquoted field, like
197 1,foo "bar" baz,42
199 would result in a parse error. Though it is still bad practice
201 to allow this format, we cannot help there are some vendors
202 that make their applications spit out lines styled like this.
203 escape_char
205 The character used for escaping certain characters inside
206 quoted fields. Limited to a single-byte character, usually in
207 the range from 0x20 (space) to 0x7e (tilde).
208 The "escape_char" defaults to being the literal double-quote
210 mark (""") in other words, the same as the default
211 "quote_char". This means that doubling the quote mark in a
212 field escapes it:
213 "foo","bar","Escape ""quote mark"" with two ""quote marks""","baz"
215 If you change the default quote_char without changing the
217 default escape_char, the escape_char will still be the quote
218 mark. If instead you want to escape the quote_char by doubling
219 it, you will need to change the escape_char to be the same as
220 what you changed the quote_char to.
221 The escape character can not be equal to the separation charac‐
223 ter.
224 allow_loose_escapes
226 By default, parsing fields that have "escape_char" characters
227 that escape characters that do not need to be escaped, like:
228 my $csv = Text::CSV_XS->new ({ escape_char => "\\" });
230 $csv->parse (qq{1,"my bar\'s",baz,42});
231 would result in a parse error. Though it is still bad practice
233 to allow this format, this option enables you to treat all
234 escape character sequences equal.
235 binary
237 If this attribute is TRUE, you may use binary characters in
238 quoted fields, including line feeds, carriage returns and NULL
239 bytes. (The latter must be escaped as ""0".) By default this
240 feature is off.
241 types
243 A set of column types; this attribute is immediately passed to
244 the types method below. You must not set this attribute other‐
245 wise, except for using the types method. For details see the
246 description of the types method below.
247 always_quote
249 By default the generated fields are quoted only, if they need
250 to, for example, if they contain the separator. If you set this
251 attribute to a TRUE value, then all fields will be quoted. This
252 is typically easier to handle in external applications. (Poor
253 creatures who aren't using Text::CSV_XS. :-)
254 keep_meta_info
256 By default, the parsing of input lines is as simple and fast as
257 possible. However, some parsing information - like quotation of
258 the original field - is lost in that process. Set this flag to
259 true to be able to retrieve that information after parsing with
260 the methods "meta_info ()", "is_quoted ()", and "is_binary ()"
261 described below. Default is false.
262 verbatim
264 This is a quite controversial attribute to set, but it makes
265 hard things possible.
266 The basic thought behind this is to tell the parser that the
268 normally special characters newline (NL) and Carriage Return
269 (CR) will not be special when this flag is set, and be dealt
270 with as being ordinary binary characters. This will ease work‐
271 ing with data with embedded newlines.
272 When "verbatim" is used with "getline ()", getline auto-chomp's
274 every line.
275 Imagine a file format like
277 M^^Hans^Janssen^Klas 2\n2A^Ja^11-06-2007#\r\n
279 where, the line ending is a very specific "#\r\n", and the
281 sep_char is a ^ (caret). None of the fields is quoted, but
282 embedded binary data is likely to be present. With the specific
283 line ending, that shouldn't be too hard to detect.
284 By default, Text::CSV_XS' parse function however is instructed
286 to only know about "\n" and "\r" to be legal line endings, and
287 so has to deal with the embedded newline as a real end-of-line,
288 so it can scan the next line if binary is true, and the newline
289 is inside a quoted field. With this attribute however, we can
290 tell parse () to parse the line as if \n is just nothing more
291 than a binary character.
292 For parse () this means that the parser has no idea about line
294 ending anymore, and getline () chomps line endings on reading.
295 To sum it up,
297 $csv = Text::CSV_XS->new ();
299 is equivalent to
301 $csv = Text::CSV_XS->new ({
303 quote_char => '"',
304 escape_char => '"',
305 sep_char => ',',
306 eol => '',
307 always_quote => 0,
308 binary => 0,
309 keep_meta_info => 0,
310 allow_loose_quotes => 0,
311 allow_loose_escapes => 0,
312 allow_whitespace => 0,
313 verbatim => 0,
314 });
315 For all of the above mentioned flags, there is an accessor method
317 available where you can inquire for the current value, or change
318 the value
319 my $quote = $csv->quote_char;
321 $csv->binary (1);
322 It is unwise to change these settings halfway through writing CSV
324 data to a stream. If however, you want to create a new stream using
325 the available CSV object, there is no harm in changing them.
326 combine
328 $status = $csv->combine (@columns);
329 This object function constructs a CSV string from the arguments,
331 returning success or failure. Failure can result from lack of
332 arguments or an argument containing an invalid character. Upon
333 success, "string ()" can be called to retrieve the resultant CSV
334 string. Upon failure, the value returned by "string ()" is unde‐
335 fined and "error_input ()" can be called to retrieve an invalid
336 argument.
337 print
339 $status = $csv->print ($io, $colref);
340 Similar to combine, but it expects an array ref as input (not an
342 array!) and the resulting string is not really created, but imme‐
343 diately written to the $io object, typically an IO handle or any
344 other object that offers a print method. Note, this implies that
345 the following is wrong:
346 open FILE, ">", "whatever";
348 $status = $csv->print (\*FILE, $colref);
349 The glob "\*FILE" is not an object, thus it doesn't have a print
351 method. The solution is to use an IO::File object or to hide the
352 glob behind an IO::Wrap object. See IO::File(3) and IO::Wrap(3) for
353 details.
354 For performance reasons the print method doesn't create a result
356 string. In particular the $csv->string (), $csv->status (),
357 $csv-fields ()> and $csv->error_input () methods are meaningless
358 after executing this method.
359 string
361 $line = $csv->string ();
362 This object function returns the input to "parse ()" or the resul‐
364 tant CSV string of "combine ()", whichever was called more
365 recently.
366 parse
368 $status = $csv->parse ($line);
369 This object function decomposes a CSV string into fields, returning
371 success or failure. Failure can result from a lack of argument or
372 the given CSV string is improperly formatted. Upon success,
373 "fields ()" can be called to retrieve the decomposed fields . Upon
374 failure, the value returned by "fields ()" is undefined and
375 "error_input ()" can be called to retrieve the invalid argument.
376 You may use the types () method for setting column types. See the
378 description below.
379 getline
381 $colref = $csv->getline ($io);
382 This is the counterpart to print, like parse is the counterpart to
384 combine: It reads a row from the IO object $io using $io->getline
385 () and parses this row into an array ref. This array ref is
386 returned by the function or undef for failure.
387 The $csv->string (), $csv->fields () and $csv->status () methods
389 are meaningless, again.
390 eof
392 $eof = $csv->eof ();
393 If "parse ()" or "getline ()" was used with an IO stream, this
395 method will return true (1) if the last call hit end of file, oth‐
396 erwise it will return false (''). This is useful to see the differ‐
397 ence between a failure and end of file.
398 types
400 $csv->types (\@tref);
401 This method is used to force that columns are of a given type. For
403 example, if you have an integer column, two double columns and a
404 string column, then you might do a
405 $csv->types ([Text::CSV_XS::IV (),
407 Text::CSV_XS::NV (),
408 Text::CSV_XS::NV (),
409 Text::CSV_XS::PV ()]);
410 Column types are used only for decoding columns, in other words by
412 the parse () and getline () methods.
413 You can unset column types by doing a
415 $csv->types (undef);
417 or fetch the current type settings with
419 $types = $csv->types ();
421 IV Set field type to integer.
423 NV Set field type to numeric/float.
425 PV Set field type to string.
427 fields
429 @columns = $csv->fields ();
430 This object function returns the input to "combine ()" or the
432 resultant decomposed fields of "parse ()", whichever was called
433 more recently.
434 meta_info
436 @flags = $csv->meta_info ();
437 This object function returns the flags of the input to "combine ()"
439 or the flags of the resultant decomposed fields of "parse ()",
440 whichever was called more recently.
441 For each field, a meta_info field will hold flags that tell some‐
443 thing about the field returned by the "fields ()" method or passed
444 to the "combine ()" method. The flags are bitwise-or'd like:
445 0x0001
447 The field was quoted.
448 0x0002
450 The field was binary.
451 See the "is_*** ()" methods below.
453 is_quoted
455 my $quoted = $csv->is_quoted ($column_idx);
456 Where $column_idx is the (zero-based) index of the column in the
458 last result of "parse ()".
459 This returns a true value if the data in the indicated column was
461 enclosed in "quote_char" quotes. This might be important for data
462 where ",20070108," is to be treated as a numeric value, and where
463 ","20070108"," is explicitly marked as character string data.
464 is_binary
466 my $binary = $csv->is_binary ($column_idx);
467 Where $column_idx is the (zero-based) index of the column in the
469 last result of "parse ()".
470 This returns a true value if the data in the indicated column con‐
472 tained any byte in the range [\x00-\x08,\x10-\x1F,\x7F-\xFF]
473 status
475 $status = $csv->status ();
476 This object function returns success (or failure) of "combine ()"
478 or "parse ()", whichever was called more recently.
479 error_input
481 $bad_argument = $csv->error_input ();
482 This object function returns the erroneous argument (if it exists)
484 of "combine ()" or "parse ()", whichever was called more recently.
485 error_diag
487 $csv->error_diag ();
488 $error_code = 0 + $csv->error_diag ();
489 $error_str = "" . $csv->error_diag ();
490 ($cde, $str) = $csv->error_diag ();
491 If (and only if) an error occured, this function returns the diag‐
493 nostics of that error.
494 If called in void context, it will print the internal error code
496 and the associated error message to STDERR.
497 If called in list context, it will return the error code and the
499 error message in that order.
500 If called in scalar context, it will return the diagnostics in a
502 single scalar, a-la $!. It will contain the error code in numeric
503 context, and the diagnostics message in string context.
504
506 Combine (...)
507 Parse (...)
508 The arguments to these two internal functions are deliberately not
510 described or documented to enable the module author(s) to change it
511 when they feel the need for it and using them is highly discouraged as
512 the API may change in future releases.
513
515 An example for creating CSV files:
516 use Text::CSV_XS;
518 my $csv = Text::CSV_XS->new;
520 open my $csv_fh, ">", "hello.csv" or die "hello.csv: $!";
522 my @sample_input_fields = (
524 'You said, "Hello!"', 5.67,
525 '"Surely"', '', '3.14159');
526 if ($csv->combine (@sample_input_fields)) {
527 my $string = $csv->string;
528 print $csv_fh "$string\n";
529 }
530 else {
531 my $err = $csv->error_input;
532 print "combine () failed on argument: ", $err, "\n";
533 }
534 close $csv_fh;
535 An example for parsing CSV lines:
537 use Text::CSV_XS;
539 my $csv = Text::CSV_XS->new ({ keep_meta_info => 1, binary => 1 });
541 my $sample_input_string =
543 qq{"I said, ""Hi!""",Yes,"",2.34,,"1.09","\x{20ac}",};
544 if ($csv->parse ($sample_input_string)) {
545 my @field = $csv->fields;
546 foreach my $col (0 .. $#field) {
547 my $quo = $csv->is_quoted ($col) ? $csv->{quote_char} : "";
548 printf "%2d: %s%s%s\n", $col, $quo, $field[$col], $quo;
549 }
550 }
551 else {
552 my $err = $csv->error_input;
553 print STDERR "parse () failed on argument: ", $err, "\n";
554 $csv->error_diag ();
555 }
556
558 More Errors & Warnings
559 At current, it is hard to tell where or why an error occured (if at
560 all). New extensions ought to be clear and concise in reporting what
561 error occurred where and why, and possibly also tell a remedy to the
562 problem. error_diag is a (very) good start, but there is more work to
563 be done here.
564 Basic calls should croak or warn on illegal parameters. Errors should
566 be documented.
567 eol
569 Discuss an option to make the eol honor the $/ setting. Maybe
570 my $csv = Text::CSV_XS->new ({ eol => $/ });
572 is already enough, and new options only make things less opaque.
574 setting meta info
576 Future extensions might include extending the "meta_info ()",
577 "is_quoted ()", and "is_binary ()" to accept setting these flags for
578 fields, so you can specify which fields are quoted in the combine
579 ()/string () combination.
580 $csv->meta_info (0, 1, 1, 3, 0, 0);
582 $csv->is_quoted (3, 1);
583 parse returning undefined fields
585 Adding an option that enables the parser to distinguish between empty
586 fields and undefined fields, like
587 $csv->always_quote (1);
589 $csv->allow_undef (1);
590 $csv->parse (qq{,"",1,"2",,""});
591 my @fld = $csv->fields ();
592 Then would return (undef, "", "1", "2", undef, "") in @fld, instead
594 of the current ("", "", "1", "2", "", "").
595 combined methods
597 Requests for adding means (methods) that combine "combine ()" and
598 "string ()" in a single call will not be honored. Likewise for "parse
599 ()" and "fields ()". Given the trouble with embedded newlines, using
600 "getline ()" and "print ()" instead is the prefered way to go.
601 Unicode
603 Make "parse ()" and "combine ()" do the right thing for Unicode
604 (UTF-8) if requested. See t/50_utf8.t. More complicated, but evenly
605 important, also for "getline ()" and "print ()".
606 Probably the best way to do this is to make a subclass
608 Text::CSV_XS::Encoded that can be passed the required encoding and
609 then behaves transparently (but slower).
610 Double double quotes
612 There seem to be applications around that write their dates like
613 1,4,""12/11/2004"",4,1
615 If we would support that, probably through allow_double_quoted Defi‐
617 nitely belongs in t/65_allow.t
618 Parse the whole file at once
620 Implement a new methods that enables the parsing of a complete file
621 at once, returning a lis of hashes. Possible extension to this could
622 be to enable a column selection on the call:
623 my @AoH = $csv->parse_file ($filename, { cols => [ 1, 4..8, 12 ]});
625 Returning something like
627 [ { fields => [ 1, 2, "foo", 4.5, undef, "", 8 ],
629 flags => [ ... ],
630 errors => [ ... ],
631 },
632 { fields => [ ... ],
633 .
634 .
635 },
636 ]
637 EBCDIC
639 The hard-coding of characters and character ranges makes this module
640 unusable on EBCDIC system. Using some #ifdef structure could enable
641 these again without loosing speed. Testing would be the hard part.
642
644 No guarantees, but this is what I have in mind right now:
645 0.31
647 - croak / carp
648 - error cause, check if error_diag () is enough
649 - return undef
650 - DIAGNOSTICS setction in pod to *describe* the errors
651 0.32
653 - allow_double_quoted
654 - Text::CSV_XS::Encoded (maybe)
655 0.33
657 - csv2csv - a script to regenerate a CSV file to follow standards
658 - EBCDIC support
659
661 Still under construction ...
662 If an error occured, $csv->error_diag () can be used to get more infor‐
664 mation on the cause of the failure. Note that for speed reasons, the
665 internal value is never cleared on success, so using the value returned
666 by error_diag () in normal cases - when no error occured - may cause
667 unexpected results.
668 Currently these errors are available:
670 1001 "sep_char is equal to quote_char or escape_char"
672 2010 "ECR - QUO char inside quotes followed by CR not part of EOL"
673 2011 "ECR - Characters after end of quoted field"
674 2012 "EOF - End of data in parsing input stream"
675 2021 "EIQ - NL char inside quotes, binary off"
676 2022 "EIQ - CR char inside quotes, binary off"
677 2023 "EIQ - QUO ..."
678 2024 "EIQ - EOF cannot be escaped, not even inside quotes"
679 2025 "EIQ - Loose unescaped escape"
680 2026 "EIQ - Binary character inside quoted field, binary off"
681 2027 "EIQ - Quoted field not terminated"
682 2030 "EIF - NL char inside unquoted verbatim, binary off"
683 2031 "EIF - CR char is first char of field, not part of EOL"
684 2032 "EIF - CR char inside unquoted, not part of EOL"
685 2033 "EIF - QUO, QUO != ESC, binary off"
686 2034 "EIF - Loose unescaped quote"
687 2035 "EIF - Escaped EOF in unquoted field"
688 2036 "EIF - ESC error"
689 2037 "EIF - Binary character in unquoted field, binary off"
690 2110 "ECB - Binary character in Combine, binary off"
691
693 perl(1), IO::File(3), L{IO::Handle(3)>, IO::Wrap(3), Text::CSV(3),
694 Text::CSV_PP(3). and Spreadsheet::Read(3).
695
697 Alan Citterman <alan@mfgrtl.com> wrote the original Perl module. Please
698 don't send mail concerning Text::CSV_XS to Alan, as he's not involved
699 in the C part which is now the main part of the module.
700 Jochen Wiedmann <joe@ispsoft.de> rewrote the encoding and decoding in C
702 by implementing a simple finite-state machine and added the variable
703 quote, escape and separator characters, the binary mode and the print
704 and getline methods. See ChangeLog releases 0.10 through 0.23.
705 H.Merijn Brand <h.m.brand@xs4all.nl> cleaned up the code, added the
707 field flags methods, wrote the major part of the test suite, completed
708 the documentation, fixed some RT bugs. See ChangeLog releases 0.25 and
709 on.
710
712 Copyright (C) 2007-2007 H.Merijn Brand for PROCURA B.V. Copyright (C)
713 1998-2001 Jochen Wiedmann. All rights reserved. Portions Copyright (C)
714 1997 Alan Citterman. All rights reserved.
715 This library is free software; you can redistribute it and/or modify it
717 under the same terms as Perl itself.
718perl v5.8.8 2007-06-21 CSV_XS(3)