1SQL::Tokenizer(3)     User Contributed Perl Documentation    SQL::Tokenizer(3)
2
3
4

NAME

6       SQL::Tokenizer - A simple SQL tokenizer.
7

VERSION

9       0.20
10

SYNOPSIS

12        use SQL::Tokenizer qw(tokenize_sql);
13
14        my $query= q{SELECT 1 + 1};
15        my @tokens= SQL::Tokenizer->tokenize($query);
16
17        # @tokens now contains ('SELECT', ' ', '1', ' ', '+', ' ', '1')
18
19        @tokens= tokenize_sql($query); # procedural interface
20

DESCRIPTION

22       SQL::Tokenizer is a simple tokenizer for SQL queries. It does not claim
23       to be a parser or query verifier. It just creates sane tokens from a
24       valid SQL query.
25
26       It supports SQL with comments like:
27
28        -- This query is used to insert a message into
29        -- logs table
30        INSERT INTO log (application, message) VALUES (?, ?)
31
32       Also supports '', "" and "\'" escaping methods, so tokenizing queries
33       like the one below should not be a problem:
34
35        INSERT INTO log (application, message)
36        VALUES ('myapp', 'Hey, this is a ''single quoted string''!')
37

API

39       tokenize_sql
40            use SQL::Tokenizer qw(tokenize_sql);
41
42            my @tokens= tokenize_sql($query);
43            my $tokens= tokenize_sql($query);
44
45            $tokens= tokenize_sql( $query, $remove_white_tokens );
46
47           "tokenize_sql" can be imported to current namespace on request. It
48           receives a SQL query, and returns an array of tokens if called in
49           list context, or an arrayref if called in scalar context.
50
51       tokenize
52            my @tokens= SQL::Tokenizer->tokenize($query);
53            my $tokens= SQL::Tokenizer->tokenize($query);
54
55            $tokens= SQL::Tokenizer->tokenize( $query, $remove_white_tokens );
56
57           This is the only available class method. It receives a SQL query,
58           and returns an array of tokens if called in list context, or an
59           arrayref if called in scalar context.
60
61           If $remove_white_tokens is true, white spaces only tokens will be
62           removed from result.
63

ACKNOWLEDGEMENTS

65       •   Evan Harris, for implementing Shell comment style and SQL
66           operators.
67
68       •   Charlie Hills, for spotting a lot of important issues I haven't
69           thought.
70
71       •   Jonas Kramer, for fixing MySQL quoted strings and treating dot as
72           punctuation character correctly.
73
74       •   Emanuele Zeppieri, for asking to fix SQL::Tokenizer to support
75           dollars as well.
76
77       •   Nigel Metheringham, for extending the dollar signal support.
78
79       •   Devin Withers, for making it not choke on CR+LF in comments.
80
81       •   Luc Lanthier, for simplifying the regex and make it not choke on
82           backslashes.
83

AUTHOR

85       Copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes
86       "<IZUT@cpan.org>". All rights reserved.
87
88       This module is free software; you can redistribute it and/or modify it
89       under the same terms as Perl itself.
90
91
92
93perl v5.38.0                      2023-07-21                 SQL::Tokenizer(3)