1SQL::SplitStatement::ToUkseenrizCeorn(t3r)ibuted Perl DoScQuLm:e:nStpaltiitoSntatement::Tokenizer(3)
2
3
4

NAME

6       SQL::SplitStatement::Tokenizer - A simple SQL tokenizer.
7

SYNOPSIS

9        use SQL::SplitStatement::Tokenizer qw(tokenize_sql);
10
11        my $query= q{SELECT 1 + 1};
12        my @tokens= tokenize_sql($query);
13
14        # @tokens now contains ('SELECT', ' ', '1', ' ', '+', ' ', '1')
15

DESCRIPTION

17       SQL::SplitStatement::Tokenizer is a simple tokenizer for SQL queries.
18       It does not claim to be a parser or query verifier. It just creates
19       sane tokens from a valid SQL query.
20
21       It supports SQL with comments like:
22
23        -- This query is used to insert a message into
24        -- logs table
25        INSERT INTO log (application, message) VALUES (?, ?)
26
27       Also supports '', "" and "\'" escaping methods, so tokenizing queries
28       like the one below should not be a problem:
29
30        INSERT INTO log (application, message)
31        VALUES ('myapp', 'Hey, this is a ''single quoted string''!')
32

API

34       tokenize_sql
35               use SQL::SplitStatement::Tokenizer qw(tokenize_sql);
36
37               my @tokens = tokenize_sql($query);
38               my $tokens = tokenize_sql($query);
39
40               $tokens = tokenize_sql( $query, $remove_white_tokens );
41
42           "tokenize_sql" can be imported to current namespace on request. It
43           receives a SQL query, and returns an array of tokens if called in
44           list context, or an arrayref if called in scalar context.
45
46           If $remove_white_tokens is true, white spaces only tokens will be
47           removed from result.
48

ACKNOWLEDGEMENTS

50       •   Igor Sutton Lopes for writing SQL::Tokenizer, which this was forked
51           from.
52
53       •   Evan Harris, for implementing Shell comment style and SQL
54           operators.
55
56       •   Charlie Hills, for spotting a lot of important issues I haven't
57           thought.
58
59       •   Jonas Kramer, for fixing MySQL quoted strings and treating dot as
60           punctuation character correctly.
61
62       •   Emanuele Zeppieri, for asking to fix SQL::Tokenizer to support
63           dollars as well.
64
65       •   Nigel Metheringham, for extending the dollar signal support.
66
67       •   Devin Withers, for making it not choke on CR+LF in comments.
68
69       •   Luc Lanthier, for simplifying the regex and make it not choke on
70           backslashes.
71

AUTHOR

73       Copyright (c) 2007, 2008, 2009, 2010, 2011 Igor Sutton Lopes
74       "<IZUT@cpan.org>". All rights reserved.
75
76       Copyright (c) 2021 Veesh Goldman "<veesh@cpan.org>"
77
78       This module is free software; you can redistribute it and/or modify it
79       under the same terms as Perl itself.
80
81
82
83perl v5.34.0                      2021-07-22 SQL::SplitStatement::Tokenizer(3)
Impressum