1erl_scan(3) Erlang Module Definition erl_scan(3)
2
6 erl_scan - The Erlang token scanner.
7
9 This module contains functions for tokenizing (scanning) characters
10 into Erlang tokens.
11
13 category() = atom()
14 error_description() = term()
16 error_info() =
18 {erl_anno:location(), module(), error_description()}
19 option() =
21 return | return_white_spaces | return_comments | text |
22 {reserved_word_fun, resword_fun()}
23 options() = option() | [option()]
25 symbol() = atom() | float() | integer() | string()
27 resword_fun() = fun((atom()) -> boolean())
29 token() =
31 {category(), Anno :: erl_anno:anno(), symbol()} |
32 {category(), Anno :: erl_anno:anno()}
33 tokens() = [token()]
35 tokens_result() =
37 {ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
38 {eof, EndLocation :: erl_anno:location()} |
39 {error,
40 ErrorInfo :: error_info(),
41 EndLocation :: erl_anno:location()}
42
44 category(Token) -> category()
45 Types:
47 Token = token()
49 Returns the category of Token.
51 column(Token) -> erl_anno:column() | undefined
53 Types:
55 Token = token()
57 Returns the column of Token's collection of annotations.
59 end_location(Token) -> erl_anno:location() | undefined
61 Types:
63 Token = token()
65 Returns the end location of the text of Token's collection of
67 annotations. If there is no text, undefined is returned.
68 format_error(ErrorDescriptor) -> string()
70 Types:
72 ErrorDescriptor = error_description()
74 Uses an ErrorDescriptor and returns a string that describes the
76 error or warning. This function is usually called implicitly
77 when an ErrorInfo structure is processed (see section Error In‐
78 formation).
79 line(Token) -> erl_anno:line()
81 Types:
83 Token = token()
85 Returns the line of Token's collection of annotations.
87 location(Token) -> erl_anno:location()
89 Types:
91 Token = token()
93 Returns the location of Token's collection of annotations.
95 reserved_word(Atom :: atom()) -> boolean()
97 Returns true if Atom is an Erlang reserved word, otherwise
99 false.
100 string(String) -> Return
102 string(String, StartLocation) -> Return
104 string(String, StartLocation, Options) -> Return
106 Types:
108 String = string()
110 Options = options()
111 Return =
112 {ok, Tokens :: tokens(), EndLocation} |
113 {error, ErrorInfo :: error_info(), ErrorLocation}
114 StartLocation = EndLocation = ErrorLocation = erl_anno:loca‐
115 tion()
116 Takes the list of characters String and tries to scan (tokenize)
118 them. Returns one of the following:
119 {ok, Tokens, EndLocation}:
121 Tokens are the Erlang tokens from String. EndLocation is the
122 first location after the last token.
123 {error, ErrorInfo, ErrorLocation}:
125 An error occurred. ErrorLocation is the first location after
126 the erroneous token.
127 string(String) is equivalent to string(String, 1), and
129 string(String, StartLocation) is equivalent to string(String,
130 StartLocation, []).
131 StartLocation indicates the initial location when scanning
133 starts. If StartLocation is a line, Anno, EndLocation, and Er‐
134 rorLocation are lines. If StartLocation is a pair of a line and
135 a column, Anno takes the form of an opaque compound data type,
136 and EndLocation and ErrorLocation are pairs of a line and a col‐
137 umn. The token annotations contain information about the column
138 and the line where the token begins, as well as the text of the
139 token (if option text is specified), all of which can be ac‐
140 cessed by calling column/1, line/1, location/1, and text/1.
141 A token is a tuple containing information about syntactic cate‐
143 gory, the token annotations, and the terminal symbol. For punc‐
144 tuation characters (such as ; and |) and reserved words, the
145 category and the symbol coincide, and the token is represented
146 by a two-tuple. Three-tuples have one of the following forms:
147 * {atom, Anno, atom()}
149 * {char, Anno, char()}
151 * {comment, Anno, string()}
153 * {float, Anno, float()}
155 * {integer, Anno, integer()}
157 * {var, Anno, atom()}
159 * {white_space, Anno, string()}
161 Valid options:
163 {reserved_word_fun, reserved_word_fun()}:
165 A callback function that is called when the scanner has
166 found an unquoted atom. If the function returns true, the
167 unquoted atom itself becomes the category of the token. If
168 the function returns false, atom becomes the category of the
169 unquoted atom.
170 return_comments:
172 Return comment tokens.
173 return_white_spaces:
175 Return white space tokens. By convention, a newline charac‐
176 ter, if present, is always the first character of the text
177 (there cannot be more than one newline in a white space to‐
178 ken).
179 return:
181 Short for [return_comments, return_white_spaces].
182 text:
184 Include the token text in the token annotation. The text is
185 the part of the input corresponding to the token.
186 symbol(Token) -> symbol()
188 Types:
190 Token = token()
192 Returns the symbol of Token.
194 text(Token) -> erl_anno:text() | undefined
196 Types:
198 Token = token()
200 Returns the text of Token's collection of annotations. If there
202 is no text, undefined is returned.
203 tokens(Continuation, CharSpec, StartLocation) -> Return
205 tokens(Continuation, CharSpec, StartLocation, Options) -> Return
207 Types:
209 Continuation = return_cont() | []
211 CharSpec = char_spec()
212 StartLocation = erl_anno:location()
213 Options = options()
214 Return =
215 {done,
216 Result :: tokens_result(),
217 LeftOverChars :: char_spec()} |
218 {more, Continuation1 :: return_cont()}
219 char_spec() = string() | eof
220 return_cont()
221 An opaque continuation.
222 This is the re-entrant scanner, which scans characters until ei‐
224 ther a dot ('.' followed by a white space) or eof is reached. It
225 returns:
226 {done, Result, LeftOverChars}:
228 Indicates that there is sufficient input data to get a re‐
229 sult. Result is:
230 {ok, Tokens, EndLocation}:
232 The scanning was successful. Tokens is the list of tokens
233 including dot.
234 {eof, EndLocation}:
236 End of file was encountered before any more tokens.
237 {error, ErrorInfo, EndLocation}:
239 An error occurred. LeftOverChars is the remaining charac‐
240 ters of the input data, starting from EndLocation.
241 {more, Continuation1}:
243 More data is required for building a term. Continuation1
244 must be passed in a new call to tokens/3,4 when more data is
245 available.
246 The CharSpec eof signals end of file. LeftOverChars then takes
248 the value eof as well.
249 tokens(Continuation, CharSpec, StartLocation) is equivalent to
251 tokens(Continuation, CharSpec, StartLocation, []).
252 For a description of the options, see string/3.
254
256 ErrorInfo is the standard ErrorInfo structure that is returned from all
257 I/O modules. The format is as follows:
258 {ErrorLocation, Module, ErrorDescriptor}
260 A string describing the error is obtained with the following call:
262 Module:format_error(ErrorDescriptor)
264
266 The continuation of the first call to the re-entrant input functions
267 must be []. For a complete description of how the re-entrant input
268 scheme works, see Armstrong, Virding and Williams: 'Concurrent Program‐
269 ming in Erlang', Chapter 13.
270
272 erl_anno(3), erl_parse(3), io(3)
273Ericsson AB stdlib 3.16.1 erl_scan(3)