1erl_scan(3) Erlang Module Definition erl_scan(3)
2
6 erl_scan - The Erlang token scanner.
7
9 This module contains functions for tokenizing (scanning) characters
10 into Erlang tokens.
11
13 category() = atom()
14 error_description() = term()
16 error_info() =
18 {erl_anno:location(), module(), error_description()}
19 option() =
21 return |
22 return_white_spaces |
23 return_comments |
24 text |
25 {reserved_word_fun, resword_fun()}
26 options() = option() | [option()]
28 symbol() = atom() | float() | integer() | string()
30 resword_fun() = fun((atom()) -> boolean())
32 token() =
34 {category(), Anno :: erl_anno:anno(), symbol()} |
35 {category(), Anno :: erl_anno:anno()}
36 tokens() = [token()]
38 tokens_result() =
40 {ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
41 {eof, EndLocation :: erl_anno:location()} |
42 {error,
43 ErrorInfo :: error_info(),
44 EndLocation :: erl_anno:location()}
45
47 category(Token) -> category()
48 Types:
50 Token = token()
52 Returns the category of Token.
54 column(Token) -> erl_anno:column() | undefined
56 Types:
58 Token = token()
60 Returns the column of Token's collection of annotations.
62 end_location(Token) -> erl_anno:location() | undefined
64 Types:
66 Token = token()
68 Returns the end location of the text of Token's collection of
70 annotations. If there is no text, undefined is returned.
71 format_error(ErrorDescriptor) -> string()
73 Types:
75 ErrorDescriptor = error_description()
77 Uses an ErrorDescriptor and returns a string that describes the
79 error or warning. This function is usually called implicitly
80 when an ErrorInfo structure is processed (see section Error
81 Information).
82 line(Token) -> erl_anno:line()
84 Types:
86 Token = token()
88 Returns the line of Token's collection of annotations.
90 location(Token) -> erl_anno:location()
92 Types:
94 Token = token()
96 Returns the location of Token's collection of annotations.
98 reserved_word(Atom :: atom()) -> boolean()
100 Returns true if Atom is an Erlang reserved word, otherwise
102 false.
103 string(String) -> Return
105 string(String, StartLocation) -> Return
107 string(String, StartLocation, Options) -> Return
109 Types:
111 String = string()
113 Options = options()
114 Return =
115 {ok, Tokens :: tokens(), EndLocation} |
116 {error, ErrorInfo :: error_info(), ErrorLocation}
117 StartLocation = EndLocation = ErrorLocation = erl_anno:loca‐
118 tion()
119 Takes the list of characters String and tries to scan (tokenize)
121 them. Returns one of the following:
122 {ok, Tokens, EndLocation}:
124 Tokens are the Erlang tokens from String. EndLocation is the
125 first location after the last token.
126 {error, ErrorInfo, ErrorLocation}:
128 An error occurred. ErrorLocation is the first location after
129 the erroneous token.
130 string(String) is equivalent to string(String, 1), and
132 string(String, StartLocation) is equivalent to string(String,
133 StartLocation, []).
134 StartLocation indicates the initial location when scanning
136 starts. If StartLocation is a line, Anno, EndLocation, and
137 ErrorLocation are lines. If StartLocation is a pair of a line
138 and a column, Anno takes the form of an opaque compound data
139 type, and EndLocation and ErrorLocation are pairs of a line and
140 a column. The token annotations contain information about the
141 column and the line where the token begins, as well as the text
142 of the token (if option text is specified), all of which can be
143 accessed by calling column/1, line/1, location/1, and text/1.
144 A token is a tuple containing information about syntactic cate‐
146 gory, the token annotations, and the terminal symbol. For punc‐
147 tuation characters (such as ; and |) and reserved words, the
148 category and the symbol coincide, and the token is represented
149 by a two-tuple. Three-tuples have one of the following forms:
150 * {atom, Anno, atom()}
152 * {char, Anno, char()}
154 * {comment, Anno, string()}
156 * {float, Anno, float()}
158 * {integer, Anno, integer()}
160 * {var, Anno, atom()}
162 * {white_space, Anno, string()}
164 Valid options:
166 {reserved_word_fun, reserved_word_fun()}:
168 A callback function that is called when the scanner has
169 found an unquoted atom. If the function returns true, the
170 unquoted atom itself becomes the category of the token. If
171 the function returns false, atom becomes the category of the
172 unquoted atom.
173 return_comments:
175 Return comment tokens.
176 return_white_spaces:
178 Return white space tokens. By convention, a newline charac‐
179 ter, if present, is always the first character of the text
180 (there cannot be more than one newline in a white space
181 token).
182 return:
184 Short for [return_comments, return_white_spaces].
185 text:
187 Include the token text in the token annotation. The text is
188 the part of the input corresponding to the token.
189 symbol(Token) -> symbol()
191 Types:
193 Token = token()
195 Returns the symbol of Token.
197 text(Token) -> erl_anno:text() | undefined
199 Types:
201 Token = token()
203 Returns the text of Token's collection of annotations. If there
205 is no text, undefined is returned.
206 tokens(Continuation, CharSpec, StartLocation) -> Return
208 tokens(Continuation, CharSpec, StartLocation, Options) -> Return
210 Types:
212 Continuation = return_cont() | []
214 CharSpec = char_spec()
215 StartLocation = erl_anno:location()
216 Options = options()
217 Return =
218 {done,
219 Result :: tokens_result(),
220 LeftOverChars :: char_spec()} |
221 {more, Continuation1 :: return_cont()}
222 char_spec() = string() | eof
223 return_cont()
224 An opaque continuation.
225 This is the re-entrant scanner, which scans characters until
227 either a dot ('.' followed by a white space) or eof is reached.
228 It returns:
229 {done, Result, LeftOverChars}:
231 Indicates that there is sufficient input data to get a
232 result. Result is:
233 {ok, Tokens, EndLocation}:
235 The scanning was successful. Tokens is the list of tokens
236 including dot.
237 {eof, EndLocation}:
239 End of file was encountered before any more tokens.
240 {error, ErrorInfo, EndLocation}:
242 An error occurred. LeftOverChars is the remaining charac‐
243 ters of the input data, starting from EndLocation.
244 {more, Continuation1}:
246 More data is required for building a term. Continuation1
247 must be passed in a new call to tokens/3,4 when more data is
248 available.
249 The CharSpec eof signals end of file. LeftOverChars then takes
251 the value eof as well.
252 tokens(Continuation, CharSpec, StartLocation) is equivalent to
254 tokens(Continuation, CharSpec, StartLocation, []).
255 For a description of the options, see string/3.
257
259 ErrorInfo is the standard ErrorInfo structure that is returned from all
260 I/O modules. The format is as follows:
261 {ErrorLocation, Module, ErrorDescriptor}
263 A string describing the error is obtained with the following call:
265 Module:format_error(ErrorDescriptor)
267
269 The continuation of the first call to the re-entrant input functions
270 must be []. For a complete description of how the re-entrant input
271 scheme works, see Armstrong, Virding and Williams: 'Concurrent Program‐
272 ming in Erlang', Chapter 13.
273
275 erl_anno(3), erl_parse(3), io(3)
276Ericsson AB stdlib 3.4.5.1 erl_scan(3)