1erl_scan(3) Erlang Module Definition erl_scan(3)
2
6 erl_scan - The Erlang token scanner.
7
9 This module contains functions for tokenizing (scanning) characters
10 into Erlang tokens.
11
13 category() = atom()
14 error_description() = term()
16 error_info() =
18 {erl_anno:location(), module(), error_description()}
19 option() =
21 return | return_white_spaces | return_comments | text |
22 {reserved_word_fun, resword_fun()} |
23 {text_fun, text_fun()}
24 options() = option() | [option()]
26 symbol() = atom() | float() | integer() | string()
28 resword_fun() = fun((atom()) -> boolean())
30 token() =
32 {category(), Anno :: erl_anno:anno(), symbol()} |
33 {category(), Anno :: erl_anno:anno()}
34 tokens() = [token()]
36 tokens_result() =
38 {ok, Tokens :: tokens(), EndLocation :: erl_anno:location()} |
39 {eof, EndLocation :: erl_anno:location()} |
40 {error,
41 ErrorInfo :: error_info(),
42 EndLocation :: erl_anno:location()}
43 text_fun() = fun((atom(), string()) -> boolean())
45
47 category(Token) -> category()
48 Types:
50 Token = token()
52 Returns the category of Token.
54 column(Token) -> erl_anno:column() | undefined
56 Types:
58 Token = token()
60 Returns the column of Token's collection of annotations.
62 end_location(Token) -> erl_anno:location() | undefined
64 Types:
66 Token = token()
68 Returns the end location of the text of Token's collection of
70 annotations. If there is no text, undefined is returned.
71 format_error(ErrorDescriptor) -> string()
73 Types:
75 ErrorDescriptor = error_description()
77 Uses an ErrorDescriptor and returns a string that describes the
79 error or warning. This function is usually called implicitly
80 when an ErrorInfo structure is processed (see section Error In‐
81 formation).
82 line(Token) -> erl_anno:line()
84 Types:
86 Token = token()
88 Returns the line of Token's collection of annotations.
90 location(Token) -> erl_anno:location()
92 Types:
94 Token = token()
96 Returns the location of Token's collection of annotations.
98 reserved_word(Atom :: atom()) -> boolean()
100 Returns true if Atom is an Erlang reserved word, otherwise
102 false.
103 string(String) -> Return
105 string(String, StartLocation) -> Return
107 string(String, StartLocation, Options) -> Return
109 Types:
111 String = string()
113 Options = options()
114 Return =
115 {ok, Tokens :: tokens(), EndLocation} |
116 {error, ErrorInfo :: error_info(), ErrorLocation}
117 StartLocation = EndLocation = ErrorLocation = erl_anno:loca‐
118 tion()
119 Takes the list of characters String and tries to scan (tokenize)
121 them. Returns one of the following:
122 {ok, Tokens, EndLocation}:
124 Tokens are the Erlang tokens from String. EndLocation is the
125 first location after the last token.
126 {error, ErrorInfo, ErrorLocation}:
128 An error occurred. ErrorLocation is the first location after
129 the erroneous token.
130 string(String) is equivalent to string(String, 1), and
132 string(String, StartLocation) is equivalent to string(String,
133 StartLocation, []).
134 StartLocation indicates the initial location when scanning
136 starts. If StartLocation is a line, Anno, EndLocation, and Er‐
137 rorLocation are lines. If StartLocation is a pair of a line and
138 a column, Anno takes the form of an opaque compound data type,
139 and EndLocation and ErrorLocation are pairs of a line and a col‐
140 umn. The token annotations contain information about the column
141 and the line where the token begins, as well as the text of the
142 token (if option text is specified), all of which can be ac‐
143 cessed by calling column/1, line/1, location/1, and text/1.
144 A token is a tuple containing information about syntactic cate‐
146 gory, the token annotations, and the terminal symbol. For punc‐
147 tuation characters (such as ; and |) and reserved words, the
148 category and the symbol coincide, and the token is represented
149 by a two-tuple. Three-tuples have one of the following forms:
150 * {atom, Anno, atom()}
152 * {char, Anno, char()}
154 * {comment, Anno, string()}
156 * {float, Anno, float()}
158 * {integer, Anno, integer()}
160 * {var, Anno, atom()}
162 * {white_space, Anno, string()}
164 Valid options:
166 {reserved_word_fun, reserved_word_fun()}:
168 A callback function that is called when the scanner has
169 found an unquoted atom. If the function returns true, the
170 unquoted atom itself becomes the category of the token. If
171 the function returns false, atom becomes the category of the
172 unquoted atom.
173 return_comments:
175 Return comment tokens.
176 return_white_spaces:
178 Return white space tokens. By convention, a newline charac‐
179 ter, if present, is always the first character of the text
180 (there cannot be more than one newline in a white space to‐
181 ken).
182 return:
184 Short for [return_comments, return_white_spaces].
185 text:
187 Include the token text in the token annotation. The text is
188 the part of the input corresponding to the token. See also
189 text_fun.
190 {text_fun, text_fun()}:
192 A callback function used to determine whether the full text
193 for the token shall be included in the token annotation. Ar‐
194 guments of the function are the category of the token and
195 the full token string. This is only used when text is not
196 present. If neither are present the text will not be saved
197 in the token annotation.
198 symbol(Token) -> symbol()
200 Types:
202 Token = token()
204 Returns the symbol of Token.
206 text(Token) -> erl_anno:text() | undefined
208 Types:
210 Token = token()
212 Returns the text of Token's collection of annotations. If there
214 is no text, undefined is returned.
215 tokens(Continuation, CharSpec, StartLocation) -> Return
217 tokens(Continuation, CharSpec, StartLocation, Options) -> Return
219 Types:
221 Continuation = return_cont() | []
223 CharSpec = char_spec()
224 StartLocation = erl_anno:location()
225 Options = options()
226 Return =
227 {done,
228 Result :: tokens_result(),
229 LeftOverChars :: char_spec()} |
230 {more, Continuation1 :: return_cont()}
231 char_spec() = string() | eof
232 return_cont()
233 An opaque continuation.
234 This is the re-entrant scanner, which scans characters until ei‐
236 ther a dot ('.' followed by a white space) or eof is reached. It
237 returns:
238 {done, Result, LeftOverChars}:
240 Indicates that there is sufficient input data to get a re‐
241 sult. Result is:
242 {ok, Tokens, EndLocation}:
244 The scanning was successful. Tokens is the list of tokens
245 including dot.
246 {eof, EndLocation}:
248 End of file was encountered before any more tokens.
249 {error, ErrorInfo, EndLocation}:
251 An error occurred. LeftOverChars is the remaining charac‐
252 ters of the input data, starting from EndLocation.
253 {more, Continuation1}:
255 More data is required for building a term. Continuation1
256 must be passed in a new call to tokens/3,4 when more data is
257 available.
258 The CharSpec eof signals end of file. LeftOverChars then takes
260 the value eof as well.
261 tokens(Continuation, CharSpec, StartLocation) is equivalent to
263 tokens(Continuation, CharSpec, StartLocation, []).
264 For a description of the options, see string/3.
266
268 ErrorInfo is the standard ErrorInfo structure that is returned from all
269 I/O modules. The format is as follows:
270 {ErrorLocation, Module, ErrorDescriptor}
272 A string describing the error is obtained with the following call:
274 Module:format_error(ErrorDescriptor)
276
278 The continuation of the first call to the re-entrant input functions
279 must be []. For a complete description of how the re-entrant input
280 scheme works, see Armstrong, Virding and Williams: 'Concurrent Program‐
281 ming in Erlang', Chapter 13.
282
284 erl_anno(3), erl_parse(3), io(3)
285Ericsson AB stdlib 4.2 erl_scan(3)