1AnyEvent::IO(3) User Contributed Perl Documentation AnyEvent::IO(3)
2
6 AnyEvent::IO - the DBI of asynchronous I/O implementations
7
9 use AnyEvent::IO;
10 # load /etc/passwd, call callback with the file data when done.
12 aio_load "/etc/passwd", sub {
13 my ($data) = @_
14 or return AE::log error => "/etc/passwd: $!";
15 warn "/etc/passwd contains ", ($data =~ y/://) , " colons.\n";
17 };
18 # the rest of the SYNOPSIS does the same, but with individual I/O calls
20 # also import O_XXX flags
22 use AnyEvent::IO qw(:DEFAULT :flags);
23 my $filedata = AE::cv;
25 # first open the file
27 aio_open "/etc/passwd", O_RDONLY, 0, sub {
28 my ($fh) = @_
29 or return AE::log error => "/etc/passwd: $!";
30 # now stat the file to get the size
32 aio_stat $fh, sub {
33 @_
34 or return AE::log error => "/etc/passwd: $!";
35 my $size = -s _;
37 # now read all the file data
39 aio_read $fh, $size, sub {
40 my ($data) = @_
41 or return AE::log error => "/etc/passwd: $!";
42 $size == length $data
44 or return AE::log error => "/etc/passwd: short read, file changed?";
45 # mostly the same as aio_load, above - $data contains
47 # the file contents now.
48 $filedata->($data);
49 };
50 };
51 };
52 my $passwd = $filedata->recv;
54 warn length $passwd, " octets.\n";
55
57 This module provides functions that do I/O in an asynchronous fashion.
58 It is to I/O the same as AnyEvent is to event libraries - it only
59 interfaces to other implementations or to a portable pure-perl
60 implementation (which does not, however, do asynchronous I/O).
61 The only other implementation that is supported (or even known to the
63 author) is IO::AIO, which is used automatically when it can be loaded
64 (via AnyEvent::AIO, which also needs to be installed). If it is not
65 available, then AnyEvent::IO falls back to its synchronous pure-perl
66 implementation.
67 Unlike AnyEvent, which model to use is currently decided at module load
69 time, not at first use. Future releases might change this.
70 RATIONALE
72 While disk I/O often seems "instant" compared to, say, socket I/O,
73 there are many situations where your program can block for extended
74 time periods when doing disk I/O. For example, you access a disk on an
75 NFS server and it is gone - can take ages to respond again, if ever. Or
76 your system is extremely busy because it creates or restores a backup -
77 reading data from disk can then take seconds. Or you use Linux, which
78 for so many years has a close-to-broken VM/IO subsystem that can often
79 induce minutes or more of delay for disk I/O, even under what I would
80 consider light I/O loads.
81 Whatever the situation, some programs just can't afford to block for
83 long times (say, half a second or more), because they need to respond
84 as fast as possible.
85 For those cases, you need asynchronous I/O.
87 The problem is, AnyEvent itself sometimes reads disk files (for
89 example, when looking at /etc/hosts), and under the above situations,
90 this can bring your program to a complete halt even if your program
91 otherwise takes care to only use asynchronous I/O for everything (e.g.
92 by using IO::AIO).
93 On the other hand, requiring IO::AIO for AnyEvent is clearly
95 impossible, as AnyEvent promises to stay pure-perl, and the overhead of
96 IO::AIO for small programs would be immense, especially when
97 asynchronous I/O isn't even needed.
98 Clearly, this calls for an abstraction layer, and that is what you are
100 looking at right now :-)
101 ASYNCHRONOUS VS. NON-BLOCKING
103 Many people are continuously confused on what the difference is between
104 asynchronous I/O and non-blocking I/O. In fact, those two terms are not
105 well defined, which often makes it hard to even talk about the
106 difference. Here is a short guideline that should leave you less
107 confused. It only talks about read operations, but the reasoning works
108 with other I/O operations as well.
109 Non-blocking I/O means that data is delivered by some external means,
111 automatically - that is, something pushes data towards your file
112 handle, without you having to do anything. Non-blocking means that if
113 your operating system currently has no data (or EOF, or some error)
114 available for you, it will not wait ("block") as it would normally do,
115 but immediately return with an error (e.g. "EWOULDBLOCK" - "I would
116 have blocked, but you forbid it").
117 Your program can then wait for data to arrive by other means, for
119 example, an I/O watcher which tells you when to re-attempt the read,
120 after which it can try to read again, and so on.
121 Often, you would expect this to work for disk files as well - if the
123 data isn't already in memory, one might want to wait for it and then
124 re-attempt the read for example. While this is sound reasoning, the
125 POSIX API does not support this, because disk drives and file systems
126 do not send data "on their own", and more so, the OS already knows that
127 data is there, it doesn't need to "wait" until it arrives from some
128 external entity, it only needs to transfer the data from disk to your
129 memory buffer.
130 So basically, while the concept is sound, the existing OS APIs do not
132 support this. Therefore, it makes no sense to switch a disk file handle
133 into non-blocking mode - it will behave exactly the same as in blocking
134 mode, namely it will block until the data has been read from the disk.
135 The alternative to non-blocking I/O that actually works with disk files
137 is usually called asynchronous I/O. Asynchronous, because the actual
138 I/O is done while your program does something else: there is no need to
139 call the read function to see if data is there, you only order the read
140 once, and it will notify you when the read has finished and the data is
141 your buffer - all the work is done in the background.
142 This works with disk files, and even with sockets and other sources. It
144 is, however, not very efficient when used with sources that could be
145 driven in a non-blocking way, because it usually has higher overhead in
146 the OS than non-blocking I/O, because it ties memory buffers for a
147 potentially unlimited time and often only a limited number of
148 operations can be done in parallel.
149 That's why asynchronous I/O makes most sense when confronted with disk
151 files, and non-blocking I/O only makes sense with sockets, pipes and
152 similar streaming sources.
153
155 By default, this module exports all "aio_"xxx functions. In addition,
156 the following import tags can be used:
157 :aio all aio_* functions, same as :DEFAULT
159 :flags the fcntl open flags (O_CREAT, O_RDONLY, ...)
160
162 The functions in this module are not meant to be the most versatile or
163 the highest-performers (they are not very slow either, of course). They
164 are primarily meant to give users of your code the option to do the I/O
165 asynchronously (by installing IO::AIO and AnyEvent::AIO), without
166 adding a dependency on those modules.
167 NAMING
169 All the functions in this module implement an I/O operation, usually
170 with the same or similar name as the Perl built-in that they mimic, but
171 with an "aio_" prefix. If you like you can think of the "aio_"xxx
172 functions as "AnyEvent I/O" or "Asynchronous I/O" variants of Perl
173 built-ins.
174 CALLING CONVENTIONS AND ERROR REPORTING
176 Each function expects a callback as their last argument. The callback
177 is usually called with the result data or result code. An error is
178 usually signalled by passing no arguments to the callback, which is
179 then free to look at $! for the error code.
180 This makes all of the following forms of error checking valid:
182 aio_open ...., sub {
184 my $fh = shift # scalar assignment - will assign undef on error
185 or return AE::log error => "...";
186 my ($fh) = @_ # list assignment - will be 0 elements on error
188 or return AE::log error => "...";
189 @_ # check the number of elements directly
191 or return AE::log error => "...";
192 CAVEAT: RELATIVE PATHS
194 When a path is specified, this path must be an absolute path, unless
195 you make certain that nothing in your process calls "chdir" or an
196 equivalent function while the request executes.
197 CAVEAT: OTHER SHARED STATE
199 Changing the "umask" while any requests execute that create files (or
200 otherwise rely on the current umask) results in undefined behaviour -
201 likewise changing anything else that would change the outcome, such as
202 your effective user or group ID.
203 CALLBACKS MIGHT BE CALLED BEFORE FUNCTION RETURNS TO CALLER
205 Unlike other functions in the AnyEvent module family, these functions
206 may call your callback instantly, before returning. This should not be
207 a real problem, as these functions never return anything useful.
208 BEHAVIOUR AT PROGRAM EXIT
210 Both AnyEvent::IO::Perl and AnyEvent::IO::IOAIO implementations make
211 sure that operations that have started will be finished on a clean
212 programs exit. That makes programs work that start some I/O operations
213 and then exit. For example this complete program:
214 use AnyEvent::IO;
216 aio_stat "path1", sub {
218 aio_stat "path2", sub {
219 warn "both stats done\n";
220 };
221 };
222 Starts a "stat" operation and then exits by "falling off the end" of
224 the program. Nevertheless, both "stat" operations will be executed, as
225 AnyEvent::IO waits for all outstanding requests to finish and you can
226 start new requests from request callbacks.
227 In fact, since AnyEvent::IO::Perl is currently synchronous, the program
229 will do both stats before falling off the end, but with
230 AnyEvent::IO::IOAIO, the program first falls of the end, then the stats
231 are executed.
232 While not guaranteed, this behaviour will be present in future
234 versions, if reasonably possible (which is extreemly likely :).
235
237 $AnyEvent::IO::MODEL
238 Contains the package name of the backend I/O model in use - at the
239 moment, this is usually "AnyEvent::IO::Perl" or
240 "AnyEvent::IO::IOAIO".
241 aio_load $path, $cb->($data)
243 Tries to open $path and read its contents into memory (obviously,
244 should only be used on files that are "small enough"), then passes
245 them to the callback as a string.
246 Example: load /etc/hosts.
248 aio_load "/etc/hosts", sub {
250 my ($hosts) = @_
251 or return AE::log error => "/etc/hosts: $!";
252 AE::log info => "/etc/hosts contains ", ($hosts =~ y/\n/), " lines\n";
254 };
255 aio_open $path, $flags, $mode, $cb->($fh)
257 Tries to open the file specified by $path with the O_XXX-flags
258 $flags (from the Fcntl module, or see below) and the mode $mode (a
259 good value is 0666 for "O_CREAT", and 0 otherwise).
260 The (normal, standard, perl) file handle associated with the opened
262 file is then passed to the callback.
263 This works very much like Perl's "sysopen" function.
265 Changing the "umask" while this request executes results in
267 undefined behaviour - likewise changing anything else that would
268 change the outcome, such as your effective user or group ID.
269 To avoid having to load Fcntl, this module provides constants for
271 "O_RDONLY", "O_WRONLY", "O_RDWR", "O_CREAT", "O_EXCL", "O_TRUNC"
272 and "O_APPEND" - you can either access them directly
273 ("AnyEvent::IO::O_RDONLY") or import them by specifying the
274 ":flags" import tag (see SYNOPSIS).
275 Example: securely open a file in /var/tmp, fail if it exists or is
277 a symlink.
278 use AnyEvent::IO qw(:flags);
280 aio_open "/var/tmp/mytmp$$", O_CREAT | O_EXCL | O_RDWR, 0600, sub {
282 my ($fh) = @_
283 or return AE::log error => "$! - denial of service attack?";
284 # now we have $fh
286 };
287 aio_close $fh, $cb->($success)
289 Closes the file handle (yes, close can block your process
290 indefinitely) and passes a true value to the callback on success.
291 Due to idiosyncrasies in perl, instead of calling "close", the file
293 handle might get closed by "dup2"'ing another file descriptor over
294 it, that is, the $fh might still be open, but can be closed safely
295 afterwards and must not be used for anything.
296 Example: close a file handle, and dirty as we are, do not even
298 bother to check for errors.
299 aio_close $fh, sub { };
301 aio_read $fh, $length, $cb->($data)
303 Tries to read $length octets from the current position from $fh and
304 passes these bytes to $cb. Otherwise the semantics are very much
305 like those of Perl's "sysread".
306 If less than $length octets have been read, $data will contain only
308 those bytes actually read. At EOF, $data will be a zero-length
309 string. If an error occurs, then nothing is passed to the callback.
310 Obviously, multiple "aio_read"'s or "aio_write"'s at the same time
312 on file handles sharing the underlying open file description
313 results in undefined behaviour, due to sharing of the current file
314 offset (and less obviously so, because OS X is not thread safe and
315 corrupts data when you try).
316 Example: read 128 octets from a file.
318 aio_read $fh, 128, sub {
320 my ($data) = @_
321 or return AE::log error "read from fh: $!";
322 if (length $data) {
324 print "read ", length $data, " octets.\n";
325 } else {
326 print "EOF\n";
327 }
328 };
329 aio_seek $fh, $offset, $whence, $callback->($offs)
331 Seeks the filehandle to the new $offset, similarly to Perl's
332 "sysseek". The $whence are the traditional values (0 to count from
333 start, 1 to count from the current position and 2 to count from the
334 end).
335 The resulting absolute offset will be passed to the callback on
337 success.
338 Example: measure the size of the file in the old-fashioned way
340 using seek.
341 aio_seek $fh, 0, 2, sub {
343 my ($size) = @_
344 or return AE::log error => "seek to end failed: $!";
345 # maybe we need to seek to the beginning again?
347 aio_seek $fh, 0, 0, sub {
348 # now we are hopefully at the beginning
349 };
350 };
351 aio_write $fh, $data, $cb->($length)
353 Tries to write the octets in $data to the current position of $fh
354 and passes the actual number of bytes written to the $cb. Otherwise
355 the semantics are very much like those of Perl's "syswrite".
356 If less than "length $data" octets have been written, $length will
358 reflect that. If an error occurs, then nothing is passed to the
359 callback.
360 Obviously, multiple "aio_read"'s or "aio_write"'s at the same time
362 on file handles sharing the underlying open file description
363 results in undefined behaviour, due to sharing of the current file
364 offset (and less obviously so, because OS X is not thread safe and
365 corrupts data when you try).
366 aio_truncate $fh_or_path, $new_length, $cb->($success)
368 Calls "truncate" on the path or perl file handle and passes a true
369 value to the callback on success.
370 Example: truncate /etc/passwd to zero length - this only works on
372 systems that support "truncate", should not be tried out for
373 obvious reasons and debian will probably open yte another security
374 bug about this example.
375 aio_truncate "/etc/passwd", sub {
377 @_
378 or return AE::log error => "/etc/passwd: $! - are you root enough?";
379 };
380 aio_utime $fh_or_path, $atime, $mtime, $cb->($success)
382 Calls "utime" on the path or perl file handle and passes a true
383 value to the callback on success.
384 The special case of both $atime and $mtime being "undef" sets the
386 times to the current time, on systems that support this.
387 Example: try to touch file.
389 aio_utime "file", undef, undef, sub { };
391 aio_chown $fh_or_path, $uid, $gid, $cb->($success)
393 Calls "chown" on the path or perl file handle and passes a true
394 value to the callback on success.
395 If $uid or $gid can be specified as "undef", in which case the uid
397 or gid of the file is not changed. This differs from Perl's "chown"
398 built-in, which wants "-1" for this.
399 Example: update the group of file to 0 (root), but leave the owner
401 alone.
402 aio_chown "file", undef, 0, sub {
404 @_
405 or return AE::log error => "chown 'file': $!";
406 };
407 aio_chmod $fh_or_path, $perms, $cb->($success)
409 Calls "chmod" on the path or perl file handle and passes a true
410 value to the callback on success.
411 Example: change file to be user/group/world-readable, but leave the
413 other flags alone.
414 aio_stat "file", sub {
416 @_
417 or return AE::log error => "file: $!";
418 aio_chmod "file", (stat _)[2] & 07777 | 00444, sub { };
420 };
421 aio_stat $fh_or_path, $cb->($success)
423 aio_lstat $path, $cb->($success)
424 Calls "stat" or "lstat" on the path or perl file handle and passes
425 a true value to the callback on success.
426 The stat data will be available by "stat"'ing the "_" file handle
428 (e.g. "-x _", "stat _" and so on).
429 Example: see if we can find the number of subdirectories of /etc.
431 aio_stat "/etc", sub {
433 @_
434 or return AE::log error => "/etc: $!";
435 (stat _)[3] >= 2
437 or return AE::log warn => "/etc has low link count - non-POSIX filesystem?";
438 print "/etc has ", (stat _)[3] - 2, " subdirectories.\n";
440 };
441 aio_link $oldpath, $newpath, $cb->($success)
443 Calls "link" on the paths and passes a true value to the callback
444 on success.
445 Example: link "file to file.bak, then rename file.new over file, to
447 atomically replace it.
448 aio_link "file", "file.bak", sub {
450 @_
451 or return AE::log error => "file: $!";
452 aio_rename "file.new", "file", sub {
454 @_
455 or return AE::log error => "file.new: $!";
456 print "file atomically replaced by file.new, backup file.bak\n";
458 };
459 };
460 aio_symlink $oldpath, $newpath, $cb->($success)
462 Calls "symlink" on the paths and passes a true value to the
463 callback on success.
464 Example: create a symlink "slink containing "random data".
466 aio_symlink "random data", "slink", sub {
468 @_
469 or return AE::log error => "slink: $!";
470 };
471 aio_readlink $path, $cb->($target)
473 Calls "readlink" on the paths and passes the link target string to
474 the callback.
475 Example: read the symlink called Fyslink> and verify that it
477 contains "random data".
478 aio_readlink "slink", sub {
480 my ($target) = @_
481 or return AE::log error => "slink: $!";
482 $target eq "random data"
484 or AE::log critical => "omg, the world will end!";
485 };
486 aio_rename $oldpath, $newpath, $cb->($success)
488 Calls "rename" on the paths and passes a true value to the callback
489 on success.
490 See "aio_link" for an example.
492 aio_unlink $path, $cb->($success)
494 Tries to unlink the object at $path and passes a true value to the
495 callback on success.
496 Example: try to delete the file tmpfile.dat~.
498 aio_unlink "tmpfile.dat~", sub { };
500 aio_mkdir $path, $perms, $cb->($success)
502 Calls "mkdir" on the path with the given permissions $perms (when
503 in doubt, 0777 is a good value) and passes a true value to the
504 callback on success.
505 Example: try to create the directory subdir and leave it to
507 whoeveer comes after us to check whether it worked.
508 aio_mkdir "subdir", 0777, sub { };
510 aio_rmdir $path, $cb->($success)
512 Tries to remove the directory at $path and passes a true value to
513 the callback on success.
514 Example: try to remove the directory subdir and don't give a damn
516 if that fails.
517 aio_rmdir "subdir", sub { };
519 aio_readdir $path, $cb->(\@names)
521 Reads all filenames from the directory specified by $path and
522 passes them to the callback, as an array reference with the names
523 (without a path prefix). The . and .. names will be filtered out
524 first.
525 The ordering of the file names is undefined - backends that are
527 capable of it (e.g. IO::AIO) will return the ordering that most
528 likely is fastest to "stat" through, and furthermore put entries
529 that likely are directories first in the array.
530 If you need best performance in recursive directory traversal or
532 when looking at really big directories, you are advised to use
533 IO::AIO directly, specifically the "aio_readdirx" and "aio_scandir"
534 functions, which have more options to tune performance.
535 Example: recursively scan a directory hierarchy, silently skip
537 diretcories we couldn't read and print all others.
538 sub scan($); # visibility-in-next statement is not so useful these days
540 sub scan($) {
541 my ($path) = @_;
542 aio_readdir $path, sub {
544 my ($names) = @_
545 or return;
546 print "$path\n";
548 for my $name (@$names) {
550 aio_lstat "$path/$name", sub {
551 scan "$path/$name"
552 if -d _;
553 };
554 }
555 };
556 }
557 scan "/etc";
559
561 See the description of "PERL_ANYEVENT_IO_MODEL" in the AnyEvent
562 manpage.
563
565 Marc Lehmann <schmorp@schmorp.de>
566 http://anyevent.schmorp.de
567perl v5.32.0 2020-07-28 AnyEvent::IO(3)