1AnyEvent::Intro(3) User Contributed Perl Documentation AnyEvent::Intro(3)
2
6 AnyEvent::Intro - an introductory tutorial to AnyEvent
7
9 This is a tutorial that will introduce you to the features of AnyEvent.
10 The first part introduces the core AnyEvent module (after swamping you
12 a bit in evangelism), which might already provide all you ever need: If
13 you are only interested in AnyEvent's event handling capabilities, read
14 no further.
15 The second part focuses on network programming using sockets, for which
17 AnyEvent offers a lot of support you can use, and a lot of workarounds
18 around portability quirks.
19
21 If you don't care for the whys and want to see code, skip this section!
22 AnyEvent is first of all just a framework to do event-based
24 programming. Typically such frameworks are an all-or-nothing thing: If
25 you use one such framework, you can't (easily, or even at all) use
26 another in the same program.
27 AnyEvent is different - it is a thin abstraction layer on top of other
29 event loops, just like DBI is an abstraction of many different database
30 APIs. Its main purpose is to move the choice of the underlying
31 framework (the event loop) from the module author to the program author
32 using the module.
33 That means you can write code that uses events to control what it does,
35 without forcing other code in the same program to use the same
36 underlying framework as you do - i.e. you can create a Perl module that
37 is event-based using AnyEvent, and users of that module can still
38 choose between using Gtk2, Tk, Event (or run inside Irssi or rxvt-
39 unicode) or any other supported event loop. AnyEvent even comes with
40 its own pure-perl event loop implementation, so your code works
41 regardless of other modules that might or might not be installed. The
42 latter is important, as AnyEvent does not have any hard dependencies to
43 other modules, which makes it easy to install, for example, when you
44 lack a C compiler. No matter what environment, AnyEvent will just cope
45 with it.
46 A typical limitation of existing Perl modules such as Net::IRC is that
48 they come with their own event loop: In Net::IRC, a program which uses
49 it needs to start the event loop of Net::IRC. That means that one
50 cannot integrate this module into a Gtk2 GUI for instance, as that
51 module, too, enforces the use of its own event loop (namely Glib).
52 Another example is LWP: it provides no event interface at all. It's a
54 pure blocking HTTP (and FTP etc.) client library, which usually means
55 that you either have to start another process or have to fork for a
56 HTTP request, or use threads (e.g. Coro::LWP), if you want to do
57 something else while waiting for the request to finish.
58 The motivation behind these designs is often that a module doesn't want
60 to depend on some complicated XS-module (Net::IRC), or that it doesn't
61 want to force the user to use some specific event loop at all (LWP),
62 out of fear of severly limiting the usefulness of the module: If your
63 module requires Glib, it will not run in a Tk program.
64 AnyEvent solves this dilemma, by not forcing module authors to either:
66 - write their own event loop (because it guarantees the availability of
68 an event loop everywhere - even on windows with no extra modules
69 installed).
70 - choose one specific event loop (because AnyEvent works with most
71 event loops available for Perl).
72 If the module author uses AnyEvent for all his (or her) event needs (IO
74 events, timers, signals, ...) then all other modules can just use his
75 module and don't have to choose an event loop or adapt to his event
76 loop. The choice of the event loop is ultimately made by the program
77 author who uses all the modules and writes the main program. And even
78 there he doesn't have to choose, he can just let AnyEvent choose the
79 most efficient event loop available on the system.
80 Read more about this in the main documentation of the AnyEvent module.
82
84 So what exactly is programming using events? It quite simply means that
85 instead of your code actively waiting for something, such as the user
86 entering something on STDIN:
87 $| = 1; print "enter your name> ";
89 my $name = <STDIN>;
91 You instead tell your event framework to notify you in the event of
93 some data being available on STDIN, by using a callback mechanism:
94 use AnyEvent;
96 $| = 1; print "enter your name> ";
98 my $name;
100 my $wait_for_input = AnyEvent->io (
102 fh => \*STDIN, # which file handle to check
103 poll => "r", # which event to wait for ("r"ead data)
104 cb => sub { # what callback to execute
105 $name = <STDIN>; # read it
106 }
107 );
108 # do something else here
110 Looks more complicated, and surely is, but the advantage of using
112 events is that your program can do something else instead of waiting
113 for input (side note: combining AnyEvent with a thread package such as
114 Coro can recoup much of the simplicity, effectively getting the best of
115 two worlds).
116 Waiting as done in the first example is also called "blocking" the
118 process because you "block"/keep your process from executing anything
119 else while you do so.
120 The second example avoids blocking by only registering interest in a
122 read event, which is fast and doesn't block your process. The callback
123 will be called only when data is available and can be read without
124 blocking.
125 The "interest" is represented by an object returned by "AnyEvent->io"
127 called a "watcher" object - thus named because it "watches" your file
128 handle (or other event sources) for the event you are interested in.
129 In the example above, we create an I/O watcher by calling the
131 "AnyEvent->io" method. A lack of further interest in some event is
132 expressed by simply forgetting about its watcher, for example by
133 "undef"-ing the only variable it is stored in. AnyEvent will
134 automatically clean up the watcher if it is no longer used, much like
135 Perl closes your file handles if you no longer use them anywhere.
136 A short note on callbacks
138 A common issue that hits people is the problem of passing parameters to
140 callbacks. Programmers used to languages such as C or C++ are often
141 used to a style where one passes the address of a function (a function
142 reference) and some data value, e.g.:
143 sub callback {
145 my ($arg) = @_;
146 $arg->method;
148 }
149 my $arg = ...;
151 call_me_back_later \&callback, $arg;
153 This is clumsy, as the place where behaviour is specified (when the
155 callback is registered) is often far away from the place where
156 behaviour is implemented. It also doesn't use Perl syntax to invoke the
157 code. There is also an abstraction penalty to pay as one has to name
158 the callback, which often is unnecessary and leads to nonsensical or
159 duplicated names.
160 In Perl, one can specify behaviour much more directly by using
162 closures. Closures are code blocks that take a reference to the
163 enclosing scope(s) when they are created. This means lexical variables
164 in scope when a closure is created can be used inside the closure:
165 my $arg = ...;
167 call_me_back_later sub { $arg->method };
169 Under most circumstances, closures are faster, use fewer resources and
171 result in much clearer code than the traditional approach. Faster,
172 because parameter passing and storing them in local variables in Perl
173 is relatively slow. Fewer resources, because closures take references
174 to existing variables without having to create new ones, and clearer
175 code because it is immediately obvious that the second example calls
176 the "method" method when the callback is invoked.
177 Apart from these, the strongest argument for using closures with
179 AnyEvent is that AnyEvent does not allow passing parameters to the
180 callback, so closures are the only way to achieve that in most cases
181 :->
182 A little hint to catch mistakes
184 AnyEvent does not check the parameters you pass in, at least not by
186 default. to enable checking, simply start your program with
187 "AE_STRICT=1" in the environment, or put "use AnyEvent::Strict" near
188 the top of your program:
189 AE_STRICT=1 perl myprogram
191 You can find more info on this and additional debugging aids later in
193 this introduction.
194 Condition Variables
196 Back to the I/O watcher example: The code is not yet a fully working
197 program, and will not work as-is. The reason is that your callback will
198 not be invoked out of the blue; you have to run the event loop first.
199 Also, event-based programs need to block sometimes too, such as when
200 there is nothing to do, and everything is waiting for new events to
201 arrive.
202 In AnyEvent, this is done using condition variables. Condition
204 variables are named "condition variables" because they represent a
205 condition that is initially false and needs to be fulfilled.
206 You can also call them "merge points", "sync points", "rendezvous
208 ports" or even callbacks and many other things (and they are often
209 called these names in other frameworks). The important point is that
210 you can create them freely and later wait for them to become true.
211 Condition variables have two sides - one side is the "producer" of the
213 condition (whatever code detects and flags the condition), the other
214 side is the "consumer" (the code that waits for that condition).
215 In our example in the previous section, the producer is the event
217 callback and there is no consumer yet - let's change that right now:
218 use AnyEvent;
220 $| = 1; print "enter your name> ";
222 my $name;
224 my $name_ready = AnyEvent->condvar;
226 my $wait_for_input = AnyEvent->io (
228 fh => \*STDIN,
229 poll => "r",
230 cb => sub {
231 $name = <STDIN>;
232 $name_ready->send;
233 }
234 );
235 # do something else here
237 # now wait until the name is available:
239 $name_ready->recv;
240 undef $wait_for_input; # watcher no longer needed
242 print "your name is $name\n";
244 This program creates an AnyEvent condvar by calling the
246 "AnyEvent->condvar" method. It then creates a watcher as usual, but
247 inside the callback it "send"s the $name_ready condition variable,
248 which causes whoever is waiting on it to continue.
249 The "whoever" in this case is the code that follows, which calls
251 "$name_ready->recv": The producer calls "send", the consumer calls
252 "recv".
253 If there is no $name available yet, then the call to
255 "$name_ready->recv" will halt your program until the condition becomes
256 true.
257 As the names "send" and "recv" imply, you can actually send and receive
259 data using this, for example, the above code could also be written like
260 this, without an extra variable to store the name in:
261 use AnyEvent;
263 $| = 1; print "enter your name> ";
265 my $name_ready = AnyEvent->condvar;
267 my $wait_for_input = AnyEvent->io (
269 fh => \*STDIN, poll => "r",
270 cb => sub { $name_ready->send (scalar <STDIN>) }
271 );
272 # do something else here
274 # now wait and fetch the name
276 my $name = $name_ready->recv;
277 undef $wait_for_input; # watcher no longer needed
279 print "your name is $name\n";
281 You can pass any number of arguments to "send", and every subsequent
283 call to "recv" will return them.
284 The "main loop"
286 Most event-based frameworks have something called a "main loop" or
287 "event loop run function" or something similar.
288 Just like in "recv" AnyEvent, these functions need to be called
290 eventually so that your event loop has a chance of actually looking for
291 the events you are interested in.
292 For example, in a Gtk2 program, the above example could also be written
294 like this:
295 use Gtk2 -init;
297 use AnyEvent;
298 ############################################
300 # create a window and some label
301 my $window = new Gtk2::Window "toplevel";
303 $window->add (my $label = new Gtk2::Label "soon replaced by name");
304 $window->show_all;
306 ############################################
308 # do our AnyEvent stuff
309 $| = 1; print "enter your name> ";
311 my $wait_for_input = AnyEvent->io (
313 fh => \*STDIN, poll => "r",
314 cb => sub {
315 # set the label
316 $label->set_text (scalar <STDIN>);
317 print "enter another name> ";
318 }
319 );
320 ############################################
322 # Now enter Gtk2's event loop
323 main Gtk2;
325 No condition variable anywhere in sight - instead, we just read a line
327 from STDIN and replace the text in the label. In fact, since nobody
328 "undef"s $wait_for_input you can enter multiple lines.
329 Instead of waiting for a condition variable, the program enters the
331 Gtk2 main loop by calling "Gtk2->main", which will block the program
332 and wait for events to arrive.
333 This also shows that AnyEvent is quite flexible - you didn't have to do
335 anything to make the AnyEvent watcher use Gtk2 (actually Glib) - it
336 just worked.
337 Admittedly, the example is a bit silly - who would want to read names
339 from standard input in a Gtk+ application? But imagine that instead of
340 doing that, you make an HTTP request in the background and display its
341 results. In fact, with event-based programming you can make many HTTP
342 requests in parallel in your program and still provide feedback to the
343 user and stay interactive.
344 And in the next part you will see how to do just that - by implementing
346 an HTTP request, on our own, with the utility modules AnyEvent comes
347 with.
348 Before that, however, let's briefly look at how you would write your
350 program using only AnyEvent, without ever calling some other event
351 loop's run function.
352 In the example using condition variables, we used those to start
354 waiting for events, and in fact, condition variables are the solution:
355 my $quit_program = AnyEvent->condvar;
357 # create AnyEvent watchers (or not) here
359 $quit_program->recv;
361 If any of your watcher callbacks decide to quit (this is often called
363 an "unloop" in other frameworks), they can just call
364 "$quit_program->send". Of course, they could also decide not to and
365 call "exit" instead, or they could decide never to quit (e.g. in a
366 long-running daemon program).
367 If you don't need some clean quit functionality and just want to run
369 the event loop, you can do this:
370 AnyEvent->condvar->recv;
372 And this is, in fact, the closest to the idea of a main loop run
374 function that AnyEvent offers.
375 Timers and other event sources
377 So far, we have used only I/O watchers. These are useful mainly to find
378 out whether a socket has data to read, or space to write more data. On
379 sane operating systems this also works for console windows/terminals
380 (typically on standard input), serial lines, all sorts of other
381 devices, basically almost everything that has a file descriptor but
382 isn't a file itself. (As usual, "sane" excludes windows - on that
383 platform you would need different functions for all of these,
384 complicating code immensely - think "socket only" on windows).
385 However, I/O is not everything - the second most important event source
387 is the clock. For example when doing an HTTP request you might want to
388 time out when the server doesn't answer within some predefined amount
389 of time.
390 In AnyEvent, timer event watchers are created by calling the
392 "AnyEvent->timer" method:
393 use AnyEvent;
395 my $cv = AnyEvent->condvar;
397 my $wait_one_and_a_half_seconds = AnyEvent->timer (
399 after => 1.5, # after how many seconds to invoke the cb?
400 cb => sub { # the callback to invoke
401 $cv->send;
402 },
403 );
404 # can do something else here
406 # now wait till our time has come
408 $cv->recv;
409 Unlike I/O watchers, timers are only interested in the amount of
411 seconds they have to wait. When (at least) that amount of time has
412 passed, AnyEvent will invoke your callback.
413 Unlike I/O watchers, which will call your callback as many times as
415 there is data available, timers are normally one-shot: after they have
416 "fired" once and invoked your callback, they are dead and no longer do
417 anything.
418 To get a repeating timer, such as a timer firing roughly once per
420 second, you can specify an "interval" parameter:
421 my $once_per_second = AnyEvent->timer (
423 after => 0, # first invoke ASAP
424 interval => 1, # then invoke every second
425 cb => sub { # the callback to invoke
426 $cv->send;
427 },
428 );
429 More esoteric sources
431 AnyEvent also has some other, more esoteric event sources you can tap
433 into: signal, child and idle watchers.
434 Signal watchers can be used to wait for "signal events", which means
436 your process was sent a signal (such as "SIGTERM" or "SIGUSR1").
437 Child-process watchers wait for a child process to exit. They are
439 useful when you fork a separate process and need to know when it exits,
440 but you do not want to wait for that by blocking.
441 Idle watchers invoke their callback when the event loop has handled all
443 outstanding events, polled for new events and didn't find any, i.e.,
444 when your process is otherwise idle. They are useful if you want to do
445 some non-trivial data processing that can be done when your program
446 doesn't have anything better to do.
447 All these watcher types are described in detail in the main AnyEvent
449 manual page.
450 Sometimes you also need to know what the current time is:
452 "AnyEvent->now" returns the time the event toolkit uses to schedule
453 relative timers, and is usually what you want. It is often cached
454 (which means it can be a bit outdated). In that case, you can use the
455 more costly "AnyEvent->time" method which will ask your operating
456 system for the current time, which is slower, but also more up to date.
457
459 So far you have seen how to register event watchers and handle events.
460 This is a great foundation to write network clients and servers, and
462 might be all that your module (or program) ever requires, but writing
463 your own I/O buffering again and again becomes tedious, not to mention
464 that it attracts errors.
465 While the core AnyEvent module is still small and self-contained, the
467 distribution comes with some very useful utility modules such as
468 AnyEvent::Handle, AnyEvent::DNS and AnyEvent::Socket. These can make
469 your life as a non-blocking network programmer a lot easier.
470 Here is a quick overview of these three modules:
472 AnyEvent::DNS
474 This module allows fully asynchronous DNS resolution. It is used mainly
475 by AnyEvent::Socket to resolve hostnames and service ports for you, but
476 is a great way to do other DNS resolution tasks, such as reverse
477 lookups of IP addresses for log files.
478 AnyEvent::Handle
480 This module handles non-blocking IO on (socket-, pipe- etc.) file
481 handles in an event based manner. It provides a wrapper object around
482 your file handle that provides queueing and buffering of incoming and
483 outgoing data for you.
484 It also implements the most common data formats, such as text lines, or
486 fixed and variable-width data blocks.
487 AnyEvent::Socket
489 This module provides you with functions that handle socket creation and
490 IP address magic. The two main functions are "tcp_connect" and
491 "tcp_server". The former will connect a (streaming) socket to an
492 internet host for you and the later will make a server socket for you,
493 to accept connections.
494 This module also comes with transparent IPv6 support, this means: If
496 you write your programs with this module, you will be IPv6 ready
497 without doing anything special.
498 It also works around a lot of portability quirks (especially on the
500 windows platform), which makes it even easier to write your programs in
501 a portable way (did you know that windows uses different error codes
502 for all socket functions and that Perl does not know about these? That
503 "Unknown error 10022" (which is "WSAEINVAL") can mean that our
504 "connect" call was successful? That unsuccessful TCP connects might
505 never be reported back to your program? That "WSAEINPROGRESS" means
506 your "connect" call was ignored instead of being in progress?
507 AnyEvent::Socket works around all of these Windows/Perl bugs for you).
508 Implementing a parallel finger client with non-blocking connects and
510 AnyEvent::Socket
511 The finger protocol is one of the simplest protocols in use on the
512 internet. Or in use in the past, as almost nobody uses it anymore.
513 It works by connecting to the finger port on another host, writing a
515 single line with a user name and then reading the finger response, as
516 specified by that user. OK, RFC 1288 specifies a vastly more complex
517 protocol, but it basically boils down to this:
518 # telnet freebsd.org finger
520 Trying 8.8.178.135...
521 Connected to freebsd.org (8.8.178.135).
522 Escape character is '^]'.
523 larry
524 Login: lile Name: Larry Lile
525 Directory: /home/lile Shell: /usr/local/bin/bash
526 No Mail.
527 Mail forwarded to: lile@stdio.com
528 No Plan.
529 So let's write a little AnyEvent function that makes a finger request:
531 use AnyEvent;
533 use AnyEvent::Socket;
534 sub finger($$) {
536 my ($user, $host) = @_;
537 # use a condvar to return results
539 my $cv = AnyEvent->condvar;
540 # first, connect to the host
542 tcp_connect $host, "finger", sub {
543 # the callback receives the socket handle - or nothing
544 my ($fh) = @_
545 or return $cv->send;
546 # now write the username
548 syswrite $fh, "$user\015\012";
549 my $response;
551 # register a read watcher
553 my $read_watcher; $read_watcher = AnyEvent->io (
554 fh => $fh,
555 poll => "r",
556 cb => sub {
557 my $len = sysread $fh, $response, 1024, length $response;
558 if ($len <= 0) {
560 # we are done, or an error occured, lets ignore the latter
561 undef $read_watcher; # no longer interested
562 $cv->send ($response); # send results
563 }
564 },
565 );
566 };
567 # pass $cv to the caller
569 $cv
570 }
571 That's a mouthful! Let's dissect this function a bit, first the overall
573 function and execution flow:
574 sub finger($$) {
576 my ($user, $host) = @_;
577 # use a condvar to return results
579 my $cv = AnyEvent->condvar;
580 # first, connect to the host
582 tcp_connect $host, "finger", sub {
583 ...
584 };
585 $cv
587 }
588 This isn't too complicated, just a function with two parameters that
590 creates a condition variable $cv, initiates a TCP connect to $host, and
591 returns $cv. The caller can use the returned $cv to receive the finger
592 response, but one could equally well pass a third argument, a callback,
593 to the function.
594 Since we are programming event'ish, we do not wait for the connect to
596 finish - it could block the program for a minute or longer!
597 Instead, we pass "tcp_connect" a callback to invoke when the connect is
599 done. The callback is called with the socket handle as its first
600 argument if the connect succeeds, and no arguments otherwise. The
601 important point is that it will always be called as soon as the outcome
602 of the TCP connect is known.
603 This style of programming is also called "continuation style": the
605 "continuation" is simply the way the program continues - normally at
606 the next line after some statement (the exception is loops or things
607 like "return"). When we are interested in events, however, we instead
608 specify the "continuation" of our program by passing a closure, which
609 makes that closure the "continuation" of the program.
610 The "tcp_connect" call is like saying "return now, and when the
612 connection is established or the attempt failed, continue there".
613 Now let's look at the callback/closure in more detail:
615 # the callback receives the socket handle - or nothing
617 my ($fh) = @_
618 or return $cv->send;
619 The first thing the callback does is to save the socket handle in $fh.
621 When there was an error (no arguments), then our instinct as expert
622 Perl programmers would tell us to "die":
623 my ($fh) = @_
625 or die "$host: $!";
626 While this would give good feedback to the user (if he happens to watch
628 standard error), our program would probably stop working here, as we
629 never report the results to anybody, certainly not the caller of our
630 "finger" function, and most event loops continue even after a "die"!
631 This is why we instead "return", but also call "$cv->send" without any
633 arguments to signal to the condvar consumer that something bad has
634 happened. The return value of "$cv->send" is irrelevant, as is the
635 return value of our callback. The "return" statement is used for the
636 side effect of, well, returning immediately from the callback.
637 Checking for errors and handling them this way is very common, which is
638 why this compact idiom is so handy.
639 As the next step in the finger protocol, we send the username to the
641 finger daemon on the other side of our connection (the kernel.org
642 finger service doesn't actually wait for a username, but the net is
643 running out of finger servers fast):
644 syswrite $fh, "$user\015\012";
646 Note that this isn't 100% clean socket programming - the socket could,
648 for whatever reasons, not accept our data. When writing a small amount
649 of data like in this example it doesn't matter, as a socket buffer is
650 almost always big enough for a mere "username", but for real-world
651 cases you might need to implement some kind of write buffering - or use
652 AnyEvent::Handle, which handles these matters for you, as shown in the
653 next section.
654 What we do have to do is implement our own read buffer - the response
656 data could arrive late or in multiple chunks, and we cannot just wait
657 for it (event-based programming, you know?).
658 To do that, we register a read watcher on the socket which waits for
660 data:
661 my $read_watcher; $read_watcher = AnyEvent->io (
663 fh => $fh,
664 poll => "r",
665 There is a trick here, however: the read watcher isn't stored in a
667 global variable, but in a local one - if the callback returns, it would
668 normally destroy the variable and its contents, which would in turn
669 unregister our watcher.
670 To avoid that, we refer to the watcher variable in the watcher
672 callback. This means that, when the "tcp_connect" callback returns,
673 perl thinks (quite correctly) that the read watcher is still in use -
674 namely inside the inner callback - and thus keeps it alive even if
675 nothing else in the program refers to it anymore (it is much like Baron
676 Münchhausen keeping himself from dying by pulling himself out of a
677 swamp).
678 The trick, however, is that instead of:
680 my $read_watcher = AnyEvent->io (...
682 The program does:
684 my $read_watcher; $read_watcher = AnyEvent->io (...
686 The reason for this is a quirk in the way Perl works: variable names
688 declared with "my" are only visible in the next statement. If the whole
689 "AnyEvent->io" call, including the callback, would be done in a single
690 statement, the callback could not refer to the $read_watcher variable
691 to "undef"ine it, so it is done in two statements.
692 Whether you'd want to format it like this is of course a matter of
694 style. This way emphasizes that the declaration and assignment really
695 are one logical statement.
696 The callback itself calls "sysread" for as many times as necessary,
698 until "sysread" returns either an error or end-of-file:
699 cb => sub {
701 my $len = sysread $fh, $response, 1024, length $response;
702 if ($len <= 0) {
704 Note that "sysread" has the ability to append data it reads to a scalar
706 if we specify an offset, a feature which we make use of in this
707 example.
708 When "sysread" indicates we are done, the callback "undef"ines the
710 watcher and then "send"s the response data to the condition variable.
711 All this has the following effects:
712 Undefining the watcher destroys it, as our callback was the only one
714 still having a reference to it. When the watcher gets destroyed, it
715 destroys the callback, which in turn means the $fh handle is no longer
716 used, so that gets destroyed as well. The result is that all resources
717 will be nicely cleaned up by perl for us.
718 Using the finger client
720 Now, we could probably write the same finger client in a simpler way if
722 we used "IO::Socket::INET", ignored the problem of multiple hosts and
723 ignored IPv6 and a few other things that "tcp_connect" handles for us.
724 But the main advantage is that we can not only run this finger function
726 in the background, we even can run multiple sessions in parallel, like
727 this:
728 my $f1 = finger "kuriyama", "freebsd.org";
730 my $f2 = finger "icculus?listarchives=1", "icculus.org";
731 my $f3 = finger "mikachu", "icculus.org";
732 print "kuriyama's gpg key\n" , $f1->recv, "\n";
734 print "icculus' plan archive\n" , $f2->recv, "\n";
735 print "mikachu's plan zomgn\n" , $f3->recv, "\n";
736 It doesn't look like it, but in fact all three requests run in
738 parallel. The code waits for the first finger request to finish first,
739 but that doesn't keep it from executing them parallel: when the first
740 "recv" call sees that the data isn't ready yet, it serves events for
741 all three requests automatically, until the first request has finished.
742 The second "recv" call might either find the data is already there, or
744 it will continue handling events until that is the case, and so on.
745 By taking advantage of network latencies, which allows us to serve
747 other requests and events while we wait for an event on one socket, the
748 overall time to do these three requests will be greatly reduced,
749 typically all three are done in the same time as the slowest of the
750 three requests.
751 By the way, you do not actually have to wait in the "recv" method on an
753 AnyEvent condition variable - after all, waiting is evil - you can also
754 register a callback:
755 $f1->cb (sub {
757 my $response = shift->recv;
758 # ...
759 });
760 The callback will be invoked only when "send" is called. In fact,
762 instead of returning a condition variable you could also pass a third
763 parameter to your finger function, the callback to invoke with the
764 response:
765 sub finger($$$) {
767 my ($user, $host, $cb) = @_;
768 How you implement it is a matter of taste - if you expect your function
770 to be used mainly in an event-based program you would normally prefer
771 to pass a callback directly. If you write a module and expect your
772 users to use it "synchronously" often (for example, a simple http-get
773 script would not really care much for events), then you would use a
774 condition variable and tell them "simply "->recv" the data".
775 Problems with the implementation and how to fix them
777 To make this example more real-world-ready, we would not only implement
779 some write buffering (for the paranoid, or maybe denial-of-service
780 aware security expert), but we would also have to handle timeouts and
781 maybe protocol errors.
782 Doing this quickly gets unwieldy, which is why we introduce
784 AnyEvent::Handle in the next section, which takes care of all these
785 details for you and lets you concentrate on the actual protocol.
786 Implementing simple HTTP and HTTPS GET requests with AnyEvent::Handle
788 The AnyEvent::Handle module has been hyped quite a bit in this document
789 so far, so let's see what it really offers.
790 As finger is such a simple protocol, let's try something slightly more
792 complicated: HTTP/1.0.
793 An HTTP GET request works by sending a single request line that
795 indicates what you want the server to do and the URI you want to act it
796 on, followed by as many "header" lines ("Header: data", same as e-mail
797 headers) as required for the request, followed by an empty line.
798 The response is formatted very similarly, first a line with the
800 response status, then again as many header lines as required, then an
801 empty line, followed by any data that the server might send.
802 Again, let's try it out with "telnet" (I condensed the output a bit -
804 if you want to see the full response, do it yourself).
805 # telnet www.google.com 80
807 Trying 209.85.135.99...
808 Connected to www.google.com (209.85.135.99).
809 Escape character is '^]'.
810 GET /test HTTP/1.0
811 HTTP/1.0 404 Not Found
813 Date: Mon, 02 Jun 2008 07:05:54 GMT
814 Content-Type: text/html; charset=UTF-8
815 <html><head>
817 [...]
818 Connection closed by foreign host.
819 The "GET ..." and the empty line were entered manually, the rest of the
821 telnet output is google's response, in this case a "404 not found" one.
822 So, here is how you would do it with "AnyEvent::Handle":
824 sub http_get {
826 my ($host, $uri, $cb) = @_;
827 # store results here
829 my ($response, $header, $body);
830 my $handle; $handle = new AnyEvent::Handle
832 connect => [$host => 'http'],
833 on_error => sub {
834 $cb->("HTTP/1.0 500 $!");
835 $handle->destroy; # explicitly destroy handle
836 },
837 on_eof => sub {
838 $cb->($response, $header, $body);
839 $handle->destroy; # explicitly destroy handle
840 };
841 $handle->push_write ("GET $uri HTTP/1.0\015\012\015\012");
843 # now fetch response status line
845 $handle->push_read (line => sub {
846 my ($handle, $line) = @_;
847 $response = $line;
848 });
849 # then the headers
851 $handle->push_read (line => "\015\012\015\012", sub {
852 my ($handle, $line) = @_;
853 $header = $line;
854 });
855 # and finally handle any remaining data as body
857 $handle->on_read (sub {
858 $body .= $_[0]->rbuf;
859 $_[0]->rbuf = "";
860 });
861 }
862 And now let's go through it step by step. First, as usual, the overall
864 "http_get" function structure:
865 sub http_get {
867 my ($host, $uri, $cb) = @_;
868 # store results here
870 my ($response, $header, $body);
871 my $handle; $handle = new AnyEvent::Handle
873 ... create handle object
874 ... push data to write
876 ... push what to expect to read queue
878 }
879 Unlike in the finger example, this time the caller has to pass a
881 callback to "http_get". Also, instead of passing a URL as one would
882 expect, the caller has to provide the hostname and URI - normally you
883 would use the "URI" module to parse a URL and separate it into those
884 parts, but that is left to the inspired reader :)
885 Since everything else is left to the caller, all "http_get" does is
887 initiate the connection by creating the AnyEvent::Handle object (which
888 calls "tcp_connect" for us) and leave everything else to its callback.
889 The handle object is created, unsurprisingly, by calling the "new"
891 method of AnyEvent::Handle:
892 my $handle; $handle = new AnyEvent::Handle
894 connect => [$host => 'http'],
895 on_error => sub {
896 $cb->("HTTP/1.0 500 $!");
897 $handle->destroy; # explicitly destroy handle
898 },
899 on_eof => sub {
900 $cb->($response, $header, $body);
901 $handle->destroy; # explicitly destroy handle
902 };
903 The "connect" argument tells AnyEvent::Handle to call "tcp_connect" for
905 the specified host and service/port.
906 The "on_error" callback will be called on any unexpected error, such as
908 a refused connection, or unexpected end-of-file while reading headers.
909 Instead of having an extra mechanism to signal errors, connection
911 errors are signalled by crafting a special "response status line", like
912 this:
913 HTTP/1.0 500 Connection refused
915 This means the caller cannot distinguish (easily) between locally-
917 generated errors and server errors, but it simplifies error handling
918 for the caller a lot.
919 The error callback also destroys the handle explicitly, because we are
921 not interested in continuing after any errors. In AnyEvent::Handle
922 callbacks you have to call "destroy" explicitly to destroy a handle.
923 Outside of those callbacks you can just forget the object reference and
924 it will be automatically cleaned up.
925 Last but not least, we set an "on_eof" callback that is called when the
927 other side indicates it has stopped writing data, which we will use to
928 gracefully shut down the handle and report the results. This callback
929 is only called when the read queue is empty - if the read queue expects
930 some data and the handle gets an EOF from the other side this will be
931 an error - after all, you did expect more to come.
932 If you wanted to write a server using AnyEvent::Handle, you would use
934 "tcp_accept" and then create the AnyEvent::Handle with the "fh"
935 argument.
936 The write queue
938 The next line sends the actual request:
940 $handle->push_write ("GET $uri HTTP/1.0\015\012\015\012");
942 No headers will be sent (this is fine for simple requests), so the
944 whole request is just a single line followed by an empty line to signal
945 the end of the headers to the server.
946 The more interesting question is why the method is called "push_write"
948 and not just write. The reason is that you can always add some write
949 data without blocking, and to do this, AnyEvent::Handle needs some
950 write queue internally - and "push_write" pushes some data onto the end
951 of that queue, just like Perl's "push" pushes data onto the end of an
952 array.
953 The deeper reason is that at some point in the future, there might be
955 "unshift_write" as well, and in any case, we will shortly meet
956 "push_read" and "unshift_read", and it's usually easiest to remember if
957 all those functions have some symmetry in their name. So "push" is used
958 as the opposite of "unshift" in AnyEvent::Handle, not as the opposite
959 of "pull" - just like in Perl.
960 Note that we call "push_write" right after creating the
962 AnyEvent::Handle object, before it has had time to actually connect to
963 the server. This is fine, pushing the read and write requests will
964 queue them in the handle object until the connection has been
965 established. Alternatively, we could do this "on demand" in the
966 "on_connect" callback.
967 If "push_write" is called with more than one argument, then you can do
969 formatted I/O. For example, this would JSON-encode your data before
970 pushing it to the write queue:
971 $handle->push_write (json => [1, 2, 3]);
973 This pretty much summarises the write queue, there is little else to
975 it.
976 Reading the response is far more interesting, because it involves the
978 more powerful and complex read queue:
979 The read queue
981 The response consists of three parts: a single line with the response
983 status, a single paragraph of headers ended by an empty line, and the
984 request body, which is the remaining data on the connection.
985 For the first two, we push two read requests onto the read queue:
987 # now fetch response status line
989 $handle->push_read (line => sub {
990 my ($handle, $line) = @_;
991 $response = $line;
992 });
993 # then the headers
995 $handle->push_read (line => "\015\012\015\012", sub {
996 my ($handle, $line) = @_;
997 $header = $line;
998 });
999 While one can just push a single callback to parse all the data on the
1001 queue, formatted I/O really comes to our aid here, since there is a
1002 ready-made "read line" read type. The first read expects a single line,
1003 ended by "\015\012" (the standard end-of-line marker in internet
1004 protocols).
1005 The second "line" is actually a single paragraph - instead of reading
1007 it line by line we tell "push_read" that the end-of-line marker is
1008 really "\015\012\015\012", which is an empty line. The result is that
1009 the whole header paragraph will be treated as a single line and read.
1010 The word "line" is interpreted very freely, much like Perl itself does
1011 it.
1012 Note that push read requests are pushed immediately after creating the
1014 handle object - since AnyEvent::Handle provides a queue we can push as
1015 many requests as we want, and AnyEvent::Handle will handle them in
1016 order.
1017 There is, however, no read type for "the remaining data". For that, we
1019 install our own "on_read" callback:
1020 # and finally handle any remaining data as body
1022 $handle->on_read (sub {
1023 $body .= $_[0]->rbuf;
1024 $_[0]->rbuf = "";
1025 });
1026 This callback is invoked every time data arrives and the read queue is
1028 empty - which in this example will only be the case when both response
1029 and header have been read. The "on_read" callback could actually have
1030 been specified when constructing the object, but doing it this way
1031 preserves logical ordering.
1032 The read callback adds the current read buffer to its $body variable
1034 and, most importantly, empties the buffer by assigning the empty string
1035 to it.
1036 Given these instructions, AnyEvent::Handle will handle incoming data -
1038 if all goes well, the callback will be invoked with the response data;
1039 if not, it will get an error.
1040 In general, you can implement pipelining (a semi-advanced feature of
1042 many protocols) very easily with AnyEvent::Handle: If you have a
1043 protocol with a request/response structure, your request
1044 methods/functions will all look like this (simplified):
1045 sub request {
1047 # send the request to the server
1049 $handle->push_write (...);
1050 # push some response handlers
1052 $handle->push_read (...);
1053 }
1054 This means you can queue as many requests as you want, and while
1056 AnyEvent::Handle goes through its read queue to handle the response
1057 data, the other side can work on the next request - queueing the
1058 request just appends some data to the write queue and installs a
1059 handler to be called later.
1060 You might ask yourself how to handle decisions you can only make after
1062 you have received some data (such as handling a short error response or
1063 a long and differently-formatted response). The answer to this problem
1064 is "unshift_read", which we will introduce together with an example in
1065 the coming sections.
1066 Using "http_get"
1068 Finally, here is how you would use "http_get":
1070 http_get "www.google.com", "/", sub {
1072 my ($response, $header, $body) = @_;
1073 print
1075 $response, "\n",
1076 $body;
1077 };
1078 And of course, you can run as many of these requests in parallel as you
1080 want (and your memory supports).
1081 HTTPS
1083 Now, as promised, let's implement the same thing for HTTPS, or more
1085 correctly, let's change our "http_get" function into a function that
1086 speaks HTTPS instead.
1087 HTTPS is a standard TLS connection (Transport Layer Security is the
1089 official name for what most people refer to as "SSL") that contains
1090 standard HTTP protocol exchanges. The only other difference to HTTP is
1091 that by default it uses port 443 instead of port 80.
1092 To implement these two differences we need two tiny changes, first, in
1094 the "connect" parameter, we replace "http" by "https" to connect to the
1095 https port:
1096 connect => [$host => 'https'],
1098 The other change deals with TLS, which is something AnyEvent::Handle
1100 does for us if the Net::SSLeay module is available. To enable TLS with
1101 AnyEvent::Handle, we pass an additional "tls" parameter to the call to
1102 "AnyEvent::Handle::new":
1103 tls => "connect",
1105 Specifying "tls" enables TLS, and the argument specifies whether
1107 AnyEvent::Handle is the server side ("accept") or the client side
1108 ("connect") for the TLS connection, as unlike TCP, there is a clear
1109 server/client relationship in TLS.
1110 That's all.
1112 Of course, all this should be handled transparently by "http_get" after
1114 parsing the URL. If you need this, see the part about exercising your
1115 inspiration earlier in this document. You could also use the
1116 AnyEvent::HTTP module from CPAN, which implements all this and works
1117 around a lot of quirks for you too.
1118 The read queue - revisited
1120 HTTP always uses the same structure in its responses, but many
1122 protocols require parsing responses differently depending on the
1123 response itself.
1124 For example, in SMTP, you normally get a single response line:
1126 220 mail.example.net Neverusesendmail 8.8.8 <mailme@example.net>
1128 But SMTP also supports multi-line responses:
1130 220-mail.example.net Neverusesendmail 8.8.8 <mailme@example.net>
1132 220-hey guys
1133 220 my response is longer than yours
1134 To handle this, we need "unshift_read". As the name (we hope) implies,
1136 "unshift_read" will not append your read request to the end of the read
1137 queue, but will prepend it to the queue instead.
1138 This is useful in the situation above: Just push your response-line
1140 read request when sending the SMTP command, and when handling it, you
1141 look at the line to see if more is to come, and "unshift_read" another
1142 reader callback if required, like this:
1143 my $response; # response lines end up in here
1145 my $read_response; $read_response = sub {
1147 my ($handle, $line) = @_;
1148 $response .= "$line\n";
1150 # check for continuation lines ("-" as 4th character")
1152 if ($line =~ /^...-/) {
1153 # if yes, then unshift another line read
1154 $handle->unshift_read (line => $read_response);
1155 } else {
1157 # otherwise we are done
1158 # free callback
1160 undef $read_response;
1161 print "we are don reading: $response\n";
1163 }
1164 };
1165 $handle->push_read (line => $read_response);
1167 This recipe can be used for all similar parsing problems, for example
1169 in NNTP, the response code to some commands indicates that more data
1170 will be sent:
1171 $handle->push_write ("article 42");
1173 # read response line
1175 $handle->push_read (line => sub {
1176 my ($handle, $status) = @_;
1177 # article data following?
1179 if ($status =~ /^2/) {
1180 # yes, read article body
1181 $handle->unshift_read (line => "\012.\015\012", sub {
1183 my ($handle, $body) = @_;
1184 $finish->($status, $body);
1186 });
1187 } else {
1189 # some error occured, no article data
1190 $finish->($status);
1192 }
1193 }
1194 Your own read queue handler
1196 Sometimes your protocol doesn't play nice, and uses lines or chunks of
1198 data not formatted in a way handled out of the box by AnyEvent::Handle.
1199 In this case you have to implement your own read parser.
1200 To make up a contorted example, imagine you are looking for an even
1202 number of characters followed by a colon (":"). Also imagine that
1203 AnyEvent::Handle has no "regex" read type which could be used, so you'd
1204 have to do it manually.
1205 To implement a read handler for this, you would "push_read" (or
1207 "unshift_read") a single code reference.
1208 This code reference will then be called each time there is (new) data
1210 available in the read buffer, and is expected to either successfully
1211 eat/consume some of that data (and return true) or to return false to
1212 indicate that it wants to be called again.
1213 If the code reference returns true, then it will be removed from the
1215 read queue (because it has parsed/consumed whatever it was supposed to
1216 consume), otherwise it stays in the front of it.
1217 The example above could be coded like this:
1219 $handle->push_read (sub {
1221 my ($handle) = @_;
1222 # check for even number of characters + ":"
1224 # and remove the data if a match is found.
1225 # if not, return false (actually nothing)
1226 $handle->{rbuf} =~ s/^( (?:..)* ) ://x
1228 or return;
1229 # we got some data in $1, pass it to whoever wants it
1231 $finish->($1);
1232 # and return true to indicate we are done
1234 1
1235 });
1236
1238 Now that you have seen how to use AnyEvent, here's what to use when you
1239 don't use it correctly, or simply hit a bug somewhere and want to debug
1240 it:
1241 Enable strict argument checking during development
1243 AnyEvent does not, by default, do any argument checking. This can
1244 lead to strange and unexpected results especially if you are just
1245 trying to find your way with AnyEvent.
1246 AnyEvent supports a special "strict" mode - off by default - which
1248 does very strict argument checking, at the expense of slowing down
1249 your program. During development, however, this mode is very useful
1250 because it quickly catches the msot common errors.
1251 You can enable this strict mode either by having an environment
1253 variable "AE_STRICT" with a true value in your environment:
1254 AE_STRICT=1 perl myprog
1256 Or you can write "use AnyEvent::Strict" in your program, which has
1258 the same effect (do not do this in production, however).
1259 Increase verbosity, configure logging
1261 AnyEvent, by default, only logs critical messages. If something
1262 doesn't work, maybe there was a warning about it that you didn't
1263 see because it was suppressed.
1264 So during development it is recommended to push up the logging
1266 level to at least warn level (5):
1267 AE_VERBOSE=5 perl myprog
1269 Other levels that might be helpful are debug (8) or even trace (9).
1271 AnyEvent's logging is quite versatile - the AnyEvent::Log manpage
1273 has all the details.
1274 Watcher wrapping, tracing, the shell
1276 For even more debugging, you can enable watcher wrapping:
1277 AE_DEBUG_WRAP=2 perl myprog
1279 This will have the effect of wrapping every watcher into a special
1281 object that stores a backtrace of when it was created, stores a
1282 backtrace when an exception occurs during watcher execution, and
1283 stores a lot of other information. If that slows down your program
1284 too much, then "AE_DEBUG_WRAP=1" avoids the costly backtraces.
1285 Here is an example of what of information is stored:
1287 59148536 DC::DB:472(Server::run)>io>DC::DB::Server::fh_read
1289 type: io watcher
1290 args: poll r fh GLOB(0x35283f0)
1291 created: 2011-09-01 23:13:46.597336 +0200 (1314911626.59734)
1292 file: ./blib/lib/Deliantra/Client/private/DC/DB.pm
1293 line: 472
1294 subname: DC::DB::Server::run
1295 context:
1296 tracing: enabled
1297 cb: CODE(0x2d1fb98) (DC::DB::Server::fh_read)
1298 invoked: 0 times
1299 created
1300 (eval 25) line 6 AnyEvent::Debug::Wrap::__ANON__('AnyEvent','fh',GLOB(0x35283f0),'poll','r','cb',CODE(0x2d1fb98)=DC::DB::Server::fh_read)
1301 DC::DB line 472 AE::io(GLOB(0x35283f0),'0',CODE(0x2d1fb98)=DC::DB::Server::fh_read)
1302 bin/deliantra line 2776 DC::DB::Server::run()
1303 bin/deliantra line 2941 main::main()
1304 There are many ways to get at this data - see the AnyEvent::Debug
1306 and AnyEvent::Log manpages for more details.
1307 The most interesting and interactive way is to create a debug
1309 shell, for example by setting "AE_DEBUG_SHELL":
1310 AE_DEBUG_WRAP=2 AE_DEBUG_SHELL=$HOME/myshell ./myprog
1312 # while myprog is running:
1314 socat readline $HOME/myshell
1315 Note that anybody who can access $HOME/myshell can make your
1317 program do anything he or she wants, so if you are not the only
1318 user on your machine, better put it into a secure location ($HOME
1319 might not be secure enough).
1320 If you don't have "socat" (a shame!) and care even less about
1322 security, you can also use TCP and "telnet":
1323 AE_DEBUG_WRAP=2 AE_DEBUG_SHELL=127.0.0.1:1234 ./myprog
1325 telnet 127.0.0.1 1234
1327 The debug shell can enable and disable tracing of watcher
1329 invocations, can display the trace output, give you a list of
1330 watchers and lets you investigate watchers in detail.
1331 This concludes our little tutorial.
1333
1335 This introduction should have explained the key concepts of AnyEvent -
1336 event watchers and condition variables, AnyEvent::Socket - basic
1337 networking utilities, and AnyEvent::Handle - a nice wrapper around
1338 sockets.
1339 You could either start coding stuff right away, look at those manual
1341 pages for the gory details, or roam CPAN for other AnyEvent modules
1342 (such as AnyEvent::IRC or AnyEvent::HTTP) to see more code examples (or
1343 simply to use them).
1344 If you need a protocol that doesn't have an implementation using
1346 AnyEvent, remember that you can mix AnyEvent with one other event
1347 framework, such as POE, so you can always use AnyEvent for your own
1348 tasks plus modules of one other event framework to fill any gaps.
1349 And last not least, you could also look at Coro, especially
1351 Coro::AnyEvent, to see how you can turn event-based programming from
1352 callback style back to the usual imperative style (also called
1353 "inversion of control" - AnyEvent calls you, but Coro lets you call
1354 AnyEvent).
1355
1357 Robin Redeker "<elmex at ta-sa.org>", Marc Lehmann
1358 <schmorp@schmorp.de>.
1359perl v5.36.0 2022-07-22 AnyEvent::Intro(3)