1POE::Component::Client:U:sHeTrTPC(o3n)tributed Perl DocuPmOeEn:t:aCtoimopnonent::Client::HTTP(3)
2
6 POE::Component::Client::HTTP - a HTTP user-agent component
7
9 use POE qw(Component::Client::HTTP);
10 POE::Component::Client::HTTP->spawn(
12 Agent => 'SpiffCrawler/0.90', # defaults to something long
13 Alias => 'ua', # defaults to 'weeble'
14 From => 'spiffster@perl.org', # defaults to undef (no header)
15 Protocol => 'HTTP/0.9', # defaults to 'HTTP/1.1'
16 Timeout => 60, # defaults to 180 seconds
17 MaxSize => 16384, # defaults to entire response
18 Streaming => 4096, # defaults to 0 (off)
19 FollowRedirects => 2 # defaults to 0 (off)
20 Proxy => "http://localhost:80", # defaults to HTTP_PROXY env. variable
21 NoProxy => [ "localhost", "127.0.0.1" ], # defs to NO_PROXY env. variable
22 );
23 $kernel->post(
25 'ua', # posts to the 'ua' alias
26 'request', # posts to ua's 'request' state
27 'response', # which of our states will receive the response
28 $request, # an HTTP::Request object
29 );
30 # This is the sub which is called when the session receives a
32 # 'response' event.
33 sub response_handler {
34 my ($request_packet, $response_packet) = @_[ARG0, ARG1];
35 # HTTP::Request
37 my $request_object = $request_packet->[0];
38 # HTTP::Response
40 my $response_object = $response_packet->[0];
41 my $stream_chunk;
43 if (! defined($response_object->content)) {
44 $stream_chunk = $response_packet->[1];
45 }
46 print(
48 "*" x 78, "\n",
49 "*** my request:\n",
50 "-" x 78, "\n",
51 $request_object->as_string(),
52 "*" x 78, "\n",
53 "*** their response:\n",
54 "-" x 78, "\n",
55 $response_object->as_string(),
56 );
57 if (defined $stream_chunk) {
59 print "-" x 40, "\n", $stream_chunk, "\n";
60 }
61 print "*" x 78, "\n";
63 }
64
66 POE::Component::Client::HTTP is an HTTP user-agent for POE. It lets
67 other sessions run while HTTP transactions are being processed, and it
68 lets several HTTP transactions be processed in parallel.
69 If POE::Component::Client::DNS is also installed, Client::HTTP will use
71 it to resolve hosts without blocking. Otherwise it will use gethostby‐
72 name(), which may have performance problems.
73 HTTP client components are not proper objects. Instead of being cre‐
75 ated, as most objects are, they are "spawned" as separate sessions. To
76 avoid confusion (and hopefully not cause other confusion), they must be
77 spawned with a "spawn" method, not created anew with a "new" one.
78
80 spawn
81 PoCo::Client::HTTP's "spawn" method takes a few named parameters:
83 Agent => $user_agent_string
85 Agent => \@list_of_agents
86 If a UserAgent header is not present in the HTTP::Request, a random
87 one will be used from those specified by the "Agent" parameter. If
88 none are supplied, POE::Component::Client::HTTP will advertise itself
89 to the server.
90 "Agent" may contain a reference to a list of user agents. If this is
92 the case, PoCo::Client::HTTP will choose one of them at random for
93 each request.
94 Alias => $session_alias
96 "Alias" sets the name by which the session will be known. If no
97 alias is given, the component defaults to "weeble". The alias lets
98 several sessions interact with HTTP components without keeping (or
99 even knowing) hard references to them. It's possible to spawn sev‐
100 eral HTTP components with different names.
101 ConnectionManager => $poco_client_keepalive
103 "ConnectionManager" sets this component's connection pool manager.
104 It expects the connection manager to be a reference to a POE::Compo‐
105 nent::Client::Keepalive object. The HTTP client component will call
106 "allocate()" on the connection manager itself so you should not have
107 done this already.
108 my $pool = POE::Component::Client::Keepalive->new(
110 keep_alive => 10, # seconds to keep connections alive
111 max_open => 100, # max concurrent connections - total
112 max_per_host => 20, # max concurrent connections - per host
113 timeout => 30, # max time (seconds) to establish a new connection
114 );
115 POE::Component::Client::HTTP->spawn(
117 # ...
118 ConnectionManager => $pool,
119 # ...
120 );
121 See POE::Component::Client::Keepalive for more information.
123 CookieJar => $cookie_jar
125 "CookieJar" sets the component's cookie jar. It expects the cookie
126 jar to be a reference to a HTTP::Cookies object.
127 From => $admin_address
129 "From" holds an e-mail address where the client's administrator
130 and/or maintainer may be reached. It defaults to undef, which means
131 no From header will be included in requests.
132 MaxSize => OCTETS
134 "MaxSize" specifies the largest response to accept from a server.
135 The content of larger responses will be truncated to OCTET octets.
136 This has been used to return the <head></head> section of web pages
137 without the need to wade through <body></body>.
138 NoProxy => [ $host_1, $host_2, ..., $host_N ]
140 NoProxy => "host1,host2,hostN"
141 "NoProxy" specifies a list of server hosts that will not be proxied.
142 It is useful for local hosts and hosts that do not properly support
143 proxying. If NoProxy is not specified, a list will be taken from the
144 NO_PROXY environment variable.
145 NoProxy => [ "localhost", "127.0.0.1" ],
147 NoProxy => "localhost,127.0.0.1",
148 Protocol => $http_protocol_string
150 "Protocol" advertises the protocol that the client wishes to see.
151 Under normal circumstances, it should be left to its default value:
152 "HTTP/1.1".
153 Proxy => [ $proxy_host, $proxy_port ]
155 Proxy => $proxy_url
156 Proxy => $proxy_url,$proxy_url,...
157 "Proxy" specifies one or more proxy hosts that requests will be
158 passed through. If not specified, proxy servers will be taken from
159 the HTTP_PROXY (or http_proxy) environment variable. No proxying
160 will occur unless Proxy is set or one of the environment variables
161 exists.
162 The proxy can be specified either as a host and port, or as one or
164 more URLs. Proxy URLs must specify the proxy port, even if it is 80.
165 Proxy => [ "127.0.0.1", 80 ],
167 Proxy => "http://127.0.0.1:80/",
168 "Proxy" may specify multiple proxies separated by commas.
170 PoCo::Client::HTTP will choose proxies from this list at random.
171 This is useful for load balancing requests through multiple gateways.
172 Proxy => "http://127.0.0.1:80/,http://127.0.0.1:81/",
174 Streaming => OCTETS
176 "Streaming" changes allows Client::HTTP to return large content in
177 chunks (of OCTETS octets each) rather than combine the entire content
178 into a single HTTP::Response object.
179 By default, Client::HTTP reads the entire content for a response into
181 memory before returning an HTTP::Response object. This is obviously
182 bad for applications like streaming MP3 clients, because they often
183 fetch songs that never end. Yes, they go on and on, my friend.
184 When "Streaming" is set to nonzero, however, the response handler
186 receives chunks of up to OCTETS octets apiece. The response handler
187 accepts slightly different parameters in this case. ARG0 is also an
188 HTTP::Response object but it does not contain response content, and
189 ARG1 contains a a chunk of raw response content, or undef if the
190 stream has ended.
191 sub streaming_response_handler {
193 my $response_packet = $_[ARG1];
194 my ($response, $data) = @$response_packet;
195 print SAVED_STREAM $data if defined $data;
196 }
197 FollowRedirects => $number_of_hops_to_follow
199 "FollowRedirects" specifies how many redirects (e.g. 302 Moved) to
200 follow. If not specified defaults to 0, and thus no redirection is
201 followed. This maintains compatibility with the previous behavior,
202 which was not to follow redirects at all.
203 If redirects are followed, a response chain should be built, and can
205 be accessed through $response_object->previous(). See HTTP::Response
206 for details here.
207 Timeout => $query_timeout
209 "Timeout" specifies the amount of time a HTTP request will wait for
210 an answer. This defaults to 180 seconds (three minutes).
211
213 Sessions communicate asynchronously with PoCo::Client::HTTP. They post
214 requests to it, and it posts responses back.
215 request
217 Requests are posted to the component's "request" state. They include
219 an HTTP::Request object which defines the request. For example:
220 $kernel->post(
222 'ua', 'request', # http session alias & state
223 'response', # my state to receive responses
224 GET 'http://poe.perl.org', # a simple HTTP request
225 'unique id', # a tag to identify the request
226 'progress', # an event to indicate progress
227 'http://1.2.3.4:80/' # proxy to use for this request
228 );
229 Requests include the state to which responses will be posted. In the
231 previous example, the handler for a 'response' state will be called
232 with each HTTP response. The "progress" handler is optional and if
233 installed, the component will provide progress metrics (see sample han‐
234 dler below). The "proxy" parameter is optional and if not defined, a
235 default proxy will be used if configured. No proxy will be used if
236 neither a default one nor a "proxy" parameter is defined.
237 pending_requests_count
239 There's also a pending_requests_count state that returns the number of
241 requests currently being processed. To receive the return value, it
242 must be invoked with $kernel->call().
243 my $count = $kernel->call('ua' => 'pending_requests_count');
245 cancel
247 Cancel a specific HTTP request. Requires a reference to the original
249 request (blessed or stringified) so it knows which one to cancel. See
250 "progress handler" below for notes on canceling streaming requests.
251 To cancel a request based on its blessed HTTP::Request object:
253 $kernel->post( component => cancel => $http_request );
255 To cancel a request based on its stringified HTTP::Request object:
257 $kernel->post( component => cancel => "$http_request" );
259 shutdown
261 Responds to all pending requests with 408 (request timeout), and then
263 shuts down the component and all subcomponents.
264
266 response handler
267 In addition to all the usual POE parameters, HTTP responses come with
269 two list references:
270 my ($request_packet, $response_packet) = @_[ARG0, ARG1];
272 $request_packet contains a reference to the original HTTP::Request
274 object. This is useful for matching responses back to the requests
275 that generated them.
276 my $http_request_object = $request_packet->[0];
278 my $http_request_tag = $request_packet->[1]; # from the 'request' post
279 $response_packet contains a reference to the resulting HTTP::Response
281 object.
282 my $http_response_object = $response_packet->[0];
284 Please see the HTTP::Request and HTTP::Response manpages for more
286 information.
287 progress handler
289 The example progress handler shows how to calculate a percentage of
291 download completion.
292 sub progress_handler {
294 my $gen_args = $_[ARG0]; # args passed to all calls
295 my $call_args = $_[ARG1]; # args specific to the call
296 my $req = $gen_args->[0]; # HTTP::Request object being serviced
298 my $tag = $gen_args->[1]; # Request ID tag from.
299 my $got = $call_args->[0]; # Number of bytes retrieved so far.
300 my $tot = $call_args->[1]; # Total bytes to be retrieved.
301 my $oct = $call_args->[2]; # Chunk of raw octets received this time.
302 my $percent = $got / $tot * 100;
304 printf(
306 "-- %.0f%% [%d/%d]: %s\n", $percent, $got, $tot, $req->uri()
307 );
308 # To cancel the request:
310 # $_[KERNEL]->post( component => cancel => $req );
311 }
312 DEPRECATION WARNING
314 The third return argument (the raw octets received) has been depre‐
316 cated. Instead of it, use the Streaming parameter to get chunks of
317 content in the response handler.
318
320 The HTTP::Request object passed to the request event can contain a CODE
321 reference as "content". This allows for sending large files without
322 wasting memory. Your callback should return a chunk of data each time
323 it is called, and an empty string when done. Don't forget to set the
324 Content-Length header correctly. Example:
325 my $request = HTTP::Request->new( PUT => 'http://...' );
327 my $file = '/path/to/large_file';
329 open my $fh, '<', $file;
331 my $upload_cb = sub {
333 if ( sysread $fh, my $buf, 4096 ) {
334 return $buf;
335 }
336 else {
337 close $fh;
338 return '';
339 }
340 };
341 $request->content_length( -s $file );
343 $request->content( $upload_cb );
345 $kernel->post( ua => request, 'response', $request );
347
349 POE::Component::Client::HTTP uses two standard environment variables:
350 HTTP_PROXY and NO_PROXY.
351 HTTP_PROXY sets the proxy server that Client::HTTP will forward
353 requests through. NO_PROXY sets a list of hosts that will not be for‐
354 warded through a proxy.
355 See the Proxy and NoProxy constructor parameters for more information
357 about these variables.
358
360 This component is built upon HTTP::Request, HTTP::Response, and POE.
361 Please see its source code and the documentation for its foundation
362 modules to learn more. If you want to use cookies, you'll need to read
363 about HTTP::Cookies as well.
364 Also see the test program, t/01_request.t, in the PoCo::Client::HTTP
366 distribution.
367
369 There is no support for CGI_PROXY or CgiProxy.
370 Secure HTTP (https) proxying is not supported at this time.
372 There is no object oriented interface. See POE::Compo‐
374 nent::Client::Keepalive and POE::Component::Client::DNS for examples of
375 a decent OO interface.
376
378 POE::Component::Client::HTTP is
379 · Copyright 1999-2006 Rocco Caputo
381 · Copyright 2004 Rob Bloodgood
383 · Copyright 2004-2005 Martijn van Beers
385 All rights are reserved. POE::Component::Client::HTTP is free soft‐
387 ware; you may redistribute it and/or modify it under the same terms as
388 Perl itself.
389
391 Joel Bernstein solved some nasty race conditions. Portugal Telecom
392 <http://www.sapo.pt/> was kind enough to support his contributions.
393 Jeff Bisbee added POD tests and documentation to pass several of them
395 to version 0.79. He's a kwalitee-increasing machine!
396
398 Rocco may be contacted by e-mail via <mailto:rcaputo@cpan.org>, and
399 Martijn may be contacted by email via <mailto:martijn@cpan.org>.
400 The preferred way to report bugs or requests is through RT though. See
402 <http://rt.cpan.org/NoAuth/Bugs.html?Dist=POE-Component-Client-HTTP> or
403 mail <mailto:bug-POE-Component-Client-HTTP@rt.cpan.org>
404 For questions, try the POE mailing list (poe@perl.org)
406perl v5.8.8 2006-10-14 POE::Component::Client::HTTP(3)