1lwpcook(3) User Contributed Perl Documentation lwpcook(3)
2
6 lwpcook - The libwww-perl cookbook
7
9 This document contain some examples that show typical usage of the
10 libwww-perl library. You should consult the documentation for the
11 individual modules for more detail.
12 All examples should be runnable programs. You can, in most cases, test
14 the code sections by piping the program text directly to perl.
15
17 It is very easy to use this library to just fetch documents from the
18 net. The LWP::Simple module provides the get() function that return
19 the document specified by its URL argument:
20 use LWP::Simple;
22 $doc = get 'http://search.cpan.org/dist/libwww-perl/';
23 or, as a perl one-liner using the getprint() function:
25 perl -MLWP::Simple -e 'getprint "http://search.cpan.org/dist/libwww-perl/"'
27 or, how about fetching the latest perl by running this command:
29 perl -MLWP::Simple -e '
31 getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
32 "perl.tar.gz"'
33 You will probably first want to find a CPAN site closer to you by
35 running something like the following command:
36 perl -MLWP::Simple -e 'getprint "http://www.cpan.org/SITES.html"'
38 Enough of this simple stuff! The LWP object oriented interface gives
40 you more control over the request sent to the server. Using this
41 interface you have full control over headers sent and how you want to
42 handle the response returned.
43 use LWP::UserAgent;
45 $ua = LWP::UserAgent->new;
46 $ua->agent("$0/0.1 " . $ua->agent);
47 # $ua->agent("Mozilla/8.0") # pretend we are very capable browser
48 $req = HTTP::Request->new(
50 GET => 'http://search.cpan.org/dist/libwww-perl/');
51 $req->header('Accept' => 'text/html');
52 # send request
54 $res = $ua->request($req);
55 # check the outcome
57 if ($res->is_success) {
58 print $res->decoded_content;
59 }
60 else {
61 print "Error: " . $res->status_line . "\n";
62 }
63 The lwp-request program (alias GET) that is distributed with the
65 library can also be used to fetch documents from WWW servers.
66
68 If you just want to check if a document is present (i.e. the URL is
69 valid) try to run code that looks like this:
70 use LWP::Simple;
72 if (head($url)) {
74 # ok document exists
75 }
76 The head() function really returns a list of meta-information about the
78 document. The first three values of the list returned are the document
79 type, the size of the document, and the age of the document.
80 More control over the request or access to all header values returned
82 require that you use the object oriented interface described for GET
83 above. Just s/GET/HEAD/g.
84
86 There is no simple procedural interface for posting data to a WWW
87 server. You must use the object oriented interface for this. The most
88 common POST operation is to access a WWW form application:
89 use LWP::UserAgent;
91 $ua = LWP::UserAgent->new;
92 my $req = HTTP::Request->new(
94 POST => 'http://rt.cpan.org/Public/Dist/Display.html');
95 $req->content_type('application/x-www-form-urlencoded');
96 $req->content('Status=Active&Name=libwww-perl');
97 my $res = $ua->request($req);
99 print $res->as_string;
100 Lazy people use the HTTP::Request::Common module to set up a suitable
102 POST request message (it handles all the escaping issues) and has a
103 suitable default for the content_type:
104 use HTTP::Request::Common qw(POST);
106 use LWP::UserAgent;
107 $ua = LWP::UserAgent->new;
108 my $req = POST 'http://rt.cpan.org/Public/Dist/Display.html',
110 [ Status => 'Active', Name => 'libwww-perl' ];
111 print $ua->request($req)->as_string;
113 The lwp-request program (alias POST) that is distributed with the
115 library can also be used for posting data.
116
118 Some sites use proxies to go through fire wall machines, or just as
119 cache in order to improve performance. Proxies can also be used for
120 accessing resources through protocols not supported directly (or
121 supported badly :-) by the libwww-perl library.
122 You should initialize your proxy setting before you start sending
124 requests:
125 use LWP::UserAgent;
127 $ua = LWP::UserAgent->new;
128 $ua->env_proxy; # initialize from environment variables
129 # or
130 $ua->proxy(ftp => 'http://proxy.myorg.com');
131 $ua->proxy(wais => 'http://proxy.myorg.com');
132 $ua->no_proxy(qw(no se fi));
133 my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
135 print $ua->request($req)->as_string;
136 The LWP::Simple interface will call env_proxy() for you automatically.
138 Applications that use the $ua->env_proxy() method will normally not use
139 the $ua->proxy() and $ua->no_proxy() methods.
140 Some proxies also require that you send it a username/password in order
142 to let requests through. You should be able to add the required
143 header, with something like this:
144 use LWP::UserAgent;
146 $ua = LWP::UserAgent->new;
148 $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
149 $req = HTTP::Request->new('GET',"http://www.perl.com");
151 $res = $ua->request($req);
153 print $res->decoded_content if $res->is_success;
154 Replace "proxy.myorg.com", "username" and "password" with something
156 suitable for your site.
157
159 Documents protected by basic authorization can easily be accessed like
160 this:
161 use LWP::UserAgent;
163 $ua = LWP::UserAgent->new;
164 $req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
165 $req->authorization_basic('aas', 'mypassword');
166 print $ua->request($req)->as_string;
167 The other alternative is to provide a subclass of LWP::UserAgent that
169 overrides the get_basic_credentials() method. Study the lwp-request
170 program for an example of this.
171
173 Some sites like to play games with cookies. By default LWP ignores
174 cookies provided by the servers it visits. LWP will collect cookies
175 and respond to cookie requests if you set up a cookie jar.
176 use LWP::UserAgent;
178 use HTTP::Cookies;
179 $ua = LWP::UserAgent->new;
181 $ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
182 autosave => 1));
183 # and then send requests just as you used to do
185 $res = $ua->request(HTTP::Request->new(GET => "http://no.yahoo.com/"));
186 print $res->status_line, "\n";
187 As you visit sites that send you cookies to keep, then the file
189 lwpcookies.txt" will grow.
190
192 URLs with https scheme are accessed in exactly the same way as with
193 http scheme, provided that an SSL interface module for LWP has been
194 properly installed (see the README.SSL file found in the libwww-perl
195 distribution for more details). If no SSL interface is installed for
196 LWP to use, then you will get "501 Protocol scheme 'https' is not
197 supported" errors when accessing such URLs.
198 Here's an example of fetching and printing a WWW page using SSL:
200 use LWP::UserAgent;
202 my $ua = LWP::UserAgent->new;
204 my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
205 my $res = $ua->request($req);
206 if ($res->is_success) {
207 print $res->as_string;
208 }
209 else {
210 print "Failed: ", $res->status_line, "\n";
211 }
212
214 If you want to mirror documents from a WWW server, then try to run code
215 similar to this at regular intervals:
216 use LWP::Simple;
218 %mirrors = (
220 'http://www.sn.no/' => 'sn.html',
221 'http://www.perl.com/' => 'perl.html',
222 'http://search.cpan.org/distlibwww-perl/' => 'lwp.html',
223 'gopher://gopher.sn.no/' => 'gopher.html',
224 );
225 while (($url, $localfile) = each(%mirrors)) {
227 mirror($url, $localfile);
228 }
229 Or, as a perl one-liner:
231 perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';
233 The document will not be transferred unless it has been updated.
235
237 If the document you want to fetch is too large to be kept in memory,
238 then you have two alternatives. You can instruct the library to write
239 the document content to a file (second $ua->request() argument is a
240 file name):
241 use LWP::UserAgent;
243 $ua = LWP::UserAgent->new;
244 my $req = HTTP::Request->new(GET =>
246 'http://www.cpan.org/authors/Gisle_Aas/libwww-perl-6.02.tar.gz');
247 $res = $ua->request($req, "libwww-perl.tar.gz");
248 if ($res->is_success) {
249 print "ok\n";
250 }
251 else {
252 print $res->status_line, "\n";
253 }
254 Or you can process the document as it arrives (second $ua->request()
256 argument is a code reference):
257 use LWP::UserAgent;
259 $ua = LWP::UserAgent->new;
260 $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt';
261 my $expected_length;
263 my $bytes_received = 0;
264 my $res =
265 $ua->request(HTTP::Request->new(GET => $URL),
266 sub {
267 my($chunk, $res) = @_;
268 $bytes_received += length($chunk);
269 unless (defined $expected_length) {
270 $expected_length = $res->content_length || 0;
271 }
272 if ($expected_length) {
273 printf STDERR "%d%% - ",
274 100 * $bytes_received / $expected_length;
275 }
276 print STDERR "$bytes_received bytes received\n";
277 # XXX Should really do something with the chunk itself
279 # print $chunk;
280 });
281 print $res->status_line, "\n";
282
284 Copyright 1996-2001, Gisle Aas
285 This library is free software; you can redistribute it and/or modify it
287 under the same terms as Perl itself.
288perl v5.16.3 2012-02-11 lwpcook(3)