1lwpcook(3) User Contributed Perl Documentation lwpcook(3)
2
6 lwpcook - The libwww-perl cookbook
7
9 This document contain some examples that show typical usage of the
10 libwww-perl library. You should consult the documentation for the
11 individual modules for more detail.
12 All examples should be runnable programs. You can, in most cases, test
14 the code sections by piping the program text directly to perl.
15
17 It is very easy to use this library to just fetch documents from the
18 net. The LWP::Simple module provides the get() function that return
19 the document specified by its URL argument:
20 use LWP::Simple;
22 $doc = get 'http://search.cpan.org/dist/libwww-perl/';
23 or, as a perl one-liner using the getprint() function:
25 perl -MLWP::Simple -e 'getprint "http://search.cpan.org/dist/libwww-perl/"'
27 or, how about fetching the latest perl by running this command:
29 perl -MLWP::Simple -e '
31 getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
32 "perl.tar.gz"'
33 You will probably first want to find a CPAN site closer to you by
35 running something like the following command:
36 perl -MLWP::Simple -e 'getprint "http://www.cpan.org/SITES.html"'
38 Enough of this simple stuff! The LWP object oriented interface gives
40 you more control over the request sent to the server. Using this
41 interface you have full control over headers sent and how you want to
42 handle the response returned.
43 use LWP::UserAgent;
45 $ua = LWP::UserAgent->new;
46 $ua->agent("$0/0.1 " . $ua->agent);
47 # $ua->agent("Mozilla/8.0") # pretend we are very capable browser
48 $req = HTTP::Request->new(
50 GET => 'http://search.cpan.org/dist/libwww-perl/');
51 $req->header('Accept' => 'text/html');
52 # send request
54 $res = $ua->request($req);
55 # check the outcome
57 if ($res->is_success) {
58 print $res->decoded_content;
59 }
60 else {
61 print "Error: " . $res->status_line . "\n";
62 }
63 The lwp-request program (alias GET) that is distributed with the
65 library can also be used to fetch documents from WWW servers.
66
68 If you just want to check if a document is present (i.e. the URL is
69 valid) try to run code that looks like this:
70 use LWP::Simple;
72 if (head($url)) {
74 # ok document exists
75 }
76 The head() function really returns a list of meta-information about the
78 document. The first three values of the list returned are the document
79 type, the size of the document, and the age of the document.
80 More control over the request or access to all header values returned
82 require that you use the object oriented interface described for GET
83 above. Just s/GET/HEAD/g.
84
86 There is no simple procedural interface for posting data to a WWW
87 server. You must use the object oriented interface for this. The most
88 common POST operation is to access a WWW form application:
89 use LWP::UserAgent;
91 $ua = LWP::UserAgent->new;
92 my $req = HTTP::Request->new(
94 POST => 'https://rt.cpan.org/Public/Dist/Display.html');
95 $req->content_type('application/x-www-form-urlencoded');
96 $req->content('Status=Active&Name=libwww-perl');
97 my $res = $ua->request($req);
99 print $res->as_string;
100 Lazy people use the HTTP::Request::Common module to set up a suitable
102 POST request message (it handles all the escaping issues) and has a
103 suitable default for the content_type:
104 use HTTP::Request::Common qw(POST);
106 use LWP::UserAgent;
107 $ua = LWP::UserAgent->new;
108 my $req = POST 'https://rt.cpan.org/Public/Dist/Display.html',
110 [ Status => 'Active', Name => 'libwww-perl' ];
111 print $ua->request($req)->as_string;
113 The lwp-request program (alias POST) that is distributed with the
115 library can also be used for posting data.
116
118 Some sites use proxies to go through fire wall machines, or just as
119 cache in order to improve performance. Proxies can also be used for
120 accessing resources through protocols not supported directly (or
121 supported badly :-) by the libwww-perl library.
122 You should initialize your proxy setting before you start sending
124 requests:
125 use LWP::UserAgent;
127 $ua = LWP::UserAgent->new;
128 $ua->env_proxy; # initialize from environment variables
129 # or
130 $ua->proxy(ftp => 'http://proxy.myorg.com');
131 $ua->proxy(wais => 'http://proxy.myorg.com');
132 $ua->no_proxy(qw(no se fi));
133 my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
135 print $ua->request($req)->as_string;
136 The LWP::Simple interface will call env_proxy() for you automatically.
138 Applications that use the $ua->env_proxy() method will normally not use
139 the $ua->proxy() and $ua->no_proxy() methods.
140 Some proxies also require that you send it a username/password in order
142 to let requests through. You should be able to add the required
143 header, with something like this:
144 use LWP::UserAgent;
146 $ua = LWP::UserAgent->new;
148 $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
149 $req = HTTP::Request->new('GET',"http://www.perl.com");
151 $res = $ua->request($req);
153 print $res->decoded_content if $res->is_success;
154 Replace "proxy.myorg.com", "username" and "password" with something
156 suitable for your site.
157
159 Documents protected by basic authorization can easily be accessed like
160 this:
161 use LWP::UserAgent;
163 $ua = LWP::UserAgent->new;
164 $req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
165 $req->authorization_basic('aas', 'mypassword');
166 print $ua->request($req)->as_string;
167 The other alternative is to provide a subclass of LWP::UserAgent that
169 overrides the get_basic_credentials() method. Study the lwp-request
170 program for an example of this.
171
173 Some sites like to play games with cookies. By default LWP ignores
174 cookies provided by the servers it visits. LWP will collect cookies
175 and respond to cookie requests if you set up a cookie jar. LWP doesn't
176 provide a cookie jar itself, but if you install HTTP::CookieJar::LWP,
177 it can be used like this:
178 use LWP::UserAgent;
180 use HTTP::CookieJar::LWP;
181 $ua = LWP::UserAgent->new(
183 cookie_jar => HTTP::CookieJar::LWP->new,
184 );
185 # and then send requests just as you used to do
187 $res = $ua->request(HTTP::Request->new(GET => "http://no.yahoo.com/"));
188 print $res->status_line, "\n";
189
191 URLs with https scheme are accessed in exactly the same way as with
192 http scheme, provided that an SSL interface module for LWP has been
193 properly installed (see the README.SSL file found in the libwww-perl
194 distribution for more details). If no SSL interface is installed for
195 LWP to use, then you will get "501 Protocol scheme 'https' is not
196 supported" errors when accessing such URLs.
197 Here's an example of fetching and printing a WWW page using SSL:
199 use LWP::UserAgent;
201 my $ua = LWP::UserAgent->new;
203 my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
204 my $res = $ua->request($req);
205 if ($res->is_success) {
206 print $res->as_string;
207 }
208 else {
209 print "Failed: ", $res->status_line, "\n";
210 }
211
213 If you want to mirror documents from a WWW server, then try to run code
214 similar to this at regular intervals:
215 use LWP::Simple;
217 %mirrors = (
219 'http://www.sn.no/' => 'sn.html',
220 'http://www.perl.com/' => 'perl.html',
221 'http://search.cpan.org/distlibwww-perl/' => 'lwp.html',
222 'gopher://gopher.sn.no/' => 'gopher.html',
223 );
224 while (($url, $localfile) = each(%mirrors)) {
226 mirror($url, $localfile);
227 }
228 Or, as a perl one-liner:
230 perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';
232 The document will not be transferred unless it has been updated.
234
236 If the document you want to fetch is too large to be kept in memory,
237 then you have two alternatives. You can instruct the library to write
238 the document content to a file (second $ua->request() argument is a
239 file name):
240 use LWP::UserAgent;
242 $ua = LWP::UserAgent->new;
243 my $req = HTTP::Request->new(GET =>
245 'http://www.cpan.org/CPAN/authors/id/O/OA/OALDERS/libwww-perl-6.26.tar.gz');
246 $res = $ua->request($req, "libwww-perl.tar.gz");
247 if ($res->is_success) {
248 print "ok\n";
249 }
250 else {
251 print $res->status_line, "\n";
252 }
253 Or you can process the document as it arrives (second $ua->request()
255 argument is a code reference):
256 use LWP::UserAgent;
258 $ua = LWP::UserAgent->new;
259 $URL = 'ftp://ftp.isc.org/pub/rfc/rfc-index.txt';
260 my $expected_length;
262 my $bytes_received = 0;
263 my $res =
264 $ua->request(HTTP::Request->new(GET => $URL),
265 sub {
266 my($chunk, $res) = @_;
267 $bytes_received += length($chunk);
268 unless (defined $expected_length) {
269 $expected_length = $res->content_length || 0;
270 }
271 if ($expected_length) {
272 printf STDERR "%d%% - ",
273 100 * $bytes_received / $expected_length;
274 }
275 print STDERR "$bytes_received bytes received\n";
276 # XXX Should really do something with the chunk itself
278 # print $chunk;
279 });
280 print $res->status_line, "\n";
281
283 Copyright 1996-2001, Gisle Aas
284 This library is free software; you can redistribute it and/or modify it
286 under the same terms as Perl itself.
287perl v5.26.3 2018-06-05 lwpcook(3)