1lwpcook(3)            User Contributed Perl Documentation           lwpcook(3)
2
3
4

NAME

6       lwpcook - The libwww-perl cookbook
7

DESCRIPTION

9       This document contain some examples that show typical usage of the
10       libwww-perl library.  You should consult the documentation for the
11       individual modules for more detail.
12
13       All examples should be runnable programs. You can, in most cases, test
14       the code sections by piping the program text directly to perl.
15

GET

17       It is very easy to use this library to just fetch documents from the
18       net.  The LWP::Simple module provides the get() function that return
19       the document specified by its URL argument:
20
21         use LWP::Simple;
22         $doc = get 'http://search.cpan.org/dist/libwww-perl/';
23
24       or, as a perl one-liner using the getprint() function:
25
26         perl -MLWP::Simple -e 'getprint "http://search.cpan.org/dist/libwww-perl/"'
27
28       or, how about fetching the latest perl by running this command:
29
30         perl -MLWP::Simple -e '
31           getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
32                    "perl.tar.gz"'
33
34       You will probably first want to find a CPAN site closer to you by
35       running something like the following command:
36
37         perl -MLWP::Simple -e 'getprint "http://www.cpan.org/SITES.html"'
38
39       Enough of this simple stuff!  The LWP object oriented interface gives
40       you more control over the request sent to the server.  Using this
41       interface you have full control over headers sent and how you want to
42       handle the response returned.
43
44         use LWP::UserAgent;
45         $ua = LWP::UserAgent->new;
46         $ua->agent("$0/0.1 " . $ua->agent);
47         # $ua->agent("Mozilla/8.0") # pretend we are very capable browser
48
49         $req = HTTP::Request->new(
50            GET => 'http://search.cpan.org/dist/libwww-perl/');
51         $req->header('Accept' => 'text/html');
52
53         # send request
54         $res = $ua->request($req);
55
56         # check the outcome
57         if ($res->is_success) {
58            print $res->decoded_content;
59         }
60         else {
61            print "Error: " . $res->status_line . "\n";
62         }
63
64       The lwp-request program (alias GET) that is distributed with the
65       library can also be used to fetch documents from WWW servers.
66
68       If you just want to check if a document is present (i.e. the URL is
69       valid) try to run code that looks like this:
70
71         use LWP::Simple;
72
73         if (head($url)) {
74            # ok document exists
75         }
76
77       The head() function really returns a list of meta-information about the
78       document.  The first three values of the list returned are the document
79       type, the size of the document, and the age of the document.
80
81       More control over the request or access to all header values returned
82       require that you use the object oriented interface described for GET
83       above.  Just s/GET/HEAD/g.
84

POST

86       There is no simple procedural interface for posting data to a WWW
87       server.  You must use the object oriented interface for this. The most
88       common POST operation is to access a WWW form application:
89
90         use LWP::UserAgent;
91         $ua = LWP::UserAgent->new;
92
93         my $req = HTTP::Request->new(
94             POST => 'https://rt.cpan.org/Public/Dist/Display.html');
95         $req->content_type('application/x-www-form-urlencoded');
96         $req->content('Status=Active&Name=libwww-perl');
97
98         my $res = $ua->request($req);
99         print $res->as_string;
100
101       Lazy people use the HTTP::Request::Common module to set up a suitable
102       POST request message (it handles all the escaping issues) and has a
103       suitable default for the content_type:
104
105         use HTTP::Request::Common qw(POST);
106         use LWP::UserAgent;
107         $ua = LWP::UserAgent->new;
108
109         my $req = POST 'https://rt.cpan.org/Public/Dist/Display.html',
110                       [ Status => 'Active', Name => 'libwww-perl' ];
111
112         print $ua->request($req)->as_string;
113
114       The lwp-request program (alias POST) that is distributed with the
115       library can also be used for posting data.
116

PROXIES

118       Some sites use proxies to go through fire wall machines, or just as
119       cache in order to improve performance.  Proxies can also be used for
120       accessing resources through protocols not supported directly (or
121       supported badly :-) by the libwww-perl library.
122
123       You should initialize your proxy setting before you start sending
124       requests:
125
126         use LWP::UserAgent;
127         $ua = LWP::UserAgent->new;
128         $ua->env_proxy; # initialize from environment variables
129         # or
130         $ua->proxy(ftp  => 'http://proxy.myorg.com');
131         $ua->proxy(wais => 'http://proxy.myorg.com');
132         $ua->no_proxy(qw(no se fi));
133
134         my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
135         print $ua->request($req)->as_string;
136
137       The LWP::Simple interface will call env_proxy() for you automatically.
138       Applications that use the $ua->env_proxy() method will normally not use
139       the $ua->proxy() and $ua->no_proxy() methods.
140
141       Some proxies also require that you send it a username/password in order
142       to let requests through.  You should be able to add the required
143       header, with something like this:
144
145        use LWP::UserAgent;
146
147        $ua = LWP::UserAgent->new;
148        $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
149
150        $req = HTTP::Request->new('GET',"http://www.perl.com");
151
152        $res = $ua->request($req);
153        print $res->decoded_content if $res->is_success;
154
155       Replace "proxy.myorg.com", "username" and "password" with something
156       suitable for your site.
157

ACCESS TO PROTECTED DOCUMENTS

159       Documents protected by basic authorization can easily be accessed like
160       this:
161
162         use LWP::UserAgent;
163         $ua = LWP::UserAgent->new;
164         $req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
165         $req->authorization_basic('aas', 'mypassword');
166         print $ua->request($req)->as_string;
167
168       The other alternative is to provide a subclass of LWP::UserAgent that
169       overrides the get_basic_credentials() method. Study the lwp-request
170       program for an example of this.
171

COOKIES

173       Some sites like to play games with cookies.  By default LWP ignores
174       cookies provided by the servers it visits.  LWP will collect cookies
175       and respond to cookie requests if you set up a cookie jar. LWP doesn't
176       provide a cookie jar itself, but if you install HTTP::CookieJar::LWP,
177       it can be used like this:
178
179         use LWP::UserAgent;
180         use HTTP::CookieJar::LWP;
181
182         $ua = LWP::UserAgent->new(
183             cookie_jar => HTTP::CookieJar::LWP->new,
184         );
185
186         # and then send requests just as you used to do
187         $res = $ua->request(HTTP::Request->new(GET => "http://no.yahoo.com/"));
188         print $res->status_line, "\n";
189

HTTPS

191       URLs with https scheme are accessed in exactly the same way as with
192       http scheme, provided that an SSL interface module for LWP has been
193       properly installed (see the README.SSL file found in the libwww-perl
194       distribution for more details).  If no SSL interface is installed for
195       LWP to use, then you will get "501 Protocol scheme 'https' is not
196       supported" errors when accessing such URLs.
197
198       Here's an example of fetching and printing a WWW page using SSL:
199
200         use LWP::UserAgent;
201
202         my $ua = LWP::UserAgent->new;
203         my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
204         my $res = $ua->request($req);
205         if ($res->is_success) {
206             print $res->as_string;
207         }
208         else {
209             print "Failed: ", $res->status_line, "\n";
210         }
211

MIRRORING

213       If you want to mirror documents from a WWW server, then try to run code
214       similar to this at regular intervals:
215
216         use LWP::Simple;
217
218         %mirrors = (
219            'http://www.sn.no/'                       => 'sn.html',
220            'http://www.perl.com/'                    => 'perl.html',
221            'http://search.cpan.org/distlibwww-perl/' => 'lwp.html',
222            'gopher://gopher.sn.no/'                  => 'gopher.html',
223         );
224
225         while (($url, $localfile) = each(%mirrors)) {
226            mirror($url, $localfile);
227         }
228
229       Or, as a perl one-liner:
230
231         perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';
232
233       The document will not be transferred unless it has been updated.
234

LARGE DOCUMENTS

236       If the document you want to fetch is too large to be kept in memory,
237       then you have two alternatives.  You can instruct the library to write
238       the document content to a file (second $ua->request() argument is a
239       file name):
240
241         use LWP::UserAgent;
242         $ua = LWP::UserAgent->new;
243
244         my $req = HTTP::Request->new(GET =>
245             'http://www.cpan.org/CPAN/authors/id/O/OA/OALDERS/libwww-perl-6.26.tar.gz');
246         $res = $ua->request($req, "libwww-perl.tar.gz");
247         if ($res->is_success) {
248            print "ok\n";
249         }
250         else {
251            print $res->status_line, "\n";
252         }
253
254       Or you can process the document as it arrives (second $ua->request()
255       argument is a code reference):
256
257         use LWP::UserAgent;
258         $ua = LWP::UserAgent->new;
259         $URL = 'ftp://ftp.isc.org/pub/rfc/rfc-index.txt';
260
261         my $expected_length;
262         my $bytes_received = 0;
263         my $res =
264            $ua->request(HTTP::Request->new(GET => $URL),
265                      sub {
266                          my($chunk, $res) = @_;
267                          $bytes_received += length($chunk);
268                          unless (defined $expected_length) {
269                             $expected_length = $res->content_length || 0;
270                          }
271                          if ($expected_length) {
272                               printf STDERR "%d%% - ",
273                                         100 * $bytes_received / $expected_length;
274                          }
275                          print STDERR "$bytes_received bytes received\n";
276
277                          # XXX Should really do something with the chunk itself
278                          # print $chunk;
279                      });
280          print $res->status_line, "\n";
281
283       Copyright 1996-2001, Gisle Aas
284
285       This library is free software; you can redistribute it and/or modify it
286       under the same terms as Perl itself.
287
288
289
290perl v5.28.0                      2018-07-16                        lwpcook(3)
Impressum