1lwpcook(3) User Contributed Perl Documentation lwpcook(3)
2
6 lwpcook - The libwww-perl cookbook
7
9 This document contain some examples that show typical usage of the
10 libwww-perl library. You should consult the documentation for the
11 individual modules for more detail.
12 All examples should be runnable programs. You can, in most cases, test
14 the code sections by piping the program text directly to perl.
15
17 It is very easy to use this library to just fetch documents from the
18 net. The LWP::Simple module provides the get() function that return
19 the document specified by its URL argument:
20 use LWP::Simple;
22 $doc = get 'http://www.linpro.no/lwp/';
23 or, as a perl one-liner using the getprint() function:
25 perl -MLWP::Simple -e 'getprint "http://www.linpro.no/lwp/"'
27 or, how about fetching the latest perl by running this command:
29 perl -MLWP::Simple -e '
31 getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
32 "perl.tar.gz"'
33 You will probably first want to find a CPAN site closer to you by
35 running something like the following command:
36 perl -MLWP::Simple -e 'getprint "http://www.perl.com/perl/CPAN/CPAN.html"'
38 Enough of this simple stuff! The LWP object oriented interface gives
40 you more control over the request sent to the server. Using this
41 interface you have full control over headers sent and how you want to
42 handle the response returned.
43 use LWP::UserAgent;
45 $ua = LWP::UserAgent->new;
46 $ua->agent("$0/0.1 " . $ua->agent);
47 # $ua->agent("Mozilla/8.0") # pretend we are very capable browser
48 $req = HTTP::Request->new(GET => 'http://www.linpro.no/lwp');
50 $req->header('Accept' => 'text/html');
51 # send request
53 $res = $ua->request($req);
54 # check the outcome
56 if ($res->is_success) {
57 print $res->decoded_content;
58 }
59 else {
60 print "Error: " . $res->status_line . "\n";
61 }
62 The lwp-request program (alias GET) that is distributed with the
64 library can also be used to fetch documents from WWW servers.
65
67 If you just want to check if a document is present (i.e. the URL is
68 valid) try to run code that looks like this:
69 use LWP::Simple;
71 if (head($url)) {
73 # ok document exists
74 }
75 The head() function really returns a list of meta-information about the
77 document. The first three values of the list returned are the document
78 type, the size of the document, and the age of the document.
79 More control over the request or access to all header values returned
81 require that you use the object oriented interface described for GET
82 above. Just s/GET/HEAD/g.
83
85 There is no simple procedural interface for posting data to a WWW
86 server. You must use the object oriented interface for this. The most
87 common POST operation is to access a WWW form application:
88 use LWP::UserAgent;
90 $ua = LWP::UserAgent->new;
91 my $req = HTTP::Request->new(POST => 'http://www.perl.com/cgi-bin/BugGlimpse');
93 $req->content_type('application/x-www-form-urlencoded');
94 $req->content('match=www&errors=0');
95 my $res = $ua->request($req);
97 print $res->as_string;
98 Lazy people use the HTTP::Request::Common module to set up a suitable
100 POST request message (it handles all the escaping issues) and has a
101 suitable default for the content_type:
102 use HTTP::Request::Common qw(POST);
104 use LWP::UserAgent;
105 $ua = LWP::UserAgent->new;
106 my $req = POST 'http://www.perl.com/cgi-bin/BugGlimpse',
108 [ search => 'www', errors => 0 ];
109 print $ua->request($req)->as_string;
111 The lwp-request program (alias POST) that is distributed with the
113 library can also be used for posting data.
114
116 Some sites use proxies to go through fire wall machines, or just as
117 cache in order to improve performance. Proxies can also be used for
118 accessing resources through protocols not supported directly (or
119 supported badly :-) by the libwww-perl library.
120 You should initialize your proxy setting before you start sending
122 requests:
123 use LWP::UserAgent;
125 $ua = LWP::UserAgent->new;
126 $ua->env_proxy; # initialize from environment variables
127 # or
128 $ua->proxy(ftp => 'http://proxy.myorg.com');
129 $ua->proxy(wais => 'http://proxy.myorg.com');
130 $ua->no_proxy(qw(no se fi));
131 my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
133 print $ua->request($req)->as_string;
134 The LWP::Simple interface will call env_proxy() for you automatically.
136 Applications that use the $ua->env_proxy() method will normally not use
137 the $ua->proxy() and $ua->no_proxy() methods.
138 Some proxies also require that you send it a username/password in order
140 to let requests through. You should be able to add the required
141 header, with something like this:
142 use LWP::UserAgent;
144 $ua = LWP::UserAgent->new;
146 $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
147 $req = HTTP::Request->new('GET',"http://www.perl.com");
149 $res = $ua->request($req);
151 print $res->decoded_content if $res->is_success;
152 Replace "proxy.myorg.com", "username" and "password" with something
154 suitable for your site.
155
157 Documents protected by basic authorization can easily be accessed like
158 this:
159 use LWP::UserAgent;
161 $ua = LWP::UserAgent->new;
162 $req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
163 $req->authorization_basic('aas', 'mypassword');
164 print $ua->request($req)->as_string;
165 The other alternative is to provide a subclass of LWP::UserAgent that
167 overrides the get_basic_credentials() method. Study the lwp-request
168 program for an example of this.
169
171 Some sites like to play games with cookies. By default LWP ignores
172 cookies provided by the servers it visits. LWP will collect cookies
173 and respond to cookie requests if you set up a cookie jar.
174 use LWP::UserAgent;
176 use HTTP::Cookies;
177 $ua = LWP::UserAgent->new;
179 $ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
180 autosave => 1));
181 # and then send requests just as you used to do
183 $res = $ua->request(HTTP::Request->new(GET => "http://www.yahoo.no"));
184 print $res->status_line, "\n";
185 As you visit sites that send you cookies to keep, then the file
187 lwpcookies.txt" will grow.
188
190 URLs with https scheme are accessed in exactly the same way as with
191 http scheme, provided that an SSL interface module for LWP has been
192 properly installed (see the README.SSL file found in the libwww-perl
193 distribution for more details). If no SSL interface is installed for
194 LWP to use, then you will get "501 Protocol scheme 'https' is not
195 supported" errors when accessing such URLs.
196 Here's an example of fetching and printing a WWW page using SSL:
198 use LWP::UserAgent;
200 my $ua = LWP::UserAgent->new;
202 my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
203 my $res = $ua->request($req);
204 if ($res->is_success) {
205 print $res->as_string;
206 }
207 else {
208 print "Failed: ", $res->status_line, "\n";
209 }
210
212 If you want to mirror documents from a WWW server, then try to run code
213 similar to this at regular intervals:
214 use LWP::Simple;
216 %mirrors = (
218 'http://www.sn.no/' => 'sn.html',
219 'http://www.perl.com/' => 'perl.html',
220 'http://www.sn.no/libwww-perl/' => 'lwp.html',
221 'gopher://gopher.sn.no/' => 'gopher.html',
222 );
223 while (($url, $localfile) = each(%mirrors)) {
225 mirror($url, $localfile);
226 }
227 Or, as a perl one-liner:
229 perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';
231 The document will not be transferred unless it has been updated.
233
235 If the document you want to fetch is too large to be kept in memory,
236 then you have two alternatives. You can instruct the library to write
237 the document content to a file (second $ua->request() argument is a
238 file name):
239 use LWP::UserAgent;
241 $ua = LWP::UserAgent->new;
242 my $req = HTTP::Request->new(GET =>
244 'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz');
245 $res = $ua->request($req, "libwww-perl.tar.gz");
246 if ($res->is_success) {
247 print "ok\n";
248 }
249 else {
250 print $res->status_line, "\n";
251 }
252 Or you can process the document as it arrives (second $ua->request()
254 argument is a code reference):
255 use LWP::UserAgent;
257 $ua = LWP::UserAgent->new;
258 $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt';
259 my $expected_length;
261 my $bytes_received = 0;
262 my $res =
263 $ua->request(HTTP::Request->new(GET => $URL),
264 sub {
265 my($chunk, $res) = @_;
266 $bytes_received += length($chunk);
267 unless (defined $expected_length) {
268 $expected_length = $res->content_length || 0;
269 }
270 if ($expected_length) {
271 printf STDERR "%d%% - ",
272 100 * $bytes_received / $expected_length;
273 }
274 print STDERR "$bytes_received bytes received\n";
275 # XXX Should really do something with the chunk itself
277 # print $chunk;
278 });
279 print $res->status_line, "\n";
280
282 Copyright 1996-2001, Gisle Aas
283 This library is free software; you can redistribute it and/or modify it
285 under the same terms as Perl itself.
286perl v5.10.1 2009-06-15 lwpcook(3)