1WWW::Mechanize::FAQ(3)User Contributed Perl DocumentationWWW::Mechanize::FAQ(3)
2
6 WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize
7
9 version 2.00
10
12 If your question isn't answered here in the FAQ, please turn to the
13 communities at:
14 · StackOverflow
16 <https://stackoverflow.com/questions/tagged/www-mechanize>
17 · #lwp on irc.perl.org
19 · <http://perlmonks.org>
21 · The libwww-perl mailing list at <http://lists.perl.org>
23
25 I have this web page that has JavaScript on it, and my Mech program doesn't
26 work.
27 That's because WWW::Mechanize doesn't operate on the JavaScript. It
28 only understands the HTML parts of the page.
29 I thought Mech was supposed to work like a web browser.
31 It does pretty much, but it doesn't support JavaScript.
32 I added some basic attempts at picking up URLs in "window.open()" calls
34 and return them in "$mech->links". They work sometimes.
35 Since Javascript is completely visible to the client, it cannot be used
37 to prevent a scraper from following links. But it can make life
38 difficult. If you want to scrape specific pages, then a solution is
39 always possible.
40 One typical use of Javascript is to perform argument checking before
42 posting to the server. The URL you want is probably just buried in the
43 Javascript function. Do a regular expression match on
44 "$mech->content()" to find the link that you want and "$mech->get" it
45 directly (this assumes that you know what you are looking for in
46 advance).
47 In more difficult cases, the Javascript is used for URL mangling to
49 satisfy the needs of some middleware. In this case you need to figure
50 out what the Javascript is doing (why are these URLs always really
51 long?). There is probably some function with one or more arguments
52 which calculates the new URL. Step one: using your favorite browser,
53 get the before and after URLs and save them to files. Edit each file,
54 converting the argument separators ('?', '&' or ';') into newlines. Now
55 it is easy to use diff or comm to find out what Javascript did to the
56 URL. Step 2 - find the function call which created the URL - you will
57 need to parse and interpret its argument list. The Javascript Debugger
58 in the Firebug extension for Firefox helps with the analysis. At this
59 point, it is fairly trivial to write your own function which emulates
60 the Javascript for the pages you want to process.
61 Here's another approach that answers the question, "It works in
63 Firefox, but why not Mech?" Everything the web server knows about the
64 client is present in the HTTP request. If two requests are identical,
65 the results should be identical. So the real question is "What is
66 different between the mech request and the Firefox request?"
67 The Firefox extension "Tamper Data" is an effective tool for examining
69 the headers of the requests to the server. Compare that with what LWP
70 is sending. Once the two are identical, the action of the server should
71 be the same as well.
72 I say "should", because this is an oversimplification - some values are
74 naturally unique, e.g. a SessionID, but if a SessionID is present, that
75 is probably sufficient, even though the value will be different between
76 the LWP request and the Firefox request. The server could use the
77 session to store information which is troublesome, but that's not the
78 first place to look (and highly unlikely to be relevant when you are
79 requesting the login page of your site).
80 Generally the problem is to be found in missing or incorrect POSTDATA
82 arguments, Cookies, User-Agents, Accepts, etc. If you are using mech,
83 then redirects and cookies should not be a problem, but are listed here
84 for completeness. If you are missing headers, "$mech->add_header" can
85 be used to add the headers that you need.
86 Which modules work like Mechanize and have JavaScript support?
88 In no particular order: Gtk2::WebKit::Mechanize, Win32::IE::Mechanize,
89 WWW::Mechanize::Firefox, WWW::Scripter, WWW::Selenium
90
92 Can I do [such-and-such] with WWW::Mechanize?
93 If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is a
94 subclass of LWP::UserAgent, so all the wondrous magic of that class is
95 inherited.
96 How do I use WWW::Mechanize through a proxy server?
98 See the docs in LWP::UserAgent on how to use the proxy. Short version:
99 $mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');
101 or get the specs from the environment:
103 $mech->env_proxy();
105 # Environment set like so:
107 gopher_proxy=http://proxy.my.place/
108 wais_proxy=http://proxy.my.place/
109 no_proxy="localhost,my.domain"
110 export gopher_proxy wais_proxy no_proxy
111 How can I see what fields are on the forms?
113 Use the mech-dump utility, optionally installed with Mechanize.
114 $ mech-dump --forms http://search.cpan.org
116 Dumping forms
117 GET http://search.cpan.org/search
118 query=
119 mode=all (option) [*all|module|dist|author]
120 <NONAME>=CPAN Search (submit)
121 How do I get Mech to handle authentication?
123 use MIME::Base64;
124 my $agent = WWW::Mechanize->new();
126 my @args = (
127 Authorization => "Basic " .
128 MIME::Base64::encode( USER . ':' . PASS )
129 );
130 $agent->credentials( ADDRESS, REALM, USER, PASS );
132 $agent->get( URL, @args );
133 If you want to use the credentials for all future requests, you can
135 also use the LWP::UserAgent "default_header()" method instead of the
136 extra arguments to "get()"
137 $mech->default_header(
139 Authorization => 'Basic ' . encode_base64( USER . ':' . PASSWORD ) );
140 How can I get WWW::Mechanize to execute this JavaScript?
142 You can't. JavaScript is entirely client-based, and WWW::Mechanize is
143 a client that doesn't understand JavaScript. See the top part of this
144 FAQ.
145 How do I check a checkbox that doesn't have a value defined?
147 Set it to the value of "on".
148 $mech->field( my_checkbox => 'on' );
150 How do I handle frames?
152 You don't deal with them as frames, per se, but as links. Extract them
153 with
154 my @frame_links = $mech->find_link( tag => "frame" );
156 How do I get a list of HTTP headers and their values?
158 All HTTP::Headers methods work on a HTTP::Response object which is
159 returned by the get(), reload(), response()/res(), click(),
160 submit_form(), and request() methods.
161 my $mech = WWW::Mechanize->new( autocheck => 1 );
163 $mech->get( 'http://my.site.com' );
164 my $response = $mech->response();
165 for my $key ( $response->header_field_names() ) {
166 print $key, " : ", $response->header( $key ), "\n";
167 }
168 How do I enable keep-alive?
170 Since WWW::Mechanize is a subclass of LWP::UserAgent, you can use the
171 same mechanism to enable keep-alive:
172 use LWP::ConnCache;
174 ...
175 $mech->conn_cache(LWP::ConnCache->new);
176 How can I change/specify the action parameter of an HTML form?
178 You can access the action of the form by utilizing the HTML::Form
179 object returned from one of the specifying form methods.
180 Using "$mech->form_number($number)":
182 my $mech = WWW::mechanize->new;
184 $mech->get('http://someurlhere.com');
185 # Access the form using its Zero-Based Index by DOM order
186 $mech->form_number(0)->action('http://newAction'); #ABS URL
187 Using "$mech->form_name($number)":
189 my $mech = WWW::mechanize->new;
191 $mech->get('http://someurlhere.com');
192 #Access the form using its Zero-Based Index by DOM order
193 $mech->form_name('trgForm')->action('http://newAction'); #ABS URL
194 How do I save an image? How do I save a large tarball?
196 An image is just content. You get the image and save it.
197 $mech->get( 'photo.jpg' );
199 $mech->save_content( '/path/to/my/directory/photo.jpg' );
200 You can also save any content directly to disk using the
202 ":content_file" flag to "get()", which is part of LWP::UserAgent.
203 $mech->get( 'http://www.cpan.org/src/stable.tar.gz',
205 ':content_file' => 'stable.tar.gz' );
206 How do I pick a specific value from a "<select>" list?
208 Find the "HTML::Form::ListInput" in the page.
209 my ($listbox) = $mech->find_all_inputs( name => 'listbox' );
211 Then create a hash for the lookup:
213 my %name_lookup;
215 @name_lookup{ $listbox->value_names } = $listbox->possible_values;
216 my $value = $name_lookup{ 'Name I want' };
217 If you have duplicate names, this method won't work, and you'll have to
219 loop over "$listbox->value_names" and "$listbox->possible_values" in
220 parallel until you find a matching name.
221 How do I get Mech to not follow redirects?
223 You use functionality in LWP::UserAgent, not Mech itself.
224 $mech->requests_redirectable( [] );
226 Or you can set "max_redirect":
228 $mech->max_redirect( 0 );
230 Both these options can also be set in the constructor. Mech doesn't
232 understand them, so will pass them through to the LWP::UserAgent
233 constructor.
234
236 My Mech program doesn't work, but it works in the browser.
237 Mechanize acts like a browser, but apparently something you're doing is
238 not matching the browser's behavior. Maybe it's expecting a certain
239 web client, or maybe you've not handling a field properly. For some
240 reason, your Mech problem isn't doing exactly what the browser is
241 doing, and when you find that, you'll have the answer.
242 My Mech program gets these 500 errors.
244 A 500 error from the web server says that the program on the server
245 side died. Probably the web server program was expecting certain
246 inputs that you didn't supply, and instead of handling it nicely, the
247 program died.
248 Whatever the cause of the 500 error, if it works in the browser, but
250 not in your Mech program, you're not acting like the browser. See the
251 previous question.
252 Why doesn't my program handle this form correctly?
254 Run mech-dump on your page and see what it says.
255 mech-dump is a marvelous diagnostic tool for figuring out what forms
257 and fields are on the page. Say you're scraping CNN.com, you'd get
258 this:
259 $ mech-dump http://www.cnn.com/
261 GET http://search.cnn.com/cnn/search
262 source=cnn (hidden readonly)
263 invocationType=search/top (hidden readonly)
264 sites=web (radio) [*web/The Web ??|cnn/CNN.com ??]
265 query= (text)
266 <NONAME>=Search (submit)
267 POST http://cgi.money.cnn.com/servlets/quote_redirect
269 query= (text)
270 <NONAME>=GET (submit)
271 POST http://polls.cnn.com/poll
273 poll_id=2112 (hidden readonly)
274 question_1=<UNDEF> (radio) [1/Simplistic option|2/VIEW RESULTS]
275 <NONAME>=VOTE (submit)
276 GET http://search.cnn.com/cnn/search
278 source=cnn (hidden readonly)
279 invocationType=search/bottom (hidden readonly)
280 sites=web (radio) [*web/??CNN.com|cnn/??]
281 query= (text)
282 <NONAME>=Search (submit)
283 Four forms, including the first one duplicated at the end. All the
285 fields, all their defaults, lovingly generated by HTML::Form's "dump"
286 method.
287 If you want to run mech-dump on something that doesn't lend itself to a
289 quick URL fetch, then use the "save_content()" method to write the HTML
290 to a file, and run mech-dump on the file.
291 Why don't https:// URLs work?
293 You need either IO::Socket::SSL or Crypt::SSLeay installed.
294 Why do I get "Input 'fieldname' is readonly"?
296 You're trying to change the value of a hidden field and you have
297 warnings on.
298 First, make sure that you actually mean to change the field that you're
300 changing, and that you don't have a typo. Usually, hidden variables
301 are set by the site you're working on for a reason. If you change the
302 value, you might be breaking some functionality by faking it out.
303 If you really do want to change a hidden value, make the changes in a
305 scope that has warnings turned off:
306 {
308 local $^W = 0;
309 $agent->field( name => $value );
310 }
311 I tried to [such-and-such] and I got this weird error.
313 Are you checking your errors?
314 Are you sure?
316 Are you checking that your action succeeded after every action?
318 Are you sure?
320 For example, if you try this:
322 $mech->get( "http://my.site.com" );
324 $mech->follow_link( "foo" );
325 and the "get" call fails for some reason, then the Mech internals will
327 be unusable for the "follow_link" and you'll get a weird error. You
328 must, after every action that GETs or POSTs a page, check that Mech
329 succeeded, or all bets are off.
330 $mech->get( "http://my.site.com" );
332 die "Can't even get the home page: ", $mech->response->status_line
333 unless $mech->success;
334 $mech->follow_link( "foo" );
336 die "Foo link failed: ", $mech->response->status_line
337 unless $mech->success;
338 How do I figure out why "$mech->get($url)" doesn't work?
340 There are many reasons why a "get()" can fail. The server can take you
341 to someplace you didn't expect. It can generate redirects which are not
342 properly handled. You can get time-outs. Servers are down more often
343 than you think! etc, etc, etc. A couple of places to start:
344 1 Check "$mech->status()" after each call
346 2 Check the URL with "$mech->uri()" to see where you ended up
347 3 Try debugging with "LWP::ConsoleLogger".
348 If things are really strange, turn on debugging with "use
350 LWP::ConsoleLogger::Everywhere;" Just put this in the main program.
351 This causes LWP to print out a trace of the HTTP traffic between client
352 and server and can be used to figure out what is happening at the
353 protocol level.
354 It is also useful to set many traps to verify that processing is
356 proceeding as expected. A Mech program should always have an "I didn't
357 expect to get here" or "I don't recognize the page that I am
358 processing" case and bail out.
359 Since errors can be transient, by the time you notice that the error
361 has occurred, it might not be possible to reproduce it manually. So for
362 automated processing it is useful to email yourself the following
363 information:
364 · where processing is taking place
366 · An Error Message
368 · $mech->uri
370 · $mech->content
372 You can also save the content of the page with "$mech->save_content(
374 'filename.html' );"
375 I submitted a form, but the server ignored everything! I got an empty form
377 back!
378 The post is handled by application software. It is common for PHP
379 programmers to use the same file both to display a form and to process
380 the arguments returned. So the first task of the application programmer
381 is to decide whether there are arguments to processes. The program can
382 check whether a particular parameter has been set, whether a hidden
383 parameter has been set, or whether the submit button has been clicked.
384 (There are probably other ways that I haven't thought of).
385 In any case, if your form is not setting the parameter (e.g. the submit
387 button) which the web application is keying on (and as an outsider
388 there is no way to know what it is keying on), it will not notice that
389 the form has been submitted. Try using "$mech->click()" instead of
390 "$mech->submit()" or vice-versa.
391 I've logged in to the server, but I get 500 errors when I try to get to
393 protected content.
394 Some web sites use distributed databases for their processing. It can
395 take a few seconds for the login/session information to percolate
396 through to all the servers. For human users with their slow reaction
397 times, this is not a problem, but a Perl script can outrun the server.
398 So try adding a sleep(5) between logging in and actually doing anything
399 (the optimal delay must be determined experimentally).
400 Mech is a big memory pig! I'm running out of RAM!
402 Mech keeps a history of every page, and the state it was in. It
403 actually keeps a clone of the full Mech object at every step along the
404 way.
405 You can limit this stack size with the "stack_depth" param in the
407 "new()" constructor. If you set stack_size to 0, Mech will not keep
408 any history.
409
411 Andy Lester <andy at petdance.com>
412
414 This software is copyright (c) 2004 by Andy Lester.
415 This is free software; you can redistribute it and/or modify it under
417 the same terms as the Perl 5 programming language system itself.
418perl v5.32.0 2020-07-28 WWW::Mechanize::FAQ(3)