1WWW::Search(3) User Contributed Perl Documentation WWW::Search(3)
2
6 WWW::Search - Virtual base class for WWW searches
7
9 use WWW::Search;
10 my $sEngine = 'AltaVista';
11 my $oSearch = new WWW::Search($sEngine);
12
14 This class is the parent for all access methods supported by the
15 "WWW::Search" library. This library implements a Perl API to web-based
16 search engines.
17 See README for a list of search engines currently supported, and for a
19 lot of interesting high-level information about this distribution.
20 Search results can be limited, and there is a pause between each
22 request to avoid overloading either the client or the server.
23 Sample program
25 Here is a sample program:
26 my $sQuery = 'Columbus Ohio sushi restaurant';
28 my $oSearch = new WWW::Search('AltaVista');
29 $oSearch->native_query(WWW::Search::escape_query($sQuery));
30 $oSearch->login($sUser, $sPassword);
31 while (my $oResult = $oSearch->next_result())
32 {
33 print $oResult->url, "\n";
34 } # while
35 $oSearch->logout;
36 Results are objects of type "WWW::SearchResult" (see WWW::SearchResult
38 for details). Note that different backends support different result
39 fields. All backends are required to support title and url.
40
42 For specific search engines, see WWW::Search::TheEngineName (replacing
43 TheEngineName with a particular search engine).
44 For details about the results of a search, see WWW::SearchResult.
46
48 new To create a new WWW::Search, call
49 $oSearch = new WWW::Search('SearchEngineName');
51 where SearchEngineName is replaced with a particular search engine.
53 For example:
54 $oSearch = new WWW::Search('Yahoo');
56 If no search engine is specified, a default (currently
58 'Null::Empty') will be chosen for you.
59 version
61 Returns the value of the $VERSION variable of the backend engine,
62 or $WWW::Search::VERSION if the backend does not contain $VERSION.
63 maintainer
65 Returns the value of the $MAINTAINER variable of the backend
66 engine, or $WWW::Search::MAINTAINER if the backend does not contain
67 $MAINTAINER.
68 installed_engines
70 Returns a list of the names of all installed backends. We can not
71 tell if they are up-to-date or working, though.
72 use WWW::Search;
74 my @asEngines = sort &WWW::Search::installed_engines();
75 local $" = ', ';
76 print (" + These WWW::Search backends are installed: @asEngines\n");
77 # Choose a backend at random (yes, this is rather silly):
78 my $oSearch = WWW::Search->new($asEngines[rand(scalar(@asEngines))]);
79 native_query
81 Specify a query (and optional options) to the current search
82 object. Previous query (if any) and its cached results (if any)
83 will be thrown away. The option values and the query must be
84 escaped; call WWW::Search::escape_query() to escape a string. The
85 search process is not actually begun until "results()" or
86 "next_result()" is called (lazy!), so native_query does not return
87 anything.
88 Example:
90 $oSearch->native_query('search-engine-specific+escaped+query+string',
92 { option1 => 'able', option2 => 'baker' } );
93 The hash of options following the query string is optional. The
95 query string is backend-specific. There are two kinds of options:
96 options specific to the backend, and generic options applicable to
97 multiple backends.
98 Generic options all begin with 'search_'. Currently a few are
100 supported:
101 search_url
103 Specifies the base URL for the search engine.
104 search_debug
106 Enables backend debugging. The default is 0 (no debugging).
107 search_parse_debug
109 Enables backend parser debugging. The default is 0 (no
110 debugging).
111 search_to_file FILE
113 Causes the search results to be saved in a set of files
114 prefixed by FILE. (Used internally by the test-suite, not
115 intended for general use.)
116 search_from_file FILE
118 Reads a search from a set of files prefixed by FILE. (Used
119 internally by the test-suite, not intended for general use.)
120 Some backends may not implement these generic options, but any
122 which do implement them must provide these semantics.
123 Backend-specific options are described in the documentation for
125 each backend. In most cases the options and their values are
126 packed together to create the query portion of the final URL.
127 Details about how the search string and option hash are interpreted
129 might be found in the search-engine-specific manual pages
130 (WWW::Search::SearchEngineName).
131 gui_query
133 Specify a query to the current search object; the query will be
134 performed with the engine's default options, as if it were typed by
135 a user in a browser window.
136 Same arguments as "native_query()" above.
138 Currently, this feature is supported by only a few backends;
140 consult the documentation for each backend to see if it is
141 implemented.
142 cookie_jar
144 Call this method (anytime before asking for results) if you want to
145 communicate cookie data with the search engine. Takes one
146 argument, either a filename or an HTTP::Cookies object. If you
147 give a filename, WWW::Search will attempt to read/store cookies
148 there (by passing the filename to HTTP::Cookies::new).
149 $oSearch->cookie_jar('/tmp/my_cookies');
151 If you give an HTTP::Cookies object, it is up to you to save the
153 cookies if/when you wish.
154 use HTTP::Cookies;
156 my $oJar = HTTP::Cookies->new(...);
157 $oSearch->cookie_jar($oJar);
158 If you pass in no arguments, the cookie jar (if any) is returned.
160 my $oJar = $oSearch->cookie_jar;
162 unless (ref $oJar) { print "No jar" };
163 date_from
165 Set/get the start date for limiting the query by a date range. See
166 the documentation for each backend to find out if date ranges are
167 supported.
168 date_to
170 Set/get the end date for limiting the query by a date range. See
171 the documentation for each backend to find out if date ranges are
172 supported.
173 env_proxy
175 Enable loading proxy settings from environment variables. The
176 proxy URL will be read from $ENV{http_proxy}. The username for
177 authentication will be read from $ENV{http_proxy_user}. The
178 password for authentication will be read from $ENV{http_proxy_pwd}.
179 If you don't want to put passwords in the environment, one solution
181 would be to subclass LWP::UserAgent and use
182 $ENV{WWW_SEARCH_USERAGENT} instead (see user_agent below).
183 env_proxy() must be called before the first retrieval is attempted.
185 Example:
187 $ENV{http_proxy } = 'http://my.proxy.com:80';
189 $ENV{http_proxy_user} = 'bugsbun';
190 $ENV{http_proxy_pwd } = 'c4rr0t5';
191 $oSearch->env_proxy('yes'); # Turn on with any true value
192 ...
193 $oSearch->env_proxy(0); # Turn off with zero
194 ...
195 if ($oSearch->env_proxy) # Test
196 http_proxy
198 Set up an HTTP proxy (for connections from behind a firewall).
199 Takes the same arguments as LWP::UserAgent::proxy().
201 This routine should be called before calling any of the result
203 functions (any method with "result" in its name).
204 Example:
206 # Turn on and set address:
208 $oSearch->http_proxy(['http','ftp'] => 'http://proxy:8080');
209 # Turn off:
210 $oSearch->http_proxy('');
211 http_proxy_user, http_proxy_pwd
213 Set/get HTTP proxy authentication data.
214 These routines set/get username and password used in proxy
216 authentication. Authentication is attempted only if all three
217 items (proxy URL, username and password) have been set.
218 Example:
220 $oSearch->http_proxy_user("myuser");
222 $oSearch->http_proxy_pwd("mypassword");
223 $oSearch->http_proxy_user(undef); # Example for no authentication
224 $username = $oSearch->http_proxy_user();
226 maximum_to_retrieve
228 Set the maximum number of hits to return. Queries resulting in
229 more than this many hits will return the first hits, up to this
230 limit. Although this specifies a maximum limit, search engines
231 might return less than this number.
232 Defaults to 500.
234 Example:
236 $max = $oSearch->maximum_to_retrieve(100);
237 You can also spell this method "maximum_to_return".
239 maximum_to_return
241 Synonym for maximum_to_retrieve
242 timeout
244 The maximum length of time any portion of the query should take, in
245 seconds.
246 Defaults to 60.
248 Example:
250 $oSearch->timeout(120);
251 login
253 Backends which need to login to the search engine should implement
254 this function. Takes two arguments, user and password. Return
255 nonzero if login was successful. Return undef or 0 if login
256 failed.
257 logout
259 Backends which need to logout from the search engine should
260 implement this function.
261 approximate_result_count
263 Some backends indicate how many results they have found, e.g. with
264 a number shown at the top of the search results page. Note that
265 there is no corresponding method that returns the actual count of
266 results; that's because results are normally retrieved in batches
267 (i.e. pages) and at any given time there's no way to know how big
268 the final list of results will be. NEW: if request has not been
269 made to the search provider, this method will return undef (used to
270 return zero). NEW: if the results page does not explicitly
271 indicate the result count, this method will return undef (used to
272 return zero).
273 approximate_hit_count
275 This is an alias for approximate_result_count().
276 results
278 Return all the results of a query as an array of WWW::SearchResult
279 objects.
280 Note: This might take a while, because a web backend will keep
282 asking the search engine for "next page of results" over and over
283 until there are no more next pages, and THEN return from this
284 function.
285 If an error occurs at any time during query processing, it will be
287 indicated in the response().
288 Example:
290 @results = $oSearch->results();
292 # Go have a cup of coffee while the previous line executes...
293 foreach $oResult (@results)
294 {
295 print $oResult->url(), "\n";
296 } # foreach
297 next_result
299 Call this method repeatedly to return each result of a query as a
300 WWW::SearchResult object. Example:
301 while ($oResult = $oSearch->next_result())
303 {
304 print $oResult->url(), "\n";
305 } # while
306 When there are no more results, or if an error occurs,
308 next_result() will return undef.
309 If an error occurs at any time during query processing, it will be
311 indicated in the response().
312 seek_result($offset)
314 Set which result should be returned next time "next_result()" is
315 called. Results are zero-indexed.
316 The only guaranteed valid offset is 0, which will replay the
318 results from the beginning. In particular, seeking past the end of
319 the current cached results probably will not do what you might
320 think it should.
321 Results are cached, so this does not re-issue the query or cause IO
323 (unless you go off the end of the results). To re-do the query,
324 create a new search object.
325 Example:
327 $oSearch->seek_result(0);
329 response
331 Returns an HTTP::Response object which resulted from the most-
332 recently-sent query. Errors can be detected like this:
333 if (! $oSearch->response->is_success)
335 {
336 print STDERR "Error: " . $oSearch->response->as_string() . "\n";
337 } # if
338 Note to backend authors: even if the backend does not involve the
340 web, it should return an HTTP::Response object.
341 submit
343 This method can be used to submit URLs to the search engines for
344 indexing. Consult the documentation for each backend to find out
345 if it is implemented there, and if so what the arguments are.
346 Returns an HTTP::Response object describing the result of the
348 submission request. Consult the documentation for each backend to
349 find out the meaning of the response.
350 opaque
352 This function provides an application a place to store one opaque
353 data element (or many, via a Perl reference). This facility is
354 useful to (for example), maintain client-specific information in
355 each active query when you have multiple concurrent queries.
356 escape_query
358 Escape a query. Before queries are sent to the internet, special
359 characters must be escaped so that a proper URL can be formed.
360 This is like escaping a URL, but all non-alphanumeric characters
361 are escaped and and spaces are converted to "+"s.
362 Example:
364 $escaped = WWW::Search::escape_query('+hi +mom');
366 # $escaped is now '%2Bhi+%2Bmom'
367 See also "unescape_query()". NOTE that this is not a method, it is
369 a plain function.
370 unescape_query
372 Unescape a query. See "escape_query()" for details.
373 Example:
375 $unescaped = WWW::Search::unescape_query('%22hi+mom%22');
377 # $unescaped eq q{"hi mom"}
378 NOTE that this is not a method, it is a plain function.
380 strip_tags
382 Given a string, returns a copy of that string with HTML tags
383 removed. This should be used by each backend as they insert the
384 title and description values into the search results objects.
385 NOTE that this is not a method, it is a plain function.
387 is_http_proxy
389 Returns true if proxy information is available.
390
392 reset_search
393 Resets internal data structures to start over with a new search (on
394 the same engine).
395 is_http_proxy_auth_data
397 Returns true if all authentication data (proxy URL, username, and
398 password) are available.
399 agent_name($sName)
401 If your search engine rejects certain browser, you can trick it
402 into thinking you're any browser type you want. See below under
403 user_agent().
404 agent_email($sName)
406 user_agent($NON_ROBOT)
407 This internal routine creates a user-agent for derived classes that
408 query the web. If any non-false argument is given, a normal
409 LWP::UserAgent (rather than a LWP::RobotUA) is used.
410 Returns the user-agent object.
412 If a backend needs the low-level LWP::UserAgent or LWP::RobotUA to
414 have a particular name, $oSearch->agent_name() and possibly
415 $oSearch->agent_email() should be called to set the desired values
416 *before* calling $oSearch->user_agent().
417 If the environment variable WWW_SEARCH_USERAGENT has a value, it
419 will be used as the class for a new user agent object. This class
420 should be a subclass of LWP::UserAgent. For example,
421 $ENV{WWW_SEARCH_USERAGENT} = 'My::Own::UserAgent';
423 # If this env.var. has no value,
424 # LWP::UserAgent or LWP::RobotUA will be used.
425 $oSearch = new WWW::Search('MyBackend');
426 $oSearch->agent_name('MySpider');
427 if ($iBackendWebsiteRequiresNonRobot)
428 {
429 $oSearch->user_agent('non-robot');
430 }
431 else
432 {
433 $oSearch->agent_email('me@here.com');
434 $oSearch->user_agent();
435 }
436 Backends should use robot-style user-agents whenever possible.
438 http_referer
440 Get / set the value of the HTTP_REFERER variable for this search
441 object. Some search engines might only accept requests that
442 originated at some specific previous page. This method lets
443 backend authors "fake" the previous page. Call this method before
444 calling http_request.
445 $oSearch->http_referer('http://prev.engine.com/wherever/setup.html');
447 $oResponse = $oSearch->http_request('GET', $url);
448 http_method
450 Get / set the method to be used for the HTTP request. Must be
451 either 'GET' or 'POST'. Call this method before calling
452 http_request. (Normally you would set this during
453 _native_setup_search().) The default is 'GET'.
454 $oSearch->http_method('POST');
456 http_request($method, $url)
458 Submit the HTTP request to the world, and return the response.
459 Similar to LWP::UserAgent::request. Handles cookies, follows
460 redirects, etc. Requires that http_referer already be set up, if
461 needed.
462 next_url
464 Get or set the URL for the next backend request. This can be used
465 to save the WWW::Search state between sessions (e.g. if you are
466 showing pages of results to the user in a web browser). Before
467 closing down a session, save the value of next_url:
468 ...
470 $oSearch->maximum_to_return(10);
471 while ($oSearch->next_result) { ... }
472 my $urlSave = $oSearch->next_url;
473 Then, when you start up the next session (e.g. after the user
475 clicks your "next" button), restore this value before calling for
476 the results:
477 $oSearch->native_query(...);
479 $oSearch->next_url($urlSave);
480 $oSearch->maximum_to_return(20);
481 while ($oSearch->next_result) { ... }
482 WARNING: It is entirely up to you to keep your interface in sync
484 with the number of hits per page being returned from the backend.
485 And, we make no guarantees whether this method will work for any
486 given backend. (Their caching scheme might not enable you to jump
487 into the middle of a list of search results, for example.)
488 split_lines
490 This internal routine splits data (typically the result of the web
491 page retrieval) into lines in a way that is OS independent. If the
492 first argument is a reference to an array, that array is taken to
493 be a list of possible delimiters for this split. For example,
494 Yahoo.pm uses <p> and <dd><li> as "line" delimiters for
495 convenience.
496 generic_option
498 This internal routine checks if an option is generic or backend
499 specific. Currently all generic options begin with 'search_'.
500 This routine is not a method.
501 _native_setup_search
503 Do some backend-specific initialization. It will be called with
504 the same arguments as native_query().
505 setup_search
507 This internal routine does generic Search setup. It calls
508 "_native_setup_search()" to do backend-specific setup.
509 need_to_delay
511 A backend should override this method in order to dictate whether
512 user_agent_delay() needs to be called before the next HTTP request
513 is sent. Return any perlish true or zero value.
514 user_agent_delay
516 According to what need_to_delay() returns, user_agent_delay() will
517 be called between requests to remote servers to avoid overloading
518 them with many back-to-back requests.
519 absurl
521 An internal routine to convert a relative URL into a absolute URL.
522 It takes two arguments, the 'base' url (usually the search engine
523 CGI URL) and the URL to be converted. Returns a URI object.
524 retrieve_some
526 An internal routine to interface with "_native_retrieve_some()".
527 Checks for overflow.
528 _native_retrieve_some
530 Fetch the next page of results from the web engine, parse the
531 results, and prepare for the next page of results.
532 If a backend defines this method, it is in total control of the WWW
534 fetch, parsing, and preparing for the next page of results. See
535 the WWW::Search::AltaVista module for example usage of the
536 _native_retrieve_some method.
537 An easier way to achieve this in a backend is to inherit
539 _native_retrieve_some from WWW::Search, and do only the HTML
540 parsing. Simply define a method _parse_tree which takes one
541 argument, an HTML::TreeBuilder object, and returns an integer, the
542 number of results found on this page. See the WWW::Search::Yahoo
543 module for example usage of the _parse_tree method.
544 A backend should, in general, define either _parse_tree() or
546 _native_retrieve_some(), but not both.
547 Additional features of the default _native_retrieve_some method:
549 Sets $self->{_prev_url} to the URL of the page just retrieved.
551 Calls $self->preprocess_results_page() on the raw HTML of the page.
553 Then, parses the page with an HTML::TreeBuilder object and passes
555 that populated object to $self->_parse_tree().
556 Additional notes on using the _parse_tree method:
558 The built-in HTML::TreeBuilder object used to parse the page has
560 store_comments turned ON. If a backend needs to use a subclassed
561 or modified HTML::TreeBuilder object, the backend should set
562 $self->{'_treebuilder'} to that object before any results are
563 retrieved. The best place to do this is at the end of
564 _native_setup_search.
565 my $oTree = new myTreeBuilder;
567 $oTree->store_pis(1); # for example
568 $self->{'_treebuilder'} = $oTree;
569 When _parse_tree() is called, the $self->next_url is cleared.
571 During parsing, the backend should set $self->next_url to the
572 appropriate URL for the next page of results. (If _parse_tree()
573 does not set the value, the search will end after parsing this page
574 of results.)
575 When _parse_tree() is called, the URL for the page being parsed can
577 be found in $self->{_prev_url}.
578 result_as_HTML
580 Given a WWW::SearchResult object, formats it human-readable with
581 HTML.
582 preprocess_results_page
584 A filter on the raw HTML of the results page. This allows the
585 backend to alter the HTML before it is parsed, such as to correct
586 for known problems, HTML that can not be parsed correctly, etc.
587 Takes one argument, a string (the HTML webpage); returns one string
589 (the same HTML, modified).
590 This method is called from within _native_retrieve_some (above)
592 before the HTML of the page is parsed.
593 See the WWW::Search::Ebay distribution 2.07 or higher for example
595 usage.
596 test_cases (DEPRECATED)
598 Deprecated.
599 Returns the value of the $TEST_CASES variable of the backend
601 engine.
602 hash_to_cgi_string (DEPRECATED)
604 Given a reference to a hash of string => string, constructs a CGI
605 parameter string that looks like 'key1=value1&key2=value2'.
606 If the value is undef, the key will not be added to the string.
608 At one time, for testing purposes, we asked backends to use this
610 function rather than piecing the URL together by hand, to ensure
611 that URLs are identical across platforms and software versions.
612 But this is no longer necessary.
613 Example:
615 $self->{_options} = {
617 'opt3' => 'val3',
618 'search_url' => 'http://www.deja.com/dnquery.xp',
619 'opt1' => 'val1',
620 'QRY' => $native_query,
621 'opt2' => 'val2',
622 };
623 $self->{_next_url} = $self->{_options}{'search_url'} .'?'.
624 $self->hash_to_cgi_string($self->{_options});
625
627 "WWW::Search" supports backends to separate search engines. Each
628 backend is implemented as a subclass of "WWW::Search".
629 WWW::Search::Yahoo provides a good sample backend.
630 A backend must have the routine "_native_setup_search()". A backend
632 must have the routine "_native_retrieve_some()" or "_parse_tree()".
633 "_native_setup_search()" is invoked before the search. It is passed a
635 single argument: the escaped, native version of the query.
636 "_native_retrieve_some()" is the core of a backend. It will be called
638 periodically to fetch URLs. It should retrieve several hits from the
639 search service and add them to the cache. It should return the number
640 of hits found, or undef when there are no more hits.
641 Internally, "_native_retrieve_some()" typically sends an HTTP request
643 to the search service, parses the HTML, extracts the links and
644 descriptions, then saves the URL for the next page of results. See the
645 code for the "WWW::Search::AltaVista" module for an example.
646 Alternatively, a backend can define the method "_parse_tree()" instead
648 of "_native_retrieve_some()". See the "WWW::Search::Ebay" module for a
649 good example.
650 If you implement a new backend, please let the authors know.
652
654 The bugs are there for you to find (some people call them Easter Eggs).
655 Desired features:
657 A portable query language.
659 A portable language would easily allow you to move queries easily
660 between different search engines. A query abstraction is non-
661 trivial and unfortunately will not be done any time soon by the
662 current maintainer. If you want to take a shot at it, please let
663 me know.
664
666 John Heidemann <johnh@isi.edu> Maintained by Martin Thurn,
667 "mthurn@cpan.org", <http://www.sandcrawler.com/SWB/cpan-modules.html>.
668
670 Copyright (c) 1996 University of Southern California. All rights
671 reserved.
672 Redistribution and use in source and binary forms are permitted
674 provided that the above copyright notice and this paragraph are
675 duplicated in all such forms and that any documentation, advertising
676 materials, and other materials related to such distribution and use
677 acknowledge that the software was developed by the University of
678 Southern California, Information Sciences Institute. The name of the
679 University may not be used to endorse or promote products derived from
680 this software without specific prior written permission.
681 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
683 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
684 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
685perl v5.32.1 2021-01-27 WWW::Search(3)