1AUTOSEARCH(1)         User Contributed Perl Documentation        AUTOSEARCH(1)
2
3
4

NAME

6       AutoSearch -- a web-search tracking application
7

SYNOPSIS

9       AutoSearch [--stats] [--verbose] -n "Query Name" -s "query string"
10       --engine engine [--mail you@where.com] [--options "opt=val"]...
11       [--filter "filter"] [--host host] [--port port] [--userid bbunny
12       --password c4rr0t5] [--ignore_channels KABC,KCBS,KNBC] qid
13
14       AutoSearch --VERSION AutoSearch --help AutoSearch --man
15

DESCRIPTION

17       AutoSearch performs a web-based search and puts the results set in
18       qid/index.html.  Subsequent searches (i.e., the second form above)
19       AutoSearch determine what changes (if any) occured to the results sent
20       since the last run.  These incremental changes are recorded in
21       qid/YYYYMMDD.html.
22
23       AutoSearch is amenable to be run as a cron job because all the input
24       parameters are saved in the web pages.  AutoSearch can act as a
25       automated query agent for a particular search.  The output files are
26       designed to be a set of web pages to easily display the results set
27       with a web browser.
28
29       Example:
30
31           AutoSearch -n 'LSAM Replication'
32               -s '"lsam replication"'
33               -e AltaVista
34               replication_query
35
36       This query (which should be all on one line) creates a directory
37       replication_query and fills it with the fascinating output of the
38       AltaVista query on "lsam replication", with pages titled ``LSAM
39       Replication''.  (Note the quoting:  the single quotes in '"lsam
40       replication"' are for the shell, the double quotes are for AltaVista to
41       search for the phrase rather than the separate words.)
42
43       A more complicated example:
44
45           AutoSearch -n 'External Links to LSAM'
46               -s '(link:www.isi.edu/lsam or link:www.isi.edu/~lsam) -url:isi.edu'
47               -e AltaVista::AdvancedWeb
48               -o coolness=hot
49
50       This query does an advanced AltaVista search and specifies the
51       (hypothetical) ``coolness'' option to the search engine.
52

OPTIONS

54       "qid"
55           The query identifer specifies the directory in which all the files
56           that relate to this query and search results will live.  It can be
57           an absolute path, or a relative path from cwd.  If the directory
58           does not exist, it will be created and a new search started.
59
60       "--stats"
61           Show search statistics: the query string, number of hits, number of
62           filtered hits, filter string, number of suspended (deleted) hits,
63           previous set size, current set size, etc.
64
65       "-v" or "--verbose"
66           Verbose: output additional messages and warnings.
67
68       "-n" or "--qn" or "--queryname"
69           Specify the query name.  The query name is used as a heading in the
70           web pages, therefore it should be a 'nice' looking version of the
71           query string.
72
73       "-s" or "--qs" or "--querystring"
74           Specify the query string.  The query string is the character string
75           which will be submitted to the search engine.  You may include
76           special characters to group or to qualify the search.
77
78       "-e" or "--engine"
79           Specify the search engine.  The query string will be submitted to
80           the user specified search engine.
81
82           In many cases there are specialized versions of search engines.
83           For example, AltaVista::AdvancedWeb and AltaVista::News allow more
84           powerful and Usenet searches.  See AltaVista or the man page for
85           your search engine for details about specialized variations.
86
87       "--listnewurls"
88           In addition to all the normal file maintenance, print all new URLs
89           to STDOUT, one per line.
90
91       "-o" or "--options"
92           Specify the query options.  The query options will be submitted to
93           the user search engine with the query string.  This feature permits
94           modification of the query string for a specific search engine or
95           option.  More than one query option may be specified.
96
97           Example: "-o what=news" causes AltaVista to search Usenet.
98           Although this works, the preferred mechanism in this case would be
99           "-e AltaVista::News" or "-e AltaVista::AdvancedNews".  Options are
100           intended for internal or expert use.
101
102       "-f" or "--uf" or "--urlfilter"
103           This option specifies a regular expression which will be compared
104           against the URLs of any results; if they match the case-insensitive
105           regular expression, they will be removed from the hit set.
106
107           Example: "-f '.*\.isi\.edu'" avoids all of ISI's web pages.
108
109       "--cleanup i"
110           Delete all traces of query results from more than i days ago.  If
111           --cleanup is given, all other options other than the qid will be
112           ignored.
113
114       "--cmdline"
115           Reconstruct the complete command line (AutoSearch and all its
116           arguments) that was used to create the query results.  Command line
117           will be shown on STDERR.  If --cmdline is given, all other options
118           other than the qid will be ignored.
119
120       "--mail user@address" or "-m user@address"
121           After search is complete, send email to that user, listing the NEW
122           results.  Email is HTML format.  Requires the Email::Send and
123           related modules.  If you send email through an SMTP server, you
124           must set environment variable SMTPSERVER to your server name or IP
125           address.  If your SMTP server requires password, you must set
126           environment variables SMTPUSERNAME and SMTPPASSWORD.  If you send
127           email via sendmail, you should set environment variable SENDMAIL if
128           the sendmail executable is not in the path.
129
130       "--emailfrom user@address"
131           If your outgoing mail server rejects email from certain users, you
132           can use this argument to set the From: header.
133
134       "--userid bbunny"
135           If the search engine requires a login/password (e.g.
136           Ebay::Completed), use this.
137
138       "--password Carr0t5"
139           If the search engine requires a login/password (e.g. Ebay::Mature),
140           use this.
141

DESCRIPTION

143       AutoSearch submits a query to a search engine, produces HTML pages that
144       reflect the set of 'hits' (filtered search results) returned by the
145       search engine, and tracks these results over time.  The URL and title
146       are displayed in the qid/index.html, the URL, the title, and
147       description are displayed in the 'weekly' files.
148
149       To organize these results, each search result is placed in a query
150       information directory (qid).  The directory becomes the search results
151       'handle', an easy way to track a set of results.  Thus a qid of
152       "/usr/local/htdocs/lsam/autosearch/load_balancing" might locate the
153       results on your web server at
154       "http://www.isi.edu/lsam/autosearch/load_balancing".
155
156       Inside the qid directory you will find files relating to this query.
157       The primary file is index.html, which reflects the latest search
158       results.  Every not-filtered hit for every search is stored in
159       index.html.  When a hit is no longer found by the search engine it a
160       removed from index.html.  As new results for a search are returned from
161       the search engine they are placed in index.html.
162
163       At the bottom of index.html, there is a heading "Weekly Search
164       Results", which is updated each time the search is submitted (see
165       "AUTOMATED SEARCHING").  The list of search runs is stored in reverse
166       chronological order.  Runs which provide no new information are
167       identified with
168
169               No Unique Results found for search on <date>
170
171       Runs which contain changes are identified by
172
173               Web search results for search on <date>
174
175       which will be linked a page detailing the changes from that run.
176
177       Detailed search results are noted in weekly files.  These files are
178       named YYYYMMDD.html and are stored in the qid directory.  The weekly
179       files include THE URL, title, and a the description (if available).
180       The title is a link to the original web page.
181

AUTOMATED SEARCHING

183       On UNIX-like systems, cron(1) may be used to establish periodic
184       searches and the web pages will be maintained by AutoSearch.  To
185       establish the first search, use the first example under SYNOPSIS.  You
186       must specify the qid, query name and query string.  If any of the items
187       are missing, you will be interactively prompted for the missing
188       item(s).
189
190       Once the first search is complete you can re-run the search with the
191       second form under SYNOPSIS.
192
193       A cron entry like:
194
195           0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching
196
197       might be used to run the search each Monday at 3:00 AM.  The query name
198       and query string may be repeated; but they will not be used.  This
199       means that with a cron line like:
200
201           0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching -n caching -s caching
202
203       a whole new search series can be originated by
204
205           rm -r /www/div7/lsam/autosearch/caching
206
207       However, the only reason to start a new search series would be to throw
208       away the old weekly files.
209
210       We don't recommend running searches more than once per day, but if so
211       the per-run files will be updated in-place.  Any changes are added to
212       the page with a comment that "Recently Added:"; and deletions are
213       indicated with "Recently Suspended:."
214

CHANGING THE LOOK OF THE PAGES

216       The basic format of these two pages is simple and customizable.  One
217       requirement is that the basic structure remain unchanged.  HTML
218       comments are used to identify sections of the document.  Almost
219       everything can be changed except for the strings which identify the
220       section starts and ends.
221
222       Noteworthy tags and their meaning:
223
224       <!--Top-->.*<!--/Top-->
225                       The text contained within this tag is placed at the top
226                       of the output page.  If the text contains AutoSearch
227                       WEB Searching, then the query name will replace it.  If
228                       the text does not contain this magic string and it is
229                       the first ever search, the user will be asked for a
230                       query name.
231
232       <!--Query{.*}/Query-->
233                       The text contained between the braces is the query
234                       string.  This is how AutoSearch maintains the query
235                       string.  You may edit this string to change the query
236                       string; but only in qid/index.html.  The text ask user
237                       is special and will force AutoSearch to request the
238                       search string from the user.
239
240       <!--SearchEngine{.*}/SearchEngine-->
241                       The text contained between the braces is the search
242                       engine.  Other engines supported are HotBot and Lycos.
243                       You may edit this string to change the engine used; but
244                       only in qid/index.html.  The text ask user is special
245                       and will force AutoSearch to to request the search
246                       string from the user.
247
248       <!--QueryOptions{.*}/QueryOptions-->
249                       The text contained between the braces specifies a query
250                       options.  Multiple occurrencs of this command are
251                       allowed to specify multiple options.
252
253       <!--URLFilter{.*}/URLFilter-->
254                       The text contained between the braces is the URL
255                       filter.  This is how AutoSearch maintains the filter.
256                       Again you may edit this string to change the query
257                       string; but only in qid/index.html.  The text ask user
258                       is special and will force AutoSearch to ask the user
259                       (STDIN) for the query string.  When setting up the
260                       first search, you must edit first_index.html, not
261                       qid/index.html.  The URL filter is a standard perl5
262                       regular expression.  URLs which do not match will be
263                       kept.
264
265       <!--Bottom-->.*<!--/Bottom-->
266                       The text contained within this tag is placed at the
267                       bottom of the output page.  This is a good place to put
268                       navigation, page owner information, etc.
269
270       The remainder of the tags fall into a triplet of ~Heading, ~Template,
271       and ~, where ~ is Summary, Weekly, Appended, and Suspended. The sub-
272       sections appear in the order given.  To produce a section AutoSearch
273       outputs the heading, the template, the section, n copies of the
274       formatted data, and an /section.  The tags and their function are:
275
276       ~Heading        The heading tag identifies the heading for a section of
277                       the output file.  The SummaryHeading is for the summary
278                       portion, etc.  The section may be empty (e.g.,
279                       Suspended) and thus no heading is output.
280
281       ~Template       The template tag identifies how each item is to be
282                       formatted.  Simple text replacement is used to change
283                       the template into the actual output text.  The text to
284                       be replaced is noted in ALLCAPS.
285
286       ~               This tag is used to locate the section (Summary,
287                       Weekly, etc.).  This section represents the actual
288                       n-items of data.
289
290       You can edit these values in the qid/index.html page of an existing
291       search.  The file first_index.html (in the directory above qid) will be
292       used as a default template for new queries.
293
294       Examples of these files can be seen in the pages under
295       "http://www.isi.edu/lsam/tools/autosearch/", or in the output generated
296       by a new AutoSearch.
297

FILES

299       first_index.html    optional file to determine the default format of
300                           the index.html file of a new query.
301
302       first_date.html     optional file to determine the default format of
303                           the YYYYMMDD.html file for a new query.
304
305       qid/index.html      (automatically created) latest search results, and
306                           reverse chronological list of periodic searches.
307
308       qid/date.html       file used as a template for the YYYYMMDD.html
309                           files.
310
311       qid/YYYYMMDD.html   (automatically created) summary of changes for a
312                           particular date (AKA 'Weekly' file).
313
314       Optional files first_index.html and first_date.html are used for the
315       initial search as a template for qid/index.html and date.html,
316       respectively.  If either of these files does not exist; a default-
317       default template is stored within the AutoSearch source.  The intention
318       of these two files is to permit a user to establish a framework for a
319       group of search sets which have a common format.  By leaving the
320       default query name and query string alone, they will be overridden by
321       command line inputs.
322

SEE ALSO

324       For the library, see WWW::Search, for the perl regular expressions, see
325       perlre.
326

AUTHORS

328       Wm. L. Scheding
329
330       AutoSearch is a re-implementation of an earlier version written by
331       Kedar Jog.
332
334       Copyright (C) 1996-1997 University of Southern California.  All rights
335       reserved.
336
337       Redistribution and use in source and binary forms are permitted
338       provided that the above copyright notice and this paragraph are
339       duplicated in all such forms and that any documentation, advertising
340       materials, and other materials related to such distribution and use
341       acknowledge that the software was developed by the University of
342       Southern California, Information Sciences Institute.  The name of the
343       University may not be used to endorse or promote products derived from
344       this software without specific prior written permission.
345
346       THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
347       WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
348       MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
349

DESIRED FEATURES

351       These are good ideas that people have suggested.
352
353       URL validation.
354           Validate the status of each URL (with HTTP HEAD requests) and
355           indicate this status in the output.
356
357       Multi-search.
358           It should be possible to merge the results of searches from two
359           search-engines.  If this merger were done as a new search engine,
360           this operation would be transparent to AutoSearch.
361

BUGS

363       None known at this time; please inform the maintainer mthurn@cpan.org
364       if any crop up.
365
366
367
368perl v5.12.0                      2008-07-14                     AUTOSEARCH(1)
Impressum