1AUTOSEARCH(1) User Contributed Perl Documentation AUTOSEARCH(1)
2
6 AutoSearch -- a web-search tracking application
7
9 AutoSearch [--stats] [--verbose] -n "Query Name" -s "query string"
10 --engine engine [--mail you@where.com] [--options "opt=val"]...
11 [--filter "filter"] [--host host] [--port port] [--userid bbunny
12 --password c4rr0t5] [--ignore_channels KABC,KCBS,KNBC] qid
13 AutoSearch --VERSION AutoSearch --help AutoSearch --man
15
17 AutoSearch performs a web-based search and puts the results set in
18 qid/index.html. Subsequent searches (i.e., the second form above)
19 AutoSearch determine what changes (if any) occured to the results sent
20 since the last run. These incremental changes are recorded in
21 qid/YYYYMMDD.html.
22 AutoSearch is amenable to be run as a cron job because all the input
24 parameters are saved in the web pages. AutoSearch can act as a
25 automated query agent for a particular search. The output files are
26 designed to be a set of web pages to easily display the results set
27 with a web browser.
28 Example:
30 AutoSearch -n 'LSAM Replication'
32 -s '"lsam replication"'
33 -e AltaVista
34 replication_query
35 This query (which should be all on one line) creates a directory
37 replication_query and fills it with the fascinating output of the
38 AltaVista query on "lsam replication", with pages titled ``LSAM
39 Replication''. (Note the quoting: the single quotes in '"lsam
40 replication"' are for the shell, the double quotes are for AltaVista to
41 search for the phrase rather than the separate words.)
42 A more complicated example:
44 AutoSearch -n 'External Links to LSAM'
46 -s '(link:www.isi.edu/lsam or link:www.isi.edu/~lsam) -url:isi.edu'
47 -e AltaVista::AdvancedWeb
48 -o coolness=hot
49 This query does an advanced AltaVista search and specifies the
51 (hypothetical) ``coolness'' option to the search engine.
52
54 "qid"
55 The query identifer specifies the directory in which all the files
56 that relate to this query and search results will live. It can be
57 an absolute path, or a relative path from cwd. If the directory
58 does not exist, it will be created and a new search started.
59 "--stats"
61 Show search statistics: the query string, number of hits, number of
62 filtered hits, filter string, number of suspended (deleted) hits,
63 previous set size, current set size, etc.
64 "-v" or "--verbose"
66 Verbose: output additional messages and warnings.
67 "-n" or "--qn" or "--queryname"
69 Specify the query name. The query name is used as a heading in the
70 web pages, therefore it should be a 'nice' looking version of the
71 query string.
72 "-s" or "--qs" or "--querystring"
74 Specify the query string. The query string is the character string
75 which will be submitted to the search engine. You may include
76 special characters to group or to qualify the search.
77 "-e" or "--engine"
79 Specify the search engine. The query string will be submitted to
80 the user specified search engine.
81 In many cases there are specialized versions of search engines.
83 For example, AltaVista::AdvancedWeb and AltaVista::News allow more
84 powerful and Usenet searches. See AltaVista or the man page for
85 your search engine for details about specialized variations.
86 "--listnewurls"
88 In addition to all the normal file maintenance, print all new URLs
89 to STDOUT, one per line.
90 "-o" or "--options"
92 Specify the query options. The query options will be submitted to
93 the user search engine with the query string. This feature permits
94 modification of the query string for a specific search engine or
95 option. More than one query option may be specified.
96 Example: "-o what=news" causes AltaVista to search Usenet.
98 Although this works, the preferred mechanism in this case would be
99 "-e AltaVista::News" or "-e AltaVista::AdvancedNews". Options are
100 intended for internal or expert use.
101 "-f" or "--uf" or "--urlfilter"
103 This option specifies a regular expression which will be compared
104 against the URLs of any results; if they match the case-insensitive
105 regular expression, they will be removed from the hit set.
106 Example: "-f '.*\.isi\.edu'" avoids all of ISI's web pages.
108 "--cleanup i"
110 Delete all traces of query results from more than i days ago. If
111 --cleanup is given, all other options other than the qid will be
112 ignored.
113 "--cmdline"
115 Reconstruct the complete command line (AutoSearch and all its
116 arguments) that was used to create the query results. Command line
117 will be shown on STDERR. If --cmdline is given, all other options
118 other than the qid will be ignored.
119 "--mail user@address" or "-m user@address"
121 After search is complete, send email to that user, listing the NEW
122 results. Email is HTML format. Requires the Email::Send and
123 related modules. If you send email through an SMTP server, you
124 must set environment variable SMTPSERVER to your server name or IP
125 address. If your SMTP server requires password, you must set
126 environment variables SMTPUSERNAME and SMTPPASSWORD. If you send
127 email via sendmail, you should set environment variable SENDMAIL if
128 the sendmail executable is not in the path.
129 "--emailfrom user@address"
131 If your outgoing mail server rejects email from certain users, you
132 can use this argument to set the From: header.
133 "--userid bbunny"
135 If the search engine requires a login/password (e.g.
136 Ebay::Completed), use this.
137 "--password Carr0t5"
139 If the search engine requires a login/password (e.g. Ebay::Mature),
140 use this.
141
143 AutoSearch submits a query to a search engine, produces HTML pages that
144 reflect the set of 'hits' (filtered search results) returned by the
145 search engine, and tracks these results over time. The URL and title
146 are displayed in the qid/index.html, the URL, the title, and
147 description are displayed in the 'weekly' files.
148 To organize these results, each search result is placed in a query
150 information directory (qid). The directory becomes the search results
151 'handle', an easy way to track a set of results. Thus a qid of
152 "/usr/local/htdocs/lsam/autosearch/load_balancing" might locate the
153 results on your web server at
154 "http://www.isi.edu/lsam/autosearch/load_balancing".
155 Inside the qid directory you will find files relating to this query.
157 The primary file is index.html, which reflects the latest search
158 results. Every not-filtered hit for every search is stored in
159 index.html. When a hit is no longer found by the search engine it a
160 removed from index.html. As new results for a search are returned from
161 the search engine they are placed in index.html.
162 At the bottom of index.html, there is a heading "Weekly Search
164 Results", which is updated each time the search is submitted (see
165 "AUTOMATED SEARCHING"). The list of search runs is stored in reverse
166 chronological order. Runs which provide no new information are
167 identified with
168 No Unique Results found for search on <date>
170 Runs which contain changes are identified by
172 Web search results for search on <date>
174 which will be linked a page detailing the changes from that run.
176 Detailed search results are noted in weekly files. These files are
178 named YYYYMMDD.html and are stored in the qid directory. The weekly
179 files include THE URL, title, and a the description (if available).
180 The title is a link to the original web page.
181
183 On UNIX-like systems, cron(1) may be used to establish periodic
184 searches and the web pages will be maintained by AutoSearch. To
185 establish the first search, use the first example under SYNOPSIS. You
186 must specify the qid, query name and query string. If any of the items
187 are missing, you will be interactively prompted for the missing
188 item(s).
189 Once the first search is complete you can re-run the search with the
191 second form under SYNOPSIS.
192 A cron entry like:
194 0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching
196 might be used to run the search each Monday at 3:00 AM. The query name
198 and query string may be repeated; but they will not be used. This
199 means that with a cron line like:
200 0 3 * * 1 /nfs/u1/wls/AutoSearch.pl /www/div7/lsam/autosearch/caching -n caching -s caching
202 a whole new search series can be originated by
204 rm -r /www/div7/lsam/autosearch/caching
206 However, the only reason to start a new search series would be to throw
208 away the old weekly files.
209 We don't recommend running searches more than once per day, but if so
211 the per-run files will be updated in-place. Any changes are added to
212 the page with a comment that "Recently Added:"; and deletions are
213 indicated with "Recently Suspended:."
214
216 The basic format of these two pages is simple and customizable. One
217 requirement is that the basic structure remain unchanged. HTML
218 comments are used to identify sections of the document. Almost
219 everything can be changed except for the strings which identify the
220 section starts and ends.
221 Noteworthy tags and their meaning:
223 <!--Top-->.*<!--/Top-->
225 The text contained within this tag is placed at the top
226 of the output page. If the text contains AutoSearch
227 WEB Searching, then the query name will replace it. If
228 the text does not contain this magic string and it is
229 the first ever search, the user will be asked for a
230 query name.
231 <!--Query{.*}/Query-->
233 The text contained between the braces is the query
234 string. This is how AutoSearch maintains the query
235 string. You may edit this string to change the query
236 string; but only in qid/index.html. The text ask user
237 is special and will force AutoSearch to request the
238 search string from the user.
239 <!--SearchEngine{.*}/SearchEngine-->
241 The text contained between the braces is the search
242 engine. Other engines supported are HotBot and Lycos.
243 You may edit this string to change the engine used; but
244 only in qid/index.html. The text ask user is special
245 and will force AutoSearch to to request the search
246 string from the user.
247 <!--QueryOptions{.*}/QueryOptions-->
249 The text contained between the braces specifies a query
250 options. Multiple occurrencs of this command are
251 allowed to specify multiple options.
252 <!--URLFilter{.*}/URLFilter-->
254 The text contained between the braces is the URL
255 filter. This is how AutoSearch maintains the filter.
256 Again you may edit this string to change the query
257 string; but only in qid/index.html. The text ask user
258 is special and will force AutoSearch to ask the user
259 (STDIN) for the query string. When setting up the
260 first search, you must edit first_index.html, not
261 qid/index.html. The URL filter is a standard perl5
262 regular expression. URLs which do not match will be
263 kept.
264 <!--Bottom-->.*<!--/Bottom-->
266 The text contained within this tag is placed at the
267 bottom of the output page. This is a good place to put
268 navigation, page owner information, etc.
269 The remainder of the tags fall into a triplet of ~Heading, ~Template,
271 and ~, where ~ is Summary, Weekly, Appended, and Suspended. The sub-
272 sections appear in the order given. To produce a section AutoSearch
273 outputs the heading, the template, the section, n copies of the
274 formatted data, and an /section. The tags and their function are:
275 ~Heading The heading tag identifies the heading for a section of
277 the output file. The SummaryHeading is for the summary
278 portion, etc. The section may be empty (e.g.,
279 Suspended) and thus no heading is output.
280 ~Template The template tag identifies how each item is to be
282 formatted. Simple text replacement is used to change
283 the template into the actual output text. The text to
284 be replaced is noted in ALLCAPS.
285 ~ This tag is used to locate the section (Summary,
287 Weekly, etc.). This section represents the actual
288 n-items of data.
289 You can edit these values in the qid/index.html page of an existing
291 search. The file first_index.html (in the directory above qid) will be
292 used as a default template for new queries.
293 Examples of these files can be seen in the pages under
295 "http://www.isi.edu/lsam/tools/autosearch/", or in the output generated
296 by a new AutoSearch.
297
299 first_index.html optional file to determine the default format of
300 the index.html file of a new query.
301 first_date.html optional file to determine the default format of
303 the YYYYMMDD.html file for a new query.
304 qid/index.html (automatically created) latest search results, and
306 reverse chronological list of periodic searches.
307 qid/date.html file used as a template for the YYYYMMDD.html
309 files.
310 qid/YYYYMMDD.html (automatically created) summary of changes for a
312 particular date (AKA 'Weekly' file).
313 Optional files first_index.html and first_date.html are used for the
315 initial search as a template for qid/index.html and date.html,
316 respectively. If either of these files does not exist; a default-
317 default template is stored within the AutoSearch source. The intention
318 of these two files is to permit a user to establish a framework for a
319 group of search sets which have a common format. By leaving the
320 default query name and query string alone, they will be overridden by
321 command line inputs.
322
324 For the library, see WWW::Search, for the perl regular expressions, see
325 perlre.
326
328 Wm. L. Scheding
329 AutoSearch is a re-implementation of an earlier version written by
331 Kedar Jog.
332
334 Copyright (C) 1996-1997 University of Southern California. All rights
335 reserved.
336 Redistribution and use in source and binary forms are permitted
338 provided that the above copyright notice and this paragraph are
339 duplicated in all such forms and that any documentation, advertising
340 materials, and other materials related to such distribution and use
341 acknowledge that the software was developed by the University of
342 Southern California, Information Sciences Institute. The name of the
343 University may not be used to endorse or promote products derived from
344 this software without specific prior written permission.
345 THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
347 WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
348 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
349
351 These are good ideas that people have suggested.
352 URL validation.
354 Validate the status of each URL (with HTTP HEAD requests) and
355 indicate this status in the output.
356 Multi-search.
358 It should be possible to merge the results of searches from two
359 search-engines. If this merger were done as a new search engine,
360 this operation would be transparent to AutoSearch.
361
363 None known at this time; please inform the maintainer mthurn@cpan.org
364 if any crop up.
365perl v5.12.0 2008-07-14 AUTOSEARCH(1)