1PERLFAQ9(1) Perl Programmers Reference Guide PERLFAQ9(1)
2
6 perlfaq9 - Networking ($Revision: 1.28 $, $Date: 2005/12/31 00:54:37 $)
7
9 This section deals with questions related to networking, the internet,
10 and a few on the web.
11 What is the correct form of response from a CGI script?
13 (Alan Flavell <flavell+www@a5.ph.gla.ac.uk> answers...)
15 The Common Gateway Interface (CGI) specifies a software interface
17 between a program ("CGI script") and a web server (HTTPD). It is not
18 specific to Perl, and has its own FAQs and tutorials, and usenet group,
19 comp.infosystems.www.authoring.cgi
20 The CGI specification is outlined in an informational RFC:
22 http://www.ietf.org/rfc/rfc3875
23 Other relevant documentation listed in:
25 http://www.perl.org/CGI_MetaFAQ.html
26 These Perl FAQs very selectively cover some CGI issues. However, Perl
28 programmers are strongly advised to use the CGI.pm module, to take care
29 of the details for them.
30 The similarity between CGI response headers (defined in the CGI speci‐
32 fication) and HTTP response headers (defined in the HTTP specification,
33 RFC2616) is intentional, but can sometimes be confusing.
34 The CGI specification defines two kinds of script: the "Parsed Header"
36 script, and the "Non Parsed Header" (NPH) script. Check your server
37 documentation to see what it supports. "Parsed Header" scripts are sim‐
38 pler in various respects. The CGI specification allows any of the usual
39 newline representations in the CGI response (it's the server's job to
40 create an accurate HTTP response based on it). So "\n" written in text
41 mode is technically correct, and recommended. NPH scripts are more
42 tricky: they must put out a complete and accurate set of HTTP transac‐
43 tion response headers; the HTTP specification calls for records to be
44 terminated with carriage-return and line-feed, i.e ASCII \015\012 writ‐
45 ten in binary mode.
46 Using CGI.pm gives excellent platform independence, including EBCDIC
48 systems. CGI.pm selects an appropriate newline representation
49 ($CGI::CRLF) and sets binmode as appropriate.
50 My CGI script runs from the command line but not the browser. (500
52 Server Error)
53 Several things could be wrong. You can go through the "Troubleshooting
55 Perl CGI scripts" guide at
56 http://www.perl.org/troubleshooting_CGI.html
58 If, after that, you can demonstrate that you've read the FAQs and that
60 your problem isn't something simple that can be easily answered, you'll
61 probably receive a courteous and useful reply to your question if you
62 post it on comp.infosystems.www.authoring.cgi (if it's something to do
63 with HTTP or the CGI protocols). Questions that appear to be Perl
64 questions but are really CGI ones that are posted to
65 comp.lang.perl.misc are not so well received.
66 The useful FAQs, related documents, and troubleshooting guides are
68 listed in the CGI Meta FAQ:
69 http://www.perl.org/CGI_MetaFAQ.html
71 How can I get better error messages from a CGI program?
73 Use the CGI::Carp module. It replaces "warn" and "die", plus the nor‐
75 mal Carp modules "carp", "croak", and "confess" functions with more
76 verbose and safer versions. It still sends them to the normal server
77 error log.
78 use CGI::Carp;
80 warn "This is a complaint";
81 die "But this one is serious";
82 The following use of CGI::Carp also redirects errors to a file of your
84 choice, placed in a BEGIN block to catch compile-time warnings as well:
85 BEGIN {
87 use CGI::Carp qw(carpout);
88 open(LOG, ">>/var/local/cgi-logs/mycgi-log")
89 or die "Unable to append to mycgi-log: $!\n";
90 carpout(*LOG);
91 }
92 You can even arrange for fatal errors to go back to the client browser,
94 which is nice for your own debugging, but might confuse the end user.
95 use CGI::Carp qw(fatalsToBrowser);
97 die "Bad error here";
98 Even if the error happens before you get the HTTP header out, the mod‐
100 ule will try to take care of this to avoid the dreaded server 500
101 errors. Normal warnings still go out to the server error log (or wher‐
102 ever you've sent them with "carpout") with the application name and
103 date stamp prepended.
104 How do I remove HTML from a string?
106 The most correct way (albeit not the fastest) is to use HTML::Parser
108 from CPAN. Another mostly correct way is to use HTML::FormatText which
109 not only removes HTML but also attempts to do a little simple format‐
110 ting of the resulting plain text.
111 Many folks attempt a simple-minded regular expression approach, like
113 "s/<.*?>//g", but that fails in many cases because the tags may con‐
114 tinue over line breaks, they may contain quoted angle-brackets, or HTML
115 comment may be present. Plus, folks forget to convert entities--like
116 "<" for example.
117 Here's one "simple-minded" approach, that works for most files:
119 #!/usr/bin/perl -p0777
121 s/<(?:[^>'"]*⎪(['"]).*?\1)*>//gs
122 If you want a more complete solution, see the 3-stage striphtml program
124 in http://www.cpan.org/authors/Tom_Christiansen/scripts/striphtml.gz .
125 Here are some tricky cases that you should think about when picking a
127 solution:
128 <IMG SRC = "foo.gif" ALT = "A > B">
130 <IMG SRC = "foo.gif"
132 ALT = "A > B">
133 <!-- <A comment> -->
135 <script>if (a<b && a>c)</script>
137 <# Just data #>
139 <![INCLUDE CDATA [ >>>>>>>>>>>> ]]>
141 If HTML comments include other tags, those solutions would also break
143 on text like this:
144 <!-- This section commented out.
146 <B>You can't see me!</B>
147 -->
148 How do I extract URLs?
150 You can easily extract all sorts of URLs from HTML with "HTML::Sim‐
152 pleLinkExtor" which handles anchors, images, objects, frames, and many
153 other tags that can contain a URL. If you need anything more complex,
154 you can create your own subclass of "HTML::LinkExtor" or
155 "HTML::Parser". You might even use "HTML::SimpleLinkExtor" as an exam‐
156 ple for something specifically suited to your needs.
157 You can use URI::Find to extract URLs from an arbitrary text document.
159 Less complete solutions involving regular expressions can save you a
161 lot of processing time if you know that the input is simple. One solu‐
162 tion from Tom Christiansen runs 100 times faster than most module based
163 approaches but only extracts URLs from anchors where the first
164 attribute is HREF and there are no other attributes.
165 #!/usr/bin/perl -n00
167 # qxurl - tchrist@perl.com
168 print "$2\n" while m{
169 < \s*
170 A \s+ HREF \s* = \s* (["']) (.*?) \1
171 \s* >
172 }gsix;
173 How do I download a file from the user's machine? How do I open a file
175 on another machine?
176 In this case, download means to use the file upload feature of HTML
178 forms. You allow the web surfer to specify a file to send to your web
179 server. To you it looks like a download, and to the user it looks like
180 an upload. No matter what you call it, you do it with what's known as
181 multipart/form-data encoding. The CGI.pm module (which comes with Perl
182 as part of the Standard Library) supports this in the start_multi‐
183 part_form() method, which isn't the same as the startform() method.
184 See the section in the CGI.pm documentation on file uploads for code
186 examples and details.
187 How do I make a pop-up menu in HTML?
189 Use the <SELECT> and <OPTION> tags. The CGI.pm module (available from
191 CPAN) supports this widget, as well as many others, including some that
192 it cleverly synthesizes on its own.
193 How do I fetch an HTML file?
195 One approach, if you have the lynx text-based HTML browser installed on
197 your system, is this:
198 $html_code = `lynx -source $url`;
200 $text_data = `lynx -dump $url`;
201 The libwww-perl (LWP) modules from CPAN provide a more powerful way to
203 do this. They don't require lynx, but like lynx, can still work
204 through proxies:
205 # simplest version
207 use LWP::Simple;
208 $content = get($URL);
209 # or print HTML from a URL
211 use LWP::Simple;
212 getprint "http://www.linpro.no/lwp/";
213 # or print ASCII from HTML from a URL
215 # also need HTML-Tree package from CPAN
216 use LWP::Simple;
217 use HTML::Parser;
218 use HTML::FormatText;
219 my ($html, $ascii);
220 $html = get("http://www.perl.com/");
221 defined $html
222 or die "Can't fetch HTML from http://www.perl.com/";
223 $ascii = HTML::FormatText->new->format(parse_html($html));
224 print $ascii;
225 How do I automate an HTML form submission?
227 If you are doing something complex, such as moving through many pages
229 and forms or a web site, you can use "WWW::Mechanize". See its docu‐
230 mentation for all the details.
231 If you're submitting values using the GET method, create a URL and
233 encode the form using the "query_form" method:
234 use LWP::Simple;
236 use URI::URL;
237 my $url = url('http://www.perl.com/cgi-bin/cpan_mod');
239 $url->query_form(module => 'DB_File', readme => 1);
240 $content = get($url);
241 If you're using the POST method, create your own user agent and encode
243 the content appropriately.
244 use HTTP::Request::Common qw(POST);
246 use LWP::UserAgent;
247 $ua = LWP::UserAgent->new();
249 my $req = POST 'http://www.perl.com/cgi-bin/cpan_mod',
250 [ module => 'DB_File', readme => 1 ];
251 $content = $ua->request($req)->as_string;
252 How do I decode or create those %-encodings on the web?
254 If you are writing a CGI script, you should be using the CGI.pm module
256 that comes with perl, or some other equivalent module. The CGI module
257 automatically decodes queries for you, and provides an escape() func‐
258 tion to handle encoding.
259 The best source of detailed information on URI encoding is RFC 2396.
261 Basically, the following substitutions do it:
262 s/([^\w()'*~!.-])/sprintf '%%%02x', ord $1/eg; # encode
264 s/%([A-Fa-f\d]{2})/chr hex $1/eg; # decode
266 s/%([[:xdigit:]]{2})/chr hex $1/eg; # same thing
267 However, you should only apply them to individual URI components, not
269 the entire URI, otherwise you'll lose information and generally mess
270 things up. If that didn't explain it, don't worry. Just go read sec‐
271 tion 2 of the RFC, it's probably the best explanation there is.
272 RFC 2396 also contains a lot of other useful information, including a
274 regexp for breaking any arbitrary URI into components (Appendix B).
275 How do I redirect to another page?
277 Specify the complete URL of the destination (even if it is on the same
279 server). This is one of the two different kinds of CGI "Location:"
280 responses which are defined in the CGI specification for a Parsed Head‐
281 ers script. The other kind (an absolute URLpath) is resolved internally
282 to the server without any HTTP redirection. The CGI specifications do
283 not allow relative URLs in either case.
284 Use of CGI.pm is strongly recommended. This example shows redirection
286 with a complete URL. This redirection is handled by the web browser.
287 use CGI qw/:standard/;
289 my $url = 'http://www.cpan.org/';
291 print redirect($url);
292 This example shows a redirection with an absolute URLpath. This redi‐
294 rection is handled by the local web server.
295 my $url = '/CPAN/index.html';
297 print redirect($url);
298 But if coded directly, it could be as follows (the final "\n" is shown
300 separately, for clarity), using either a complete URL or an absolute
301 URLpath.
302 print "Location: $url\n"; # CGI response header
304 print "\n"; # end of headers
305 How do I put a password on my web pages?
307 To enable authentication for your web server, you need to configure
309 your web server. The configuration is different for different sorts of
310 web servers---apache does it differently from iPlanet which does it
311 differently from IIS. Check your web server documentation for the
312 details for your particular server.
313 How do I edit my .htpasswd and .htgroup files with Perl?
315 The HTTPD::UserAdmin and HTTPD::GroupAdmin modules provide a consistent
317 OO interface to these files, regardless of how they're stored. Data‐
318 bases may be text, dbm, Berkeley DB or any database with a DBI compati‐
319 ble driver. HTTPD::UserAdmin supports files used by the "Basic" and
320 "Digest" authentication schemes. Here's an example:
321 use HTTPD::UserAdmin ();
323 HTTPD::UserAdmin
324 ->new(DB => "/foo/.htpasswd")
325 ->add($username => $password);
326 How do I make sure users can't enter values into a form that cause my
328 CGI script to do bad things?
329 See the security references listed in the CGI Meta FAQ
331 http://www.perl.org/CGI_MetaFAQ.html
333 How do I parse a mail header?
335 For a quick-and-dirty solution, try this solution derived from "split"
337 in perlfunc:
338 $/ = '';
340 $header = <MSG>;
341 $header =~ s/\n\s+/ /g; # merge continuation lines
342 %head = ( UNIX_FROM_LINE, split /^([-\w]+):\s*/m, $header );
343 That solution doesn't do well if, for example, you're trying to main‐
345 tain all the Received lines. A more complete approach is to use the
346 Mail::Header module from CPAN (part of the MailTools package).
347 How do I decode a CGI form?
349 (contributed by brian d foy)
351 Use the CGI.pm module that comes with Perl. It's quick, it's easy, and
353 it actually does quite a bit of work to ensure things happen correctly.
354 It handles GET, POST, and HEAD requests, multipart forms, multivalued
355 fields, query string and message body combinations, and many other
356 things you probably don't want to think about.
357 It doesn't get much easier: the CGI module automatically parses the
359 input and makes each value available through the "param()" function.
360 use CGI qw(:standard);
362 my $total = param( 'price' ) + param( 'shipping' );
364 my @items = param( 'item' ); # multiple values, same field name
366 If you want an object-oriented approach, CGI.pm can do that too.
368 use CGI;
370 my $cgi = CGI->new();
372 my $total = $cgi->param( 'price' ) + $cgi->param( 'shipping' );
374 my @items = $cgi->param( 'item' );
376 You might also try CGI::Minimal which is a lightweight version of the
378 same thing. Other CGI::* modules on CPAN might work better for you,
379 too.
380 Many people try to write their own decoder (or copy one from another
382 program) and then run into one of the many "gotchas" of the task. It's
383 much easier and less hassle to use CGI.pm.
384 How do I check a valid mail address?
386 You can't, at least, not in real time. Bummer, eh?
388 Without sending mail to the address and seeing whether there's a human
390 on the other end to answer you, you cannot determine whether a mail
391 address is valid. Even if you apply the mail header standard, you can
392 have problems, because there are deliverable addresses that aren't
393 RFC-822 (the mail header standard) compliant, and addresses that aren't
394 deliverable which are compliant.
395 You can use the Email::Valid or RFC::RFC822::Address which check the
397 format of the address, although they cannot actually tell you if it is
398 a deliverable address (i.e. that mail to the address will not bounce).
399 Modules like Mail::CheckUser and Mail::EXPN try to interact with the
400 domain name system or particular mail servers to learn even more, but
401 their methods do not work everywhere---especially for security con‐
402 scious administrators.
403 Many are tempted to try to eliminate many frequently-invalid mail
405 addresses with a simple regex, such as "/^[\w.-]+\@(?:[\w-]+\.)+\w+$/".
406 It's a very bad idea. However, this also throws out many valid ones,
407 and says nothing about potential deliverability, so it is not sug‐
408 gested. Instead, see http://www.cpan.org/authors/Tom_Chris‐
409 tiansen/scripts/ckaddr.gz , which actually checks against the full RFC
410 spec (except for nested comments), looks for addresses you may not wish
411 to accept mail to (say, Bill Clinton or your postmaster), and then
412 makes sure that the hostname given can be looked up in the DNS MX
413 records. It's not fast, but it works for what it tries to do.
414 Our best advice for verifying a person's mail address is to have them
416 enter their address twice, just as you normally do to change a pass‐
417 word. This usually weeds out typos. If both versions match, send mail
418 to that address with a personal message that looks somewhat like:
419 Dear someuser@host.com,
421 Please confirm the mail address you gave us Wed May 6 09:38:41
423 MDT 1998 by replying to this message. Include the string
424 "Rumpelstiltskin" in that reply, but spelled in reverse; that is,
425 start with "Nik...". Once this is done, your confirmed address will
426 be entered into our records.
427 If you get the message back and they've followed your directions, you
429 can be reasonably assured that it's real.
430 A related strategy that's less open to forgery is to give them a PIN
432 (personal ID number). Record the address and PIN (best that it be a
433 random one) for later processing. In the mail you send, ask them to
434 include the PIN in their reply. But if it bounces, or the message is
435 included via a "vacation" script, it'll be there anyway. So it's best
436 to ask them to mail back a slight alteration of the PIN, such as with
437 the characters reversed, one added or subtracted to each digit, etc.
438 How do I decode a MIME/BASE64 string?
440 The MIME-Base64 package (available from CPAN) handles this as well as
442 the MIME/QP encoding. Decoding BASE64 becomes as simple as:
443 use MIME::Base64;
445 $decoded = decode_base64($encoded);
446 The MIME-Tools package (available from CPAN) supports extraction with
448 decoding of BASE64 encoded attachments and content directly from email
449 messages.
450 If the string to decode is short (less than 84 bytes long) a more
452 direct approach is to use the unpack() function's "u" format after
453 minor transliterations:
454 tr#A-Za-z0-9+/##cd; # remove non-base64 chars
456 tr#A-Za-z0-9+/# -_#; # convert to uuencoded format
457 $len = pack("c", 32 + 0.75*length); # compute length byte
458 print unpack("u", $len . $_); # uudecode and print
459 How do I return the user's mail address?
461 On systems that support getpwuid, the $< variable, and the Sys::Host‐
463 name module (which is part of the standard perl distribution), you can
464 probably try using something like this:
465 use Sys::Hostname;
467 $address = sprintf('%s@%s', scalar getpwuid($<), hostname);
468 Company policies on mail address can mean that this generates addresses
470 that the company's mail system will not accept, so you should ask for
471 users' mail addresses when this matters. Furthermore, not all systems
472 on which Perl runs are so forthcoming with this information as is Unix.
473 The Mail::Util module from CPAN (part of the MailTools package) pro‐
475 vides a mailaddress() function that tries to guess the mail address of
476 the user. It makes a more intelligent guess than the code above, using
477 information given when the module was installed, but it could still be
478 incorrect. Again, the best way is often just to ask the user.
479 How do I send mail?
481 Use the "sendmail" program directly:
483 open(SENDMAIL, "⎪/usr/lib/sendmail -oi -t -odq")
485 or die "Can't fork for sendmail: $!\n";
486 print SENDMAIL <<"EOF";
487 From: User Originating Mail <me\@host>
488 To: Final Destination <you\@otherhost>
489 Subject: A relevant subject line
490 Body of the message goes here after the blank line
492 in as many lines as you like.
493 EOF
494 close(SENDMAIL) or warn "sendmail didn't close nicely";
495 The -oi option prevents sendmail from interpreting a line consisting of
497 a single dot as "end of message". The -t option says to use the head‐
498 ers to decide who to send the message to, and -odq says to put the mes‐
499 sage into the queue. This last option means your message won't be
500 immediately delivered, so leave it out if you want immediate delivery.
501 Alternate, less convenient approaches include calling mail (sometimes
503 called mailx) directly or simply opening up port 25 have having an
504 intimate conversation between just you and the remote SMTP daemon,
505 probably sendmail.
506 Or you might be able use the CPAN module Mail::Mailer:
508 use Mail::Mailer;
510 $mailer = Mail::Mailer->new();
512 $mailer->open({ From => $from_address,
513 To => $to_address,
514 Subject => $subject,
515 })
516 or die "Can't open: $!\n";
517 print $mailer $body;
518 $mailer->close();
519 The Mail::Internet module uses Net::SMTP which is less Unix-centric
521 than Mail::Mailer, but less reliable. Avoid raw SMTP commands. There
522 are many reasons to use a mail transport agent like sendmail. These
523 include queuing, MX records, and security.
524 How do I use MIME to make an attachment to a mail message?
526 This answer is extracted directly from the MIME::Lite documentation.
528 Create a multipart message (i.e., one with attachments).
529 use MIME::Lite;
531 ### Create a new multipart message:
533 $msg = MIME::Lite->new(
534 From =>'me@myhost.com',
535 To =>'you@yourhost.com',
536 Cc =>'some@other.com, some@more.com',
537 Subject =>'A message with 2 parts...',
538 Type =>'multipart/mixed'
539 );
540 ### Add parts (each "attach" has same arguments as "new"):
542 $msg->attach(Type =>'TEXT',
543 Data =>"Here's the GIF file you wanted"
544 );
545 $msg->attach(Type =>'image/gif',
546 Path =>'aaa000123.gif',
547 Filename =>'logo.gif'
548 );
549 $text = $msg->as_string;
551 MIME::Lite also includes a method for sending these things.
553 $msg->send;
555 This defaults to using sendmail but can be customized to use SMTP via
557 Net::SMTP.
558 How do I read mail?
560 While you could use the Mail::Folder module from CPAN (part of the
562 MailFolder package) or the Mail::Internet module from CPAN (part of the
563 MailTools package), often a module is overkill. Here's a mail sorter.
564 #!/usr/bin/perl
566 my(@msgs, @sub);
568 my $msgno = -1;
569 $/ = ''; # paragraph reads
570 while (<>) {
571 if (/^From /m) {
572 /^Subject:\s*(?:Re:\s*)*(.*)/mi;
573 $sub[++$msgno] = lc($1) ⎪⎪ '';
574 }
575 $msgs[$msgno] .= $_;
576 }
577 for my $i (sort { $sub[$a] cmp $sub[$b] ⎪⎪ $a <=> $b } (0 .. $#msgs)) {
578 print $msgs[$i];
579 }
580 Or more succinctly,
582 #!/usr/bin/perl -n00
584 # bysub2 - awkish sort-by-subject
585 BEGIN { $msgno = -1 }
586 $sub[++$msgno] = (/^Subject:\s*(?:Re:\s*)*(.*)/mi)[0] if /^From/m;
587 $msg[$msgno] .= $_;
588 END { print @msg[ sort { $sub[$a] cmp $sub[$b] ⎪⎪ $a <=> $b } (0 .. $#msg) ] }
589 How do I find out my hostname, domainname, or IP address?
591 gethostbyname, Socket, Net::Domain, Sys::Hostname" (contributed by
593 brian d foy)
594 The Net::Domain module, which is part of the standard distribution
596 starting in perl5.7.3, can get you the fully qualified domain name
597 (FQDN), the host name, or the domain name.
598 use Net::Domain qw(hostname hostfqdn hostdomain);
600 my $host = hostfqdn();
602 The "Sys::Hostname" module, included in the standard distribution since
604 perl5.6, can also get the hostname.
605 use Sys::Hostname;
607 $host = hostname();
609 To get the IP address, you can use the "gethostbyname" built-in func‐
611 tion to turn the name into a number. To turn that number into the dot‐
612 ted octet form (a.b.c.d) that most people expect, use the "inet_ntoa"
613 function from the <Socket> module, which also comes with perl.
614 use Socket;
616 my $address = inet_ntoa(
618 scalar gethostbyname( $host ⎪⎪ 'localhost' )
619 );
620 How do I fetch a news article or the active newsgroups?
622 Use the Net::NNTP or News::NNTPClient modules, both available from
624 CPAN. This can make tasks like fetching the newsgroup list as simple
625 as
626 perl -MNews::NNTPClient
628 -e 'print News::NNTPClient->new->list("newsgroups")'
629 How do I fetch/put an FTP file?
631 LWP::Simple (available from CPAN) can fetch but not put. Net::FTP
633 (also available from CPAN) is more complex but can put as well as
634 fetch.
635 How can I do RPC in Perl?
637 (Contributed by brian d foy)
639 Use one of the RPC modules you can find on CPAN (
641 http://search.cpan.org/search?query=RPC&mode=all ).
642
644 Copyright (c) 1997-2006 Tom Christiansen, Nathan Torkington, and other
645 authors as noted. All rights reserved.
646 This documentation is free; you can redistribute it and/or modify it
648 under the same terms as Perl itself.
649 Irrespective of its distribution, all code examples in this file are
651 hereby placed into the public domain. You are permitted and encouraged
652 to use this code in your own programs for fun or for profit as you see
653 fit. A simple comment in the code giving credit would be courteous but
654 is not required.
655perl v5.8.8 2006-01-07 PERLFAQ9(1)