1PERLFAQ9(1) Perl Programmers Reference Guide PERLFAQ9(1)
2
6 perlfaq9 - Networking
7
9 This section deals with questions related to networking, the internet,
10 and a few on the web.
11 What is the correct form of response from a CGI script?
13 (Alan Flavell <flavell+www@a5.ph.gla.ac.uk> answers...)
14 The Common Gateway Interface (CGI) specifies a software interface
16 between a program ("CGI script") and a web server (HTTPD). It is not
17 specific to Perl, and has its own FAQs and tutorials, and usenet group,
18 comp.infosystems.www.authoring.cgi
19 The CGI specification is outlined in an informational RFC:
21 http://www.ietf.org/rfc/rfc3875
22 Other relevant documentation listed in:
24 http://www.perl.org/CGI_MetaFAQ.html
25 These Perl FAQs very selectively cover some CGI issues. However, Perl
27 programmers are strongly advised to use the "CGI.pm" module, to take
28 care of the details for them.
29 The similarity between CGI response headers (defined in the CGI
31 specification) and HTTP response headers (defined in the HTTP
32 specification, RFC2616) is intentional, but can sometimes be confusing.
33 The CGI specification defines two kinds of script: the "Parsed Header"
35 script, and the "Non Parsed Header" (NPH) script. Check your server
36 documentation to see what it supports. "Parsed Header" scripts are
37 simpler in various respects. The CGI specification allows any of the
38 usual newline representations in the CGI response (it's the server's
39 job to create an accurate HTTP response based on it). So "\n" written
40 in text mode is technically correct, and recommended. NPH scripts are
41 more tricky: they must put out a complete and accurate set of HTTP
42 transaction response headers; the HTTP specification calls for records
43 to be terminated with carriage-return and line-feed, i.e ASCII \015\012
44 written in binary mode.
45 Using "CGI.pm" gives excellent platform independence, including EBCDIC
47 systems. "CGI.pm" selects an appropriate newline representation
48 ($CGI::CRLF) and sets binmode as appropriate.
49 My CGI script runs from the command line but not the browser. (500 Server
51 Error)
52 Several things could be wrong. You can go through the "Troubleshooting
53 Perl CGI scripts" guide at
54 http://www.perl.org/troubleshooting_CGI.html
56 If, after that, you can demonstrate that you've read the FAQs and that
58 your problem isn't something simple that can be easily answered, you'll
59 probably receive a courteous and useful reply to your question if you
60 post it on comp.infosystems.www.authoring.cgi (if it's something to do
61 with HTTP or the CGI protocols). Questions that appear to be Perl
62 questions but are really CGI ones that are posted to
63 comp.lang.perl.misc are not so well received.
64 The useful FAQs, related documents, and troubleshooting guides are
66 listed in the CGI Meta FAQ:
67 http://www.perl.org/CGI_MetaFAQ.html
69 How can I get better error messages from a CGI program?
71 Use the "CGI::Carp" module. It replaces "warn" and "die", plus the
72 normal "Carp" modules "carp", "croak", and "confess" functions with
73 more verbose and safer versions. It still sends them to the normal
74 server error log.
75 use CGI::Carp;
77 warn "This is a complaint";
78 die "But this one is serious";
79 The following use of "CGI::Carp" also redirects errors to a file of
81 your choice, placed in a "BEGIN" block to catch compile-time warnings
82 as well:
83 BEGIN {
85 use CGI::Carp qw(carpout);
86 open(LOG, ">>/var/local/cgi-logs/mycgi-log")
87 or die "Unable to append to mycgi-log: $!\n";
88 carpout(*LOG);
89 }
90 You can even arrange for fatal errors to go back to the client browser,
92 which is nice for your own debugging, but might confuse the end user.
93 use CGI::Carp qw(fatalsToBrowser);
95 die "Bad error here";
96 Even if the error happens before you get the HTTP header out, the
98 module will try to take care of this to avoid the dreaded server 500
99 errors. Normal warnings still go out to the server error log (or
100 wherever you've sent them with "carpout") with the application name and
101 date stamp prepended.
102 How do I remove HTML from a string?
104 The most correct way (albeit not the fastest) is to use "HTML::Parser"
105 from CPAN. Another mostly correct way is to use "HTML::FormatText"
106 which not only removes HTML but also attempts to do a little simple
107 formatting of the resulting plain text.
108 Many folks attempt a simple-minded regular expression approach, like
110 "s/<.*?>//g", but that fails in many cases because the tags may
111 continue over line breaks, they may contain quoted angle-brackets, or
112 HTML comment may be present. Plus, folks forget to convert
113 entities--like "<" for example.
114 Here's one "simple-minded" approach, that works for most files:
116 #!/usr/bin/perl -p0777
118 s/<(?:[^>'"]*|(['"]).*?\1)*>//gs
119 If you want a more complete solution, see the 3-stage striphtml program
121 in http://www.cpan.org/authors/Tom_Christiansen/scripts/striphtml.gz .
122 Here are some tricky cases that you should think about when picking a
124 solution:
125 <IMG SRC = "foo.gif" ALT = "A > B">
127 <IMG SRC = "foo.gif"
129 ALT = "A > B">
130 <!-- <A comment> -->
132 <script>if (a<b && a>c)</script>
134 <# Just data #>
136 <![INCLUDE CDATA [ >>>>>>>>>>>> ]]>
138 If HTML comments include other tags, those solutions would also break
140 on text like this:
141 <!-- This section commented out.
143 <B>You can't see me!</B>
144 -->
145 How do I extract URLs?
147 You can easily extract all sorts of URLs from HTML with
148 "HTML::SimpleLinkExtor" which handles anchors, images, objects, frames,
149 and many other tags that can contain a URL. If you need anything more
150 complex, you can create your own subclass of "HTML::LinkExtor" or
151 "HTML::Parser". You might even use "HTML::SimpleLinkExtor" as an
152 example for something specifically suited to your needs.
153 You can use "URI::Find" to extract URLs from an arbitrary text
155 document.
156 Less complete solutions involving regular expressions can save you a
158 lot of processing time if you know that the input is simple. One
159 solution from Tom Christiansen runs 100 times faster than most module
160 based approaches but only extracts URLs from anchors where the first
161 attribute is HREF and there are no other attributes.
162 #!/usr/bin/perl -n00
164 # qxurl - tchrist@perl.com
165 print "$2\n" while m{
166 < \s*
167 A \s+ HREF \s* = \s* (["']) (.*?) \1
168 \s* >
169 }gsix;
170 How do I download a file from the user's machine? How do I open a file on
172 another machine?
173 In this case, download means to use the file upload feature of HTML
174 forms. You allow the web surfer to specify a file to send to your web
175 server. To you it looks like a download, and to the user it looks like
176 an upload. No matter what you call it, you do it with what's known as
177 multipart/form-data encoding. The "CGI.pm" module (which comes with
178 Perl as part of the Standard Library) supports this in the
179 "start_multipart_form()" method, which isn't the same as the
180 "startform()" method.
181 See the section in the "CGI.pm" documentation on file uploads for code
183 examples and details.
184 How do I make an HTML pop-up menu with Perl?
186 (contributed by brian d foy)
187 The "CGI.pm" module (which comes with Perl) has functions to create the
189 HTML form widgets. See the "CGI.pm" documentation for more examples.
190 use CGI qw/:standard/;
192 print header,
193 start_html('Favorite Animals'),
194 start_form,
196 "What's your favorite animal? ",
197 popup_menu(
198 -name => 'animal',
199 -values => [ qw( Llama Alpaca Camel Ram ) ]
200 ),
201 submit,
202 end_form,
204 end_html;
205 How do I fetch an HTML file?
207 (contributed by brian d foy)
208 Use the libwww-perl distribution. The "LWP::Simple" module can fetch
210 web resources and give their content back to you as a string:
211 use LWP::Simple qw(get);
213 my $html = get( "http://www.example.com/index.html" );
215 It can also store the resource directly in a file:
217 use LWP::Simple qw(getstore);
219 getstore( "http://www.example.com/index.html", "foo.html" );
221 If you need to do something more complicated, you can use
223 "LWP::UserAgent" module to create your own user-agent (e.g. browser) to
224 get the job done. If you want to simulate an interactive web browser,
225 you can use the "WWW::Mechanize" module.
226 How do I automate an HTML form submission?
228 If you are doing something complex, such as moving through many pages
229 and forms or a web site, you can use "WWW::Mechanize". See its
230 documentation for all the details.
231 If you're submitting values using the GET method, create a URL and
233 encode the form using the "query_form" method:
234 use LWP::Simple;
236 use URI::URL;
237 my $url = url('http://www.perl.com/cgi-bin/cpan_mod');
239 $url->query_form(module => 'DB_File', readme => 1);
240 $content = get($url);
241 If you're using the POST method, create your own user agent and encode
243 the content appropriately.
244 use HTTP::Request::Common qw(POST);
246 use LWP::UserAgent;
247 $ua = LWP::UserAgent->new();
249 my $req = POST 'http://www.perl.com/cgi-bin/cpan_mod',
250 [ module => 'DB_File', readme => 1 ];
251 $content = $ua->request($req)->as_string;
252 How do I decode or create those %-encodings on the web?
254 (contributed by brian d foy)
255 Those "%" encodings handle reserved characters in URIs, as described in
257 RFC 2396, Section 2. This encoding replaces the reserved character with
258 the hexadecimal representation of the character's number from the US-
259 ASCII table. For instance, a colon, ":", becomes %3A.
260 In CGI scripts, you don't have to worry about decoding URIs if you are
262 using "CGI.pm". You shouldn't have to process the URI yourself, either
263 on the way in or the way out.
264 If you have to encode a string yourself, remember that you should never
266 try to encode an already-composed URI. You need to escape the
267 components separately then put them together. To encode a string, you
268 can use the the "URI::Escape" module. The "uri_escape" function returns
269 the escaped string:
270 my $original = "Colon : Hash # Percent %";
272 my $escaped = uri_escape( $original );
274 print "$escaped\n"; # 'Colon%20%3A%20Hash%20%23%20Percent%20%25'
276 To decode the string, use the "uri_unescape" function:
278 my $unescaped = uri_unescape( $escaped );
280 print $unescaped; # back to original
282 If you wanted to do it yourself, you simply need to replace the
284 reserved characters with their encodings. A global substitution is one
285 way to do it:
286 # encode
288 $string =~ s/([^^A-Za-z0-9\-_.!~*'()])/ sprintf "%%%0x", ord $1 /eg;
289 #decode
291 $string =~ s/%([A-Fa-f\d]{2})/chr hex $1/eg;
292 How do I redirect to another page?
294 Specify the complete URL of the destination (even if it is on the same
295 server). This is one of the two different kinds of CGI "Location:"
296 responses which are defined in the CGI specification for a Parsed
297 Headers script. The other kind (an absolute URLpath) is resolved
298 internally to the server without any HTTP redirection. The CGI
299 specifications do not allow relative URLs in either case.
300 Use of "CGI.pm" is strongly recommended. This example shows
302 redirection with a complete URL. This redirection is handled by the web
303 browser.
304 use CGI qw/:standard/;
306 my $url = 'http://www.cpan.org/';
308 print redirect($url);
309 This example shows a redirection with an absolute URLpath. This
311 redirection is handled by the local web server.
312 my $url = '/CPAN/index.html';
314 print redirect($url);
315 But if coded directly, it could be as follows (the final "\n" is shown
317 separately, for clarity), using either a complete URL or an absolute
318 URLpath.
319 print "Location: $url\n"; # CGI response header
321 print "\n"; # end of headers
322 How do I put a password on my web pages?
324 To enable authentication for your web server, you need to configure
325 your web server. The configuration is different for different sorts of
326 web servers--apache does it differently from iPlanet which does it
327 differently from IIS. Check your web server documentation for the
328 details for your particular server.
329 How do I edit my .htpasswd and .htgroup files with Perl?
331 The "HTTPD::UserAdmin" and "HTTPD::GroupAdmin" modules provide a
332 consistent OO interface to these files, regardless of how they're
333 stored. Databases may be text, dbm, Berkeley DB or any database with a
334 DBI compatible driver. "HTTPD::UserAdmin" supports files used by the
335 "Basic" and "Digest" authentication schemes. Here's an example:
336 use HTTPD::UserAdmin ();
338 HTTPD::UserAdmin
339 ->new(DB => "/foo/.htpasswd")
340 ->add($username => $password);
341 How do I make sure users can't enter values into a form that cause my CGI
343 script to do bad things?
344 See the security references listed in the CGI Meta FAQ
345 http://www.perl.org/CGI_MetaFAQ.html
347 How do I parse a mail header?
349 For a quick-and-dirty solution, try this solution derived from "split"
350 in perlfunc:
351 $/ = '';
353 $header = <MSG>;
354 $header =~ s/\n\s+/ /g; # merge continuation lines
355 %head = ( UNIX_FROM_LINE, split /^([-\w]+):\s*/m, $header );
356 That solution doesn't do well if, for example, you're trying to
358 maintain all the Received lines. A more complete approach is to use
359 the "Mail::Header" module from CPAN (part of the "MailTools" package).
360 How do I decode a CGI form?
362 (contributed by brian d foy)
363 Use the "CGI.pm" module that comes with Perl. It's quick, it's easy,
365 and it actually does quite a bit of work to ensure things happen
366 correctly. It handles GET, POST, and HEAD requests, multipart forms,
367 multivalued fields, query string and message body combinations, and
368 many other things you probably don't want to think about.
369 It doesn't get much easier: the "CGI.pm" module automatically parses
371 the input and makes each value available through the "param()"
372 function.
373 use CGI qw(:standard);
375 my $total = param( 'price' ) + param( 'shipping' );
377 my @items = param( 'item' ); # multiple values, same field name
379 If you want an object-oriented approach, "CGI.pm" can do that too.
381 use CGI;
383 my $cgi = CGI->new();
385 my $total = $cgi->param( 'price' ) + $cgi->param( 'shipping' );
387 my @items = $cgi->param( 'item' );
389 You might also try "CGI::Minimal" which is a lightweight version of the
391 same thing. Other CGI::* modules on CPAN might work better for you,
392 too.
393 Many people try to write their own decoder (or copy one from another
395 program) and then run into one of the many "gotchas" of the task. It's
396 much easier and less hassle to use "CGI.pm".
397 How do I check a valid mail address?
399 (partly contributed by Aaron Sherman)
400 This isn't as simple a question as it sounds. There are two parts:
402 a) How do I verify that an email address is correctly formatted?
404 b) How do I verify that an email address targets a valid recipient?
406 Without sending mail to the address and seeing whether there's a human
408 on the other end to answer you, you cannot fully answer part b, but
409 either the "Email::Valid" or the "RFC::RFC822::Address" module will do
410 both part a and part b as far as you can in real-time.
411 If you want to just check part a to see that the address is valid
413 according to the mail header standard with a simple regular expression,
414 you can have problems, because there are deliverable addresses that
415 aren't RFC-2822 (the latest mail header standard) compliant, and
416 addresses that aren't deliverable which, are compliant. However, the
417 following will match valid RFC-2822 addresses that do not have
418 comments, folding whitespace, or any other obsolete or non-essential
419 elements. This just matches the address itself:
420 my $atom = qr{[a-zA-Z0-9_!#\$\%&'*+/=?\^`{}~|\-]+};
422 my $dot_atom = qr{$atom(?:\.$atom)*};
423 my $quoted = qr{"(?:\\[^\r\n]|[^\\"])*"};
424 my $local = qr{(?:$dot_atom|$quoted)};
425 my $quotedpair = qr{\\[\x00-\x09\x0B-\x0c\x0e-\x7e]};
426 my $domain_lit = qr{\[(?:$quotedpair|[\x21-\x5a\x5e-\x7e])*\]};
427 my $domain = qr{(?:$dot_atom|$domain_lit)};
428 my $addr_spec = qr{$local\@$domain};
429 Just match an address against "/^${addr_spec}$/" to see if it follows
431 the RFC2822 specification. However, because it is impossible to be
432 sure that such a correctly formed address is actually the correct way
433 to reach a particular person or even has a mailbox associated with it,
434 you must be very careful about how you use this.
435 Our best advice for verifying a person's mail address is to have them
437 enter their address twice, just as you normally do to change a
438 password. This usually weeds out typos. If both versions match, send
439 mail to that address with a personal message. If you get the message
440 back and they've followed your directions, you can be reasonably
441 assured that it's real.
442 A related strategy that's less open to forgery is to give them a PIN
444 (personal ID number). Record the address and PIN (best that it be a
445 random one) for later processing. In the mail you send, ask them to
446 include the PIN in their reply. But if it bounces, or the message is
447 included via a "vacation" script, it'll be there anyway. So it's best
448 to ask them to mail back a slight alteration of the PIN, such as with
449 the characters reversed, one added or subtracted to each digit, etc.
450 How do I decode a MIME/BASE64 string?
452 The "MIME-Base64" package (available from CPAN) handles this as well as
453 the MIME/QP encoding. Decoding BASE64 becomes as simple as:
454 use MIME::Base64;
456 $decoded = decode_base64($encoded);
457 The "MIME-Tools" package (available from CPAN) supports extraction with
459 decoding of BASE64 encoded attachments and content directly from email
460 messages.
461 If the string to decode is short (less than 84 bytes long) a more
463 direct approach is to use the "unpack()" function's "u" format after
464 minor transliterations:
465 tr#A-Za-z0-9+/##cd; # remove non-base64 chars
467 tr#A-Za-z0-9+/# -_#; # convert to uuencoded format
468 $len = pack("c", 32 + 0.75*length); # compute length byte
469 print unpack("u", $len . $_); # uudecode and print
470 How do I return the user's mail address?
472 On systems that support getpwuid, the $< variable, and the
473 "Sys::Hostname" module (which is part of the standard perl
474 distribution), you can probably try using something like this:
475 use Sys::Hostname;
477 $address = sprintf('%s@%s', scalar getpwuid($<), hostname);
478 Company policies on mail address can mean that this generates addresses
480 that the company's mail system will not accept, so you should ask for
481 users' mail addresses when this matters. Furthermore, not all systems
482 on which Perl runs are so forthcoming with this information as is Unix.
483 The "Mail::Util" module from CPAN (part of the "MailTools" package)
485 provides a "mailaddress()" function that tries to guess the mail
486 address of the user. It makes a more intelligent guess than the code
487 above, using information given when the module was installed, but it
488 could still be incorrect. Again, the best way is often just to ask the
489 user.
490 How do I send mail?
492 Use the "sendmail" program directly:
493 open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq")
495 or die "Can't fork for sendmail: $!\n";
496 print SENDMAIL <<"EOF";
497 From: User Originating Mail <me\@host>
498 To: Final Destination <you\@otherhost>
499 Subject: A relevant subject line
500 Body of the message goes here after the blank line
502 in as many lines as you like.
503 EOF
504 close(SENDMAIL) or warn "sendmail didn't close nicely";
505 The -oi option prevents "sendmail" from interpreting a line consisting
507 of a single dot as "end of message". The -t option says to use the
508 headers to decide who to send the message to, and -odq says to put the
509 message into the queue. This last option means your message won't be
510 immediately delivered, so leave it out if you want immediate delivery.
511 Alternate, less convenient approaches include calling "mail" (sometimes
513 called "mailx") directly or simply opening up port 25 have having an
514 intimate conversation between just you and the remote SMTP daemon,
515 probably "sendmail".
516 Or you might be able use the CPAN module "Mail::Mailer":
518 use Mail::Mailer;
520 $mailer = Mail::Mailer->new();
522 $mailer->open({ From => $from_address,
523 To => $to_address,
524 Subject => $subject,
525 })
526 or die "Can't open: $!\n";
527 print $mailer $body;
528 $mailer->close();
529 The "Mail::Internet" module uses "Net::SMTP" which is less Unix-centric
531 than "Mail::Mailer", but less reliable. Avoid raw SMTP commands.
532 There are many reasons to use a mail transport agent like "sendmail".
533 These include queuing, MX records, and security.
534 How do I use MIME to make an attachment to a mail message?
536 This answer is extracted directly from the "MIME::Lite" documentation.
537 Create a multipart message (i.e., one with attachments).
538 use MIME::Lite;
540 ### Create a new multipart message:
542 $msg = MIME::Lite->new(
543 From =>'me@myhost.com',
544 To =>'you@yourhost.com',
545 Cc =>'some@other.com, some@more.com',
546 Subject =>'A message with 2 parts...',
547 Type =>'multipart/mixed'
548 );
549 ### Add parts (each "attach" has same arguments as "new"):
551 $msg->attach(Type =>'TEXT',
552 Data =>"Here's the GIF file you wanted"
553 );
554 $msg->attach(Type =>'image/gif',
555 Path =>'aaa000123.gif',
556 Filename =>'logo.gif'
557 );
558 $text = $msg->as_string;
560 "MIME::Lite" also includes a method for sending these things.
562 $msg->send;
564 This defaults to using sendmail but can be customized to use SMTP via
566 Net::SMTP.
567 How do I read mail?
569 While you could use the "Mail::Folder" module from CPAN (part of the
570 "MailFolder" package) or the "Mail::Internet" module from CPAN (part of
571 the "MailTools" package), often a module is overkill. Here's a mail
572 sorter.
573 #!/usr/bin/perl
575 my(@msgs, @sub);
577 my $msgno = -1;
578 $/ = ''; # paragraph reads
579 while (<>) {
580 if (/^From /m) {
581 /^Subject:\s*(?:Re:\s*)*(.*)/mi;
582 $sub[++$msgno] = lc($1) || '';
583 }
584 $msgs[$msgno] .= $_;
585 }
586 for my $i (sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msgs)) {
587 print $msgs[$i];
588 }
589 Or more succinctly,
591 #!/usr/bin/perl -n00
593 # bysub2 - awkish sort-by-subject
594 BEGIN { $msgno = -1 }
595 $sub[++$msgno] = (/^Subject:\s*(?:Re:\s*)*(.*)/mi)[0] if /^From/m;
596 $msg[$msgno] .= $_;
597 END { print @msg[ sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msg) ] }
598 How do I find out my hostname, domainname, or IP address?
600 gethostbyname, Socket, Net::Domain, Sys::Hostname" (contributed by
601 brian d foy)
602 The "Net::Domain" module, which is part of the standard distribution
604 starting in perl5.7.3, can get you the fully qualified domain name
605 (FQDN), the host name, or the domain name.
606 use Net::Domain qw(hostname hostfqdn hostdomain);
608 my $host = hostfqdn();
610 The "Sys::Hostname" module, included in the standard distribution since
612 perl5.6, can also get the hostname.
613 use Sys::Hostname;
615 $host = hostname();
617 To get the IP address, you can use the "gethostbyname" built-in
619 function to turn the name into a number. To turn that number into the
620 dotted octet form (a.b.c.d) that most people expect, use the
621 "inet_ntoa" function from the "Socket" module, which also comes with
622 perl.
623 use Socket;
625 my $address = inet_ntoa(
627 scalar gethostbyname( $host || 'localhost' )
628 );
629 How do I fetch a news article or the active newsgroups?
631 Use the "Net::NNTP" or "News::NNTPClient" modules, both available from
632 CPAN. This can make tasks like fetching the newsgroup list as simple
633 as
634 perl -MNews::NNTPClient
636 -e 'print News::NNTPClient->new->list("newsgroups")'
637 How do I fetch/put an FTP file?
639 "LWP::Simple" (available from CPAN) can fetch but not put. "Net::FTP"
640 (also available from CPAN) is more complex but can put as well as
641 fetch.
642 How can I do RPC in Perl?
644 (Contributed by brian d foy)
645 Use one of the RPC modules you can find on CPAN (
647 http://search.cpan.org/search?query=RPC&mode=all ).
648
650 Copyright (c) 1997-2010 Tom Christiansen, Nathan Torkington, and other
651 authors as noted. All rights reserved.
652 This documentation is free; you can redistribute it and/or modify it
654 under the same terms as Perl itself.
655 Irrespective of its distribution, all code examples in this file are
657 hereby placed into the public domain. You are permitted and encouraged
658 to use this code in your own programs for fun or for profit as you see
659 fit. A simple comment in the code giving credit would be courteous but
660 is not required.
661perl v5.12.4 2011-06-07 PERLFAQ9(1)