1Locale::Maketext(3) User Contributed Perl Documentation Locale::Maketext(3)
2
6 Locale::Maketext - framework for localization
7
9 package MyProgram;
10 use strict;
11 use MyProgram::L10N;
12 # ...which inherits from Locale::Maketext
13 my $lh = MyProgram::L10N->get_handle() || die "What language?";
14 ...
15 # And then any messages your program emits, like:
16 warn $lh->maketext( "Can't open file [_1]: [_2]\n", $f, $! );
17 ...
18
20 It is a common feature of applications (whether run directly, or via
21 the Web) for them to be "localized" -- i.e., for them to a present an
22 English interface to an English-speaker, a German interface to a
23 German-speaker, and so on for all languages it's programmed with.
24 Locale::Maketext is a framework for software localization; it provides
25 you with the tools for organizing and accessing the bits of text and
26 text-processing code that you need for producing localized
27 applications.
28 In order to make sense of Maketext and how all its components fit
30 together, you should probably go read Locale::Maketext::TPJ13, and then
31 read the following documentation.
32 You may also want to read over the source for "File::Findgrep" and its
34 constituent modules -- they are a complete (if small) example
35 application that uses Maketext.
36
38 The basic design of Locale::Maketext is object-oriented, and
39 Locale::Maketext is an abstract base class, from which you derive a
40 "project class". The project class (with a name like
41 "TkBocciBall::Localize", which you then use in your module) is in turn
42 the base class for all the "language classes" for your project (with
43 names "TkBocciBall::Localize::it", "TkBocciBall::Localize::en",
44 "TkBocciBall::Localize::fr", etc.).
45 A language class is a class containing a lexicon of phrases as class
47 data, and possibly also some methods that are of use in interpreting
48 phrases in the lexicon, or otherwise dealing with text in that
49 language.
50 An object belonging to a language class is called a "language handle";
52 it's typically a flyweight object.
53 The normal course of action is to call:
55 use TkBocciBall::Localize; # the localization project class
57 $lh = TkBocciBall::Localize->get_handle();
58 # Depending on the user's locale, etc., this will
59 # make a language handle from among the classes available,
60 # and any defaults that you declare.
61 die "Couldn't make a language handle??" unless $lh;
62 From then on, you use the "maketext" function to access entries in
64 whatever lexicon(s) belong to the language handle you got. So, this:
65 print $lh->maketext("You won!"), "\n";
67 ...emits the right text for this language. If the object in $lh
69 belongs to class "TkBocciBall::Localize::fr" and
70 %TkBocciBall::Localize::fr::Lexicon contains "("You won!" => "Tu as
71 gagné!")", then the above code happily tells the user "Tu as gagné!".
72
74 Locale::Maketext offers a variety of methods, which fall into three
75 categories:
76 • Methods to do with constructing language handles.
78 • "maketext" and other methods to do with accessing %Lexicon data for
80 a given language handle.
81 • Methods that you may find it handy to use, from routines of yours
83 that you put in %Lexicon entries.
84 These are covered in the following section.
86 Construction Methods
88 These are to do with constructing a language handle:
89 • $lh = YourProjClass->get_handle( ...langtags... ) || die "lg-
91 handle?";
92 This tries loading classes based on the language-tags you give
94 (like "("en-US", "sk", "kon", "es-MX", "ja", "i-klingon")", and for
95 the first class that succeeds, returns
96 YourProjClass::language->new().
97 If it runs thru the entire given list of language-tags, and finds
99 no classes for those exact terms, it then tries "superordinate"
100 language classes. So if no "en-US" class (i.e.,
101 YourProjClass::en_us) was found, nor classes for anything else in
102 that list, we then try its superordinate, "en" (i.e.,
103 YourProjClass::en), and so on thru the other language-tags in the
104 given list: "es". (The other language-tags in our example list:
105 happen to have no superordinates.)
106 If none of those language-tags leads to loadable classes, we then
108 try classes derived from YourProjClass->fallback_languages() and
109 then if nothing comes of that, we use classes named by
110 YourProjClass->fallback_language_classes(). Then in the (probably
111 quite unlikely) event that that fails, we just return undef.
112 • $lh = YourProjClass->get_handle() || die "lg-handle?";
114 When "get_handle" is called with an empty parameter list, magic
116 happens:
117 If "get_handle" senses that it's running in program that was
119 invoked as a CGI, then it tries to get language-tags out of the
120 environment variable "HTTP_ACCEPT_LANGUAGE", and it pretends that
121 those were the languages passed as parameters to "get_handle".
122 Otherwise (i.e., if not a CGI), this tries various OS-specific ways
124 to get the language-tags for the current locale/language, and then
125 pretends that those were the value(s) passed to "get_handle".
126 Currently this OS-specific stuff consists of looking in the
128 environment variables "LANG" and "LANGUAGE"; and on MSWin machines
129 (where those variables are typically unused), this also tries using
130 the module Win32::Locale to get a language-tag for whatever
131 language/locale is currently selected in the "Regional Settings"
132 (or "International"?) Control Panel. I welcome further
133 suggestions for making this do the Right Thing under other
134 operating systems that support localization.
135 If you're using localization in an application that keeps a
137 configuration file, you might consider something like this in your
138 project class:
139 sub get_handle_via_config {
141 my $class = $_[0];
142 my $chosen_language = $Config_settings{'language'};
143 my $lh;
144 if($chosen_language) {
145 $lh = $class->get_handle($chosen_language)
146 || die "No language handle for \"$chosen_language\""
147 . " or the like";
148 } else {
149 # Config file missing, maybe?
150 $lh = $class->get_handle()
151 || die "Can't get a language handle";
152 }
153 return $lh;
154 }
155 • $lh = YourProjClass::langname->new();
157 This constructs a language handle. You usually don't call this
159 directly, but instead let "get_handle" find a language class to
160 "use" and to then call ->new on.
161 • $lh->init();
163 This is called by ->new to initialize newly-constructed language
165 handles. If you define an init method in your class, remember that
166 it's usually considered a good idea to call $lh->SUPER::init in it
167 (presumably at the beginning), so that all classes get a chance to
168 initialize a new object however they see fit.
169 • YourProjClass->fallback_languages()
171 "get_handle" appends the return value of this to the end of
173 whatever list of languages you pass "get_handle". Unless you
174 override this method, your project class will inherit
175 Locale::Maketext's "fallback_languages", which currently returns
176 "('i-default', 'en', 'en-US')". ("i-default" is defined in RFC
177 2277).
178 This method (by having it return the name of a language-tag that
180 has an existing language class) can be used for making sure that
181 "get_handle" will always manage to construct a language handle
182 (assuming your language classes are in an appropriate @INC
183 directory). Or you can use the next method:
184 • YourProjClass->fallback_language_classes()
186 "get_handle" appends the return value of this to the end of the
188 list of classes it will try using. Unless you override this
189 method, your project class will inherit Locale::Maketext's
190 "fallback_language_classes", which currently returns an empty list,
191 "()". By setting this to some value (namely, the name of a
192 loadable language class), you can be sure that "get_handle" will
193 always manage to construct a language handle.
194 The "maketext" Method
196 This is the most important method in Locale::Maketext:
197 $text = $lh->maketext(I<key>, ...parameters for this phrase...);
199 This looks in the %Lexicon of the language handle $lh and all its
201 superclasses, looking for an entry whose key is the string key.
202 Assuming such an entry is found, various things then happen, depending
203 on the value found:
204 If the value is a scalarref, the scalar is dereferenced and returned
206 (and any parameters are ignored).
207 If the value is a coderef, we return &$value($lh, ...parameters...).
209 If the value is a string that doesn't look like it's in Bracket
211 Notation, we return it (after replacing it with a scalarref, in its
212 %Lexicon).
213 If the value does look like it's in Bracket Notation, then we compile
215 it into a sub, replace the string in the %Lexicon with the new coderef,
216 and then we return &$new_sub($lh, ...parameters...).
217 Bracket Notation is discussed in a later section. Note that trying to
219 compile a string into Bracket Notation can throw an exception if the
220 string is not syntactically valid (say, by not balancing brackets
221 right.)
222 Also, calling &$coderef($lh, ...parameters...) can throw any sort of
224 exception (if, say, code in that sub tries to divide by zero). But a
225 very common exception occurs when you have Bracket Notation text that
226 says to call a method "foo", but there is no such method. (E.g., "You
227 have [quatn,_1,ball]." will throw an exception on trying to call
228 $lh->quatn($_[1],'ball') -- you presumably meant "quant".) "maketext"
229 catches these exceptions, but only to make the error message more
230 readable, at which point it rethrows the exception.
231 An exception may be thrown if key is not found in any of $lh's %Lexicon
233 hashes. What happens if a key is not found, is discussed in a later
234 section, "Controlling Lookup Failure".
235 Note that you might find it useful in some cases to override the
237 "maketext" method with an "after method", if you want to translate
238 encodings, or even scripts:
239 package YrProj::zh_cn; # Chinese with PRC-style glyphs
241 use base ('YrProj::zh_tw'); # Taiwan-style
242 sub maketext {
243 my $self = shift(@_);
244 my $value = $self->maketext(@_);
245 return Chineeze::taiwan2mainland($value);
246 }
247 Or you may want to override it with something that traps any
249 exceptions, if that's critical to your program:
250 sub maketext {
252 my($lh, @stuff) = @_;
253 my $out;
254 eval { $out = $lh->SUPER::maketext(@stuff) };
255 return $out unless $@;
256 ...otherwise deal with the exception...
257 }
258 Other than those two situations, I don't imagine that it's useful to
260 override the "maketext" method. (If you run into a situation where it
261 is useful, I'd be interested in hearing about it.)
262 $lh->fail_with or $lh->fail_with(PARAM)
264 $lh->failure_handler_auto
265 These two methods are discussed in the section "Controlling Lookup
266 Failure".
267 $lh->blacklist(@list)
269 $lh->whitelist(@list)
270 These methods are discussed in the section "Bracket Notation
271 Security".
272 Utility Methods
274 These are methods that you may find it handy to use, generally from
275 %Lexicon routines of yours (whether expressed as Bracket Notation or
276 not).
277 $language->quant($number, $singular)
279 $language->quant($number, $singular, $plural)
280 $language->quant($number, $singular, $plural, $negative)
281 This is generally meant to be called from inside Bracket Notation
282 (which is discussed later), as in
283 "Your search matched [quant,_1,document]!"
285 It's for quantifying a noun (i.e., saying how much of it there is,
287 while giving the correct form of it). The behavior of this method
288 is handy for English and a few other Western European languages,
289 and you should override it for languages where it's not suitable.
290 You can feel free to read the source, but the current
291 implementation is basically as this pseudocode describes:
292 if $number is 0 and there's a $negative,
294 return $negative;
295 elsif $number is 1,
296 return "1 $singular";
297 elsif there's a $plural,
298 return "$number $plural";
299 else
300 return "$number " . $singular . "s";
301 #
302 # ...except that we actually call numf to
303 # stringify $number before returning it.
304 So for English (with Bracket Notation) "...[quant,_1,file]..." is
306 fine (for 0 it returns "0 files", for 1 it returns "1 file", and
307 for more it returns "2 files", etc.)
308 But for "directory", you'd want "[quant,_1,directory,directories]"
310 so that our elementary "quant" method doesn't think that the plural
311 of "directory" is "directorys". And you might find that the output
312 may sound better if you specify a negative form, as in:
313 "[quant,_1,file,files,No files] matched your query.\n"
315 Remember to keep in mind verb agreement (or adjectives too, in
317 other languages), as in:
318 "[quant,_1,document] were matched.\n"
320 Because if _1 is one, you get "1 document were matched". An
322 acceptable hack here is to do something like this:
323 "[quant,_1,document was, documents were] matched.\n"
325 $language->numf($number)
327 This returns the given number formatted nicely according to this
328 language's conventions. Maketext's default method is mostly to
329 just take the normal string form of the number (applying sprintf
330 "%G" for only very large numbers), and then to add commas as
331 necessary. (Except that we apply "tr/,./.,/" if
332 $language->{'numf_comma'} is true; that's a bit of a hack that's
333 useful for languages that express two million as "2.000.000" and
334 not as "2,000,000").
335 If you want anything fancier, consider overriding this with
337 something that uses Number::Format, or does something else
338 entirely.
339 Note that numf is called by quant for stringifying all quantifying
341 numbers.
342 $language->numerate($number, $singular, $plural, $negative)
344 This returns the given noun form which is appropriate for the
345 quantity $number according to this language's conventions.
346 "numerate" is used internally by "quant" to quantify nouns. Use it
347 directly -- usually from bracket notation -- to avoid "quant"'s
348 implicit call to "numf" and output of a numeric quantity.
349 $language->sprintf($format, @items)
351 This is just a wrapper around Perl's normal "sprintf" function.
352 It's provided so that you can use "sprintf" in Bracket Notation:
353 "Couldn't access datanode [sprintf,%10x=~[%s~],_1,_2]!\n"
355 returning...
357 Couldn't access datanode Stuff=[thangamabob]!
359 $language->language_tag()
361 Currently this just takes the last bit of "ref($language)", turns
362 underscores to dashes, and returns it. So if $language is an
363 object of class Hee::HOO::Haw::en_us, $language->language_tag()
364 returns "en-us". (Yes, the usual representation for that language
365 tag is "en-US", but case is never considered meaningful in
366 language-tag comparison.)
367 You may override this as you like; Maketext doesn't use it for
369 anything.
370 $language->encoding()
372 Currently this isn't used for anything, but it's provided (with
373 default value of "(ref($language) && $language->{'encoding'})) or
374 "iso-8859-1"" ) as a sort of suggestion that it may be
375 useful/necessary to associate encodings with your language handles
376 (whether on a per-class or even per-handle basis.)
377 Language Handle Attributes and Internals
379 A language handle is a flyweight object -- i.e., it doesn't
380 (necessarily) carry any data of interest, other than just being a
381 member of whatever class it belongs to.
382 A language handle is implemented as a blessed hash. Subclasses of
384 yours can store whatever data you want in the hash. Currently the only
385 hash entry used by any crucial Maketext method is "fail", so feel free
386 to use anything else as you like.
387 Remember: Don't be afraid to read the Maketext source if there's any
389 point on which this documentation is unclear. This documentation is
390 vastly longer than the module source itself.
391
393 These are Locale::Maketext's assumptions about the class hierarchy
394 formed by all your language classes:
395 • You must have a project base class, which you load, and which you
397 then use as the first argument in the call to
398 YourProjClass->get_handle(...). It should derive (whether directly
399 or indirectly) from Locale::Maketext. It doesn't matter how you
400 name this class, although assuming this is the localization
401 component of your Super Mega Program, good names for your project
402 class might be SuperMegaProgram::Localization,
403 SuperMegaProgram::L10N, SuperMegaProgram::I18N,
404 SuperMegaProgram::International, or even
405 SuperMegaProgram::Languages or SuperMegaProgram::Messages.
406 • Language classes are what YourProjClass->get_handle will try to
408 load. It will look for them by taking each language-tag (skipping
409 it if it doesn't look like a language-tag or locale-tag!), turning
410 it to all lowercase, turning dashes to underscores, and appending
411 it to YourProjClass . "::". So this:
412 $lh = YourProjClass->get_handle(
414 'en-US', 'fr', 'kon', 'i-klingon', 'i-klingon-romanized'
415 );
416 will try loading the classes YourProjClass::en_us (note
418 lowercase!), YourProjClass::fr, YourProjClass::kon,
419 YourProjClass::i_klingon and YourProjClass::i_klingon_romanized.
420 (And it'll stop at the first one that actually loads.)
421 • I assume that each language class derives (directly or indirectly)
423 from your project class, and also defines its @ISA, its %Lexicon,
424 or both. But I anticipate no dire consequences if these
425 assumptions do not hold.
426 • Language classes may derive from other language classes (although
428 they should have "use Thatclassname" or "use base
429 qw(...classes...)"). They may derive from the project class. They
430 may derive from some other class altogether. Or via multiple
431 inheritance, it may derive from any mixture of these.
432 • I foresee no problems with having multiple inheritance in your
434 hierarchy of language classes. (As usual, however, Perl will
435 complain bitterly if you have a cycle in the hierarchy: i.e., if
436 any class is its own ancestor.)
437
439 A typical %Lexicon entry is meant to signify a phrase, taking some
440 number (0 or more) of parameters. An entry is meant to be accessed by
441 via a string key in $lh->maketext(key, ...parameters...), which should
442 return a string that is generally meant for be used for "output" to the
443 user -- regardless of whether this actually means printing to STDOUT,
444 writing to a file, or putting into a GUI widget.
445 While the key must be a string value (since that's a basic restriction
447 that Perl places on hash keys), the value in the lexicon can currently
448 be of several types: a defined scalar, scalarref, or coderef. The use
449 of these is explained above, in the section 'The "maketext" Method',
450 and Bracket Notation for strings is discussed in the next section.
451 While you can use arbitrary unique IDs for lexicon keys (like
453 "_min_larger_max_error"), it is often useful for if an entry's key is
454 itself a valid value, like this example error message:
455 "Minimum ([_1]) is larger than maximum ([_2])!\n",
457 Compare this code that uses an arbitrary ID...
459 die $lh->maketext( "_min_larger_max_error", $min, $max )
461 if $min > $max;
462 ...to this code that uses a key-as-value:
464 die $lh->maketext(
466 "Minimum ([_1]) is larger than maximum ([_2])!\n",
467 $min, $max
468 ) if $min > $max;
469 The second is, in short, more readable. In particular, it's obvious
471 that the number of parameters you're feeding to that phrase (two) is
472 the number of parameters that it wants to be fed. (Since you see _1
473 and a _2 being used in the key there.)
474 Also, once a project is otherwise complete and you start to localize
476 it, you can scrape together all the various keys you use, and pass it
477 to a translator; and then the translator's work will go faster if what
478 he's presented is this:
479 "Minimum ([_1]) is larger than maximum ([_2])!\n",
481 => "", # fill in something here, Jacques!
482 rather than this more cryptic mess:
484 "_min_larger_max_error"
486 => "", # fill in something here, Jacques
487 I think that keys as lexicon values makes the completed lexicon entries
489 more readable:
490 "Minimum ([_1]) is larger than maximum ([_2])!\n",
492 => "Le minimum ([_1]) est plus grand que le maximum ([_2])!\n",
493 Also, having valid values as keys becomes very useful if you set up an
495 _AUTO lexicon. _AUTO lexicons are discussed in a later section.
496 I almost always use keys that are themselves valid lexicon values. One
498 notable exception is when the value is quite long. For example, to get
499 the screenful of data that a command-line program might return when
500 given an unknown switch, I often just use a brief, self-explanatory key
501 such as "_USAGE_MESSAGE". At that point I then go and immediately to
502 define that lexicon entry in the ProjectClass::L10N::en lexicon (since
503 English is always my "project language"):
504 '_USAGE_MESSAGE' => <<'EOSTUFF',
506 ...long long message...
507 EOSTUFF
508 and then I can use it as:
510 getopt('oDI', \%opts) or die $lh->maketext('_USAGE_MESSAGE');
512 Incidentally, note that each class's %Lexicon inherits-and-extends the
514 lexicons in its superclasses. This is not because these are special
515 hashes per se, but because you access them via the "maketext" method,
516 which looks for entries across all the %Lexicon hashes in a language
517 class and all its ancestor classes. (This is because the idea of
518 "class data" isn't directly implemented in Perl, but is instead left to
519 individual class-systems to implement as they see fit..)
520 Note that you may have things stored in a lexicon besides just phrases
522 for output: for example, if your program takes input from the
523 keyboard, asking a "(Y/N)" question, you probably need to know what the
524 equivalent of "Y[es]/N[o]" is in whatever language. You probably also
525 need to know what the equivalents of the answers "y" and "n" are. You
526 can store that information in the lexicon (say, under the keys
527 "~answer_y" and "~answer_n", and the long forms as "~answer_yes" and
528 "~answer_no", where "~" is just an ad-hoc character meant to indicate
529 to programmers/translators that these are not phrases for output).
530 Or instead of storing this in the language class's lexicon, you can
532 (and, in some cases, really should) represent the same bit of knowledge
533 as code in a method in the language class. (That leaves a tidy
534 distinction between the lexicon as the things we know how to say, and
535 the rest of the things in the lexicon class as things that we know how
536 to do.) Consider this example of a processor for responses to French
537 "oui/non" questions:
538 sub y_or_n {
540 return undef unless defined $_[1] and length $_[1];
541 my $answer = lc $_[1]; # smash case
542 return 1 if $answer eq 'o' or $answer eq 'oui';
543 return 0 if $answer eq 'n' or $answer eq 'non';
544 return undef;
545 }
546 ...which you'd then call in a construct like this:
548 my $response;
550 until(defined $response) {
551 print $lh->maketext("Open the pod bay door (y/n)? ");
552 $response = $lh->y_or_n( get_input_from_keyboard_somehow() );
553 }
554 if($response) { $pod_bay_door->open() }
555 else { $pod_bay_door->leave_closed() }
556 Other data worth storing in a lexicon might be things like filenames
558 for language-targetted resources:
559 ...
561 "_main_splash_png"
562 => "/styles/en_us/main_splash.png",
563 "_main_splash_imagemap"
564 => "/styles/en_us/main_splash.incl",
565 "_general_graphics_path"
566 => "/styles/en_us/",
567 "_alert_sound"
568 => "/styles/en_us/hey_there.wav",
569 "_forward_icon"
570 => "left_arrow.png",
571 "_backward_icon"
572 => "right_arrow.png",
573 # In some other languages, left equals
574 # BACKwards, and right is FOREwards.
575 ...
576 You might want to do the same thing for expressing key bindings or the
578 like (since hardwiring "q" as the binding for the function that quits a
579 screen/menu/program is useful only if your language happens to
580 associate "q" with "quit"!)
581
583 Bracket Notation is a crucial feature of Locale::Maketext. I mean
584 Bracket Notation to provide a replacement for the use of sprintf
585 formatting. Everything you do with Bracket Notation could be done with
586 a sub block, but bracket notation is meant to be much more concise.
587 Bracket Notation is a like a miniature "template" system (in the sense
589 of Text::Template, not in the sense of C++ templates), where normal
590 text is passed thru basically as is, but text in special regions is
591 specially interpreted. In Bracket Notation, you use square brackets
592 ("[...]"), not curly braces ("{...}") to note sections that are
593 specially interpreted.
594 For example, here all the areas that are taken literally are underlined
596 with a "^", and all the in-bracket special regions are underlined with
597 an X:
598 "Minimum ([_1]) is larger than maximum ([_2])!\n",
600 ^^^^^^^^^ XX ^^^^^^^^^^^^^^^^^^^^^^^^^^ XX ^^^^
601 When that string is compiled from bracket notation into a real Perl
603 sub, it's basically turned into:
604 sub {
606 my $lh = $_[0];
607 my @params = @_;
608 return join '',
609 "Minimum (",
610 ...some code here...
611 ") is larger than maximum (",
612 ...some code here...
613 ")!\n",
614 }
615 # to be called by $lh->maketext(KEY, params...)
616 In other words, text outside bracket groups is turned into string
618 literals. Text in brackets is rather more complex, and currently
619 follows these rules:
620 • Bracket groups that are empty, or which consist only of whitespace,
622 are ignored. (Examples: "[]", "[ ]", or a [ and a ] with
623 returns and/or tabs and/or spaces between them.
624 Otherwise, each group is taken to be a comma-separated group of
626 items, and each item is interpreted as follows:
627 • An item that is "_digits" or "_-digits" is interpreted as
629 $_[value]. I.e., "_1" becomes with $_[1], and "_-3" is interpreted
630 as $_[-3] (in which case @_ should have at least three elements in
631 it). Note that $_[0] is the language handle, and is typically not
632 named directly.
633 • An item "_*" is interpreted to mean "all of @_ except $_[0]".
635 I.e., @_[1..$#_]. Note that this is an empty list in the case of
636 calls like $lh->maketext(key) where there are no parameters (except
637 $_[0], the language handle).
638 • Otherwise, each item is interpreted as a string literal.
640 The group as a whole is interpreted as follows:
642 • If the first item in a bracket group looks like a method name, then
644 that group is interpreted like this:
645 $lh->that_method_name(
647 ...rest of items in this group...
648 ),
649 • If the first item in a bracket group is "*", it's taken as
651 shorthand for the so commonly called "quant" method. Similarly, if
652 the first item in a bracket group is "#", it's taken to be
653 shorthand for "numf".
654 • If the first item in a bracket group is the empty-string, or "_*"
656 or "_digits" or "_-digits", then that group is interpreted as just
657 the interpolation of all its items:
658 join('',
660 ...rest of items in this group...
661 ),
662 Examples: "[_1]" and "[,_1]", which are synonymous; and
664 ""[,ID-(,_4,-,_2,)]"", which compiles as "join "", "ID-(", $_[4],
665 "-", $_[2], ")"".
666 • Otherwise this bracket group is invalid. For example, in the group
668 "[!@#,whatever]", the first item "!@#" is neither the empty-string,
669 "_number", "_-number", "_*", nor a valid method name; and so
670 Locale::Maketext will throw an exception of you try compiling an
671 expression containing this bracket group.
672 Note, incidentally, that items in each group are comma-separated, not
674 "/\s*,\s*/"-separated. That is, you might expect that this bracket
675 group:
676 "Hoohah [foo, _1 , bar ,baz]!"
678 would compile to this:
680 sub {
682 my $lh = $_[0];
683 return join '',
684 "Hoohah ",
685 $lh->foo( $_[1], "bar", "baz"),
686 "!",
687 }
688 But it actually compiles as this:
690 sub {
692 my $lh = $_[0];
693 return join '',
694 "Hoohah ",
695 $lh->foo(" _1 ", " bar ", "baz"), # note the <space> in " bar "
696 "!",
697 }
698 In the notation discussed so far, the characters "[" and "]" are given
700 special meaning, for opening and closing bracket groups, and "," has a
701 special meaning inside bracket groups, where it separates items in the
702 group. This begs the question of how you'd express a literal "[" or
703 "]" in a Bracket Notation string, and how you'd express a literal comma
704 inside a bracket group. For this purpose I've adopted "~" (tilde) as
705 an escape character: "~[" means a literal '[' character anywhere in
706 Bracket Notation (i.e., regardless of whether you're in a bracket group
707 or not), and ditto for "~]" meaning a literal ']', and "~," meaning a
708 literal comma. (Altho "," means a literal comma outside of bracket
709 groups -- it's only inside bracket groups that commas are special.)
710 And on the off chance you need a literal tilde in a bracket expression,
712 you get it with "~~".
713 Currently, an unescaped "~" before a character other than a bracket or
715 a comma is taken to mean just a "~" and that character. I.e., "~X"
716 means the same as "~~X" -- i.e., one literal tilde, and then one
717 literal "X". However, by using "~X", you are assuming that no future
718 version of Maketext will use "~X" as a magic escape sequence. In
719 practice this is not a great problem, since first off you can just
720 write "~~X" and not worry about it; second off, I doubt I'll add lots
721 of new magic characters to bracket notation; and third off, you aren't
722 likely to want literal "~" characters in your messages anyway, since
723 it's not a character with wide use in natural language text.
724 Brackets must be balanced -- every openbracket must have one matching
726 closebracket, and vice versa. So these are all invalid:
727 "I ate [quant,_1,rhubarb pie."
729 "I ate [quant,_1,rhubarb pie[."
730 "I ate quant,_1,rhubarb pie]."
731 "I ate quant,_1,rhubarb pie[."
732 Currently, bracket groups do not nest. That is, you cannot say:
734 "Foo [bar,baz,[quux,quuux]]\n";
736 If you need a notation that's that powerful, use normal Perl:
738 %Lexicon = (
740 ...
741 "some_key" => sub {
742 my $lh = $_[0];
743 join '',
744 "Foo ",
745 $lh->bar('baz', $lh->quux('quuux')),
746 "\n",
747 },
748 ...
749 );
750 Or write the "bar" method so you don't need to pass it the output from
752 calling quux.
753 I do not anticipate that you will need (or particularly want) to nest
755 bracket groups, but you are welcome to email me with convincing (real-
756 life) arguments to the contrary.
757
759 Locale::Maketext does not use any special syntax to differentiate
760 bracket notation methods from normal class or object methods. This
761 design makes it vulnerable to format string attacks whenever it is used
762 to process strings provided by untrusted users.
763 Locale::Maketext does support blacklist and whitelist functionality to
765 limit which methods may be called as bracket notation methods.
766 By default, Locale::Maketext blacklists all methods in the
768 Locale::Maketext namespace that begin with the '_' character, and all
769 methods which include Perl's namespace separator characters.
770 The default blacklist for Locale::Maketext also prevents use of the
772 following methods in bracket notation:
773 blacklist
775 encoding
776 fail_with
777 failure_handler_auto
778 fallback_language_classes
779 fallback_languages
780 get_handle
781 init
782 language_tag
783 maketext
784 new
785 whitelist
786 This list can be extended by either blacklisting additional "known bad"
788 methods, or whitelisting only "known good" methods.
789 To prevent specific methods from being called in bracket notation, use
791 the blacklist() method:
792 my $lh = MyProgram::L10N->get_handle();
794 $lh->blacklist(qw{my_internal_method my_other_method});
795 $lh->maketext('[my_internal_method]'); # dies
796 To limit the allowed bracked notation methods to a specific list, use
798 the whitelist() method:
799 my $lh = MyProgram::L10N->get_handle();
801 $lh->whitelist('numerate', 'numf');
802 $lh->maketext('[_1] [numerate, _1,shoe,shoes]', 12); # works
803 $lh->maketext('[my_internal_method]'); # dies
804 The blacklist() and whitelist() methods extend their internal lists
806 whenever they are called. To reset the blacklist or whitelist, create a
807 new maketext object.
808 my $lh = MyProgram::L10N->get_handle();
810 $lh->blacklist('numerate');
811 $lh->blacklist('numf');
812 $lh->maketext('[_1] [numerate,_1,shoe,shoes]', 12); # dies
813 For lexicons that use an internal cache, translations which have
815 already been cached in their compiled form are not affected by
816 subsequent changes to the whitelist or blacklist settings. Lexicons
817 that use an external cache will have their cache cleared whenever the
818 whitelist of blacklist setings change. The difference between the two
819 types of caching is explained in the "Readonly Lexicons" section.
820 Methods disallowed by the blacklist cannot be permitted by the
822 whitelist.
823
825 If maketext goes to look in an individual %Lexicon for an entry for key
826 (where key does not start with an underscore), and sees none, but does
827 see an entry of "_AUTO" => some_true_value, then we actually define
828 $Lexicon{key} = key right then and there, and then use that value as if
829 it had been there all along. This happens before we even look in any
830 superclass %Lexicons!
831 (This is meant to be somewhat like the AUTOLOAD mechanism in Perl's
833 function call system -- or, looked at another way, like the AutoLoader
834 module.)
835 I can picture all sorts of circumstances where you just do not want
837 lookup to be able to fail (since failing normally means that maketext
838 throws a "die", although see the next section for greater control over
839 that). But here's one circumstance where _AUTO lexicons are meant to
840 be especially useful:
841 As you're writing an application, you decide as you go what messages
843 you need to emit. Normally you'd go to write this:
844 if(-e $filename) {
846 go_process_file($filename)
847 } else {
848 print qq{Couldn't find file "$filename"!\n};
849 }
850 but since you anticipate localizing this, you write:
852 use ThisProject::I18N;
854 my $lh = ThisProject::I18N->get_handle();
855 # For the moment, assume that things are set up so
856 # that we load class ThisProject::I18N::en
857 # and that that's the class that $lh belongs to.
858 ...
859 if(-e $filename) {
860 go_process_file($filename)
861 } else {
862 print $lh->maketext(
863 qq{Couldn't find file "[_1]"!\n}, $filename
864 );
865 }
866 Now, right after you've just written the above lines, you'd normally
868 have to go open the file ThisProject/I18N/en.pm, and immediately add an
869 entry:
870 "Couldn't find file \"[_1]\"!\n"
872 => "Couldn't find file \"[_1]\"!\n",
873 But I consider that somewhat of a distraction from the work of getting
875 the main code working -- to say nothing of the fact that I often have
876 to play with the program a few times before I can decide exactly what
877 wording I want in the messages (which in this case would require me to
878 go changing three lines of code: the call to maketext with that key,
879 and then the two lines in ThisProject/I18N/en.pm).
880 However, if you set "_AUTO => 1" in the %Lexicon in,
882 ThisProject/I18N/en.pm (assuming that English (en) is the language that
883 all your programmers will be using for this project's internal message
884 keys), then you don't ever have to go adding lines like this
885 "Couldn't find file \"[_1]\"!\n"
887 => "Couldn't find file \"[_1]\"!\n",
888 to ThisProject/I18N/en.pm, because if _AUTO is true there, then just
890 looking for an entry with the key "Couldn't find file \"[_1]\"!\n" in
891 that lexicon will cause it to be added, with that value!
892 Note that the reason that keys that start with "_" are immune to _AUTO
894 isn't anything generally magical about the underscore character -- I
895 just wanted a way to have most lexicon keys be autoable, except for
896 possibly a few, and I arbitrarily decided to use a leading underscore
897 as a signal to distinguish those few.
898
900 If your lexicon is a tied hash the simple act of caching the compiled
901 value can be fatal.
902 For example a GDBM_File GDBM_READER tied hash will die with something
904 like:
905 gdbm store returned -1, errno 2, key "..." at ...
907 All you need to do is turn on caching outside of the lexicon hash
909 itself like so:
910 sub init {
912 my ($lh) = @_;
913 ...
914 $lh->{'use_external_lex_cache'} = 1;
915 ...
916 }
917 And then instead of storing the compiled value in the lexicon hash it
919 will store it in $lh->{'_external_lex_cache'}
920
922 If you call $lh->maketext(key, ...parameters...), and there's no entry
923 key in $lh's class's %Lexicon, nor in the superclass %Lexicon hash, and
924 if we can't auto-make key (because either it starts with a "_", or
925 because none of its lexicons have "_AUTO => 1,"), then we have failed
926 to find a normal way to maketext key. What then happens in these
927 failure conditions, depends on the $lh object's "fail" attribute.
928 If the language handle has no "fail" attribute, maketext will simply
930 throw an exception (i.e., it calls "die", mentioning the key whose
931 lookup failed, and naming the line number where the calling
932 $lh->maketext(key,...) was.
933 If the language handle has a "fail" attribute whose value is a coderef,
935 then $lh->maketext(key,...params...) gives up and calls:
936 return $that_subref->($lh, $key, @params);
938 Otherwise, the "fail" attribute's value should be a string denoting a
940 method name, so that $lh->maketext(key,...params...) can give up with:
941 return $lh->$that_method_name($phrase, @params);
943 The "fail" attribute can be accessed with the "fail_with" method:
945 # Set to a coderef:
947 $lh->fail_with( \&failure_handler );
948 # Set to a method name:
950 $lh->fail_with( 'failure_method' );
951 # Set to nothing (i.e., so failure throws a plain exception)
953 $lh->fail_with( undef );
954 # Get the current value
956 $handler = $lh->fail_with();
957 Now, as to what you may want to do with these handlers: Maybe you'd
959 want to log what key failed for what class, and then die. Maybe you
960 don't like "die" and instead you want to send the error message to
961 STDOUT (or wherever) and then merely "exit()".
962 Or maybe you don't want to "die" at all! Maybe you could use a handler
964 like this:
965 # Make all lookups fall back onto an English value,
967 # but only after we log it for later fingerpointing.
968 my $lh_backup = ThisProject->get_handle('en');
969 open(LEX_FAIL_LOG, ">>wherever/lex.log") || die "GNAARGH $!";
970 sub lex_fail {
971 my($failing_lh, $key, $params) = @_;
972 print LEX_FAIL_LOG scalar(localtime), "\t",
973 ref($failing_lh), "\t", $key, "\n";
974 return $lh_backup->maketext($key,@params);
975 }
976 Some users have expressed that they think this whole mechanism of
978 having a "fail" attribute at all, seems a rather pointless
979 complication. But I want Locale::Maketext to be usable for software
980 projects of any scale and type; and different software projects have
981 different ideas of what the right thing is to do in failure conditions.
982 I could simply say that failure always throws an exception, and that if
983 you want to be careful, you'll just have to wrap every call to
984 $lh->maketext in an eval { }. However, I want programmers to reserve
985 the right (via the "fail" attribute) to treat lookup failure as
986 something other than an exception of the same level of severity as a
987 config file being unreadable, or some essential resource being
988 inaccessible.
989 One possibly useful value for the "fail" attribute is the method name
991 "failure_handler_auto". This is a method defined in the class
992 Locale::Maketext itself. You set it with:
993 $lh->fail_with('failure_handler_auto');
995 Then when you call $lh->maketext(key, ...parameters...) and there's no
997 key in any of those lexicons, maketext gives up with
998 return $lh->failure_handler_auto($key, @params);
1000 But failure_handler_auto, instead of dying or anything, compiles $key,
1002 caching it in
1003 $lh->{'failure_lex'}{$key} = $compiled
1005 and then calls the compiled value, and returns that. (I.e., if $key
1007 looks like bracket notation, $compiled is a sub, and we return
1008 &{$compiled}(@params); but if $key is just a plain string, we just
1009 return that.)
1010 The effect of using "failure_auto_handler" is like an AUTO lexicon,
1012 except that it 1) compiles $key even if it starts with "_", and 2) you
1013 have a record in the new hashref $lh->{'failure_lex'} of all the keys
1014 that have failed for this object. This should avoid your program dying
1015 -- as long as your keys aren't actually invalid as bracket code, and as
1016 long as they don't try calling methods that don't exist.
1017 "failure_auto_handler" may not be exactly what you want, but I hope it
1019 at least shows you that maketext failure can be mitigated in any number
1020 of very flexible ways. If you can formalize exactly what you want, you
1021 should be able to express that as a failure handler. You can even make
1022 it default for every object of a given class, by setting it in that
1023 class's init:
1024 sub init {
1026 my $lh = $_[0]; # a newborn handle
1027 $lh->SUPER::init();
1028 $lh->fail_with('my_clever_failure_handler');
1029 return;
1030 }
1031 sub my_clever_failure_handler {
1032 ...you clever things here...
1033 }
1034
1036 Here is a brief checklist on how to use Maketext to localize
1037 applications:
1038 • Decide what system you'll use for lexicon keys. If you insist, you
1040 can use opaque IDs (if you're nostalgic for "catgets"), but I have
1041 better suggestions in the section "Entries in Each Lexicon", above.
1042 Assuming you opt for meaningful keys that double as values (like
1043 "Minimum ([_1]) is larger than maximum ([_2])!\n"), you'll have to
1044 settle on what language those should be in. For the sake of
1045 argument, I'll call this English, specifically American English,
1046 "en-US".
1047 • Create a class for your localization project. This is the name of
1049 the class that you'll use in the idiom:
1050 use Projname::L10N;
1052 my $lh = Projname::L10N->get_handle(...) || die "Language?";
1053 Assuming you call your class Projname::L10N, create a class
1055 consisting minimally of:
1056 package Projname::L10N;
1058 use base qw(Locale::Maketext);
1059 ...any methods you might want all your languages to share...
1060 # And, assuming you want the base class to be an _AUTO lexicon,
1062 # as is discussed a few sections up:
1063 1;
1065 • Create a class for the language your internal keys are in. Name
1067 the class after the language-tag for that language, in lowercase,
1068 with dashes changed to underscores. Assuming your project's first
1069 language is US English, you should call this Projname::L10N::en_us.
1070 It should consist minimally of:
1071 package Projname::L10N::en_us;
1073 use base qw(Projname::L10N);
1074 %Lexicon = (
1075 '_AUTO' => 1,
1076 );
1077 1;
1078 (For the rest of this section, I'll assume that this "first
1080 language class" of Projname::L10N::en_us has _AUTO lexicon.)
1081 • Go and write your program. Everywhere in your program where you
1083 would say:
1084 print "Foobar $thing stuff\n";
1086 instead do it thru maketext, using no variable interpolation in the
1088 key:
1089 print $lh->maketext("Foobar [_1] stuff\n", $thing);
1091 If you get tired of constantly saying "print $lh->maketext",
1093 consider making a functional wrapper for it, like so:
1094 use Projname::L10N;
1096 our $lh;
1097 $lh = Projname::L10N->get_handle(...) || die "Language?";
1098 sub pmt (@) { print( $lh->maketext(@_)) }
1099 # "pmt" is short for "Print MakeText"
1100 $Carp::Verbose = 1;
1101 # so if maketext fails, we see made the call to pmt
1102 Besides whole phrases meant for output, anything language-dependent
1104 should be put into the class Projname::L10N::en_us, whether as
1105 methods, or as lexicon entries -- this is discussed in the section
1106 "Entries in Each Lexicon", above.
1107 • Once the program is otherwise done, and once its localization for
1109 the first language works right (via the data and methods in
1110 Projname::L10N::en_us), you can get together the data for
1111 translation. If your first language lexicon isn't an _AUTO
1112 lexicon, then you already have all the messages explicitly in the
1113 lexicon (or else you'd be getting exceptions thrown when you call
1114 $lh->maketext to get messages that aren't in there). But if you
1115 were (advisedly) lazy and are using an _AUTO lexicon, then you've
1116 got to make a list of all the phrases that you've so far been
1117 letting _AUTO generate for you. There are very many ways to
1118 assemble such a list. The most straightforward is to simply grep
1119 the source for every occurrence of "maketext" (or calls to wrappers
1120 around it, like the above "pmt" function), and to log the following
1121 phrase.
1122 • You may at this point want to consider whether your base class
1124 (Projname::L10N), from which all lexicons inherit from
1125 (Projname::L10N::en, Projname::L10N::es, etc.), should be an _AUTO
1126 lexicon. It may be true that in theory, all needed messages will
1127 be in each language class; but in the presumably unlikely or
1128 "impossible" case of lookup failure, you should consider whether
1129 your program should throw an exception, emit text in English (or
1130 whatever your project's first language is), or some more complex
1131 solution as described in the section "Controlling Lookup Failure",
1132 above.
1133 • Submit all messages/phrases/etc. to translators.
1135 (You may, in fact, want to start with localizing to one other
1137 language at first, if you're not sure that you've properly
1138 abstracted the language-dependent parts of your code.)
1139 Translators may request clarification of the situation in which a
1141 particular phrase is found. For example, in English we are
1142 entirely happy saying "n files found", regardless of whether we
1143 mean "I looked for files, and found n of them" or the rather
1144 distinct situation of "I looked for something else (like lines in
1145 files), and along the way I saw n files." This may involve
1146 rethinking things that you thought quite clear: should "Edit" on a
1147 toolbar be a noun ("editing") or a verb ("to edit")? Is there
1148 already a conventionalized way to express that menu option,
1149 separate from the target language's normal word for "to edit"?
1150 In all cases where the very common phenomenon of quantification
1152 (saying "N files", for any value of N) is involved, each translator
1153 should make clear what dependencies the number causes in the
1154 sentence. In many cases, dependency is limited to words adjacent
1155 to the number, in places where you might expect them ("I found
1156 the-?PLURAL N empty-?PLURAL directory-?PLURAL"), but in some cases
1157 there are unexpected dependencies ("I found-?PLURAL ..."!) as well
1158 as long-distance dependencies "The N directory-?PLURAL could not be
1159 deleted-?PLURAL"!).
1160 Remind the translators to consider the case where N is 0: "0 files
1162 found" isn't exactly natural-sounding in any language, but it may
1163 be unacceptable in many -- or it may condition special kinds of
1164 agreement (similar to English "I didN'T find ANY files").
1165 Remember to ask your translators about numeral formatting in their
1167 language, so that you can override the "numf" method as
1168 appropriate. Typical variables in number formatting are: what to
1169 use as a decimal point (comma? period?); what to use as a thousands
1170 separator (space? nonbreaking space? comma? period? small middot?
1171 prime? apostrophe?); and even whether the so-called "thousands
1172 separator" is actually for every third digit -- I've heard reports
1173 of two hundred thousand being expressible as "2,00,000" for some
1174 Indian (Subcontinental) languages, besides the less surprising
1175 "200 000", "200.000", "200,000", and "200'000". Also, using a set
1176 of numeral glyphs other than the usual ASCII "0"-"9" might be
1177 appreciated, as via "tr/0-9/\x{0966}-\x{096F}/" for getting digits
1178 in Devanagari script (for Hindi, Konkani, others).
1179 The basic "quant" method that Locale::Maketext provides should be
1181 good for many languages. For some languages, it might be useful to
1182 modify it (or its constituent "numerate" method) to take a plural
1183 form in the two-argument call to "quant" (as in "[quant,_1,files]")
1184 if it's all-around easier to infer the singular form from the
1185 plural, than to infer the plural form from the singular.
1186 But for other languages (as is discussed at length in
1188 Locale::Maketext::TPJ13), simple "quant"/"numf" is not enough. For
1189 the particularly problematic Slavic languages, what you may need is
1190 a method which you provide with the number, the citation form of
1191 the noun to quantify, and the case and gender that the sentence's
1192 syntax projects onto that noun slot. The method would then be
1193 responsible for determining what grammatical number that numeral
1194 projects onto its noun phrase, and what case and gender it may
1195 override the normal case and gender with; and then it would look up
1196 the noun in a lexicon providing all needed inflected forms.
1197 • You may also wish to discuss with the translators the question of
1199 how to relate different subforms of the same language tag,
1200 considering how this reacts with "get_handle"'s treatment of these.
1201 For example, if a user accepts interfaces in "en, fr", and you have
1202 interfaces available in "en-US" and "fr", what should they get?
1203 You may wish to resolve this by establishing that "en" and "en-US"
1204 are effectively synonymous, by having one class zero-derive from
1205 the other.
1206 For some languages this issue may never come up (Danish is rarely
1208 expressed as "da-DK", but instead is just "da"). And for other
1209 languages, the whole concept of a "generic" form may verge on being
1210 uselessly vague, particularly for interfaces involving voice media
1211 in forms of Arabic or Chinese.
1212 • Once you've localized your program/site/etc. for all desired
1214 languages, be sure to show the result (whether live, or via
1215 screenshots) to the translators. Once they approve, make every
1216 effort to have it then checked by at least one other speaker of
1217 that language. This holds true even when (or especially when) the
1218 translation is done by one of your own programmers. Some kinds of
1219 systems may be harder to find testers for than others, depending on
1220 the amount of domain-specific jargon and concepts involved -- it's
1221 easier to find people who can tell you whether they approve of your
1222 translation for "delete this message" in an email-via-Web
1223 interface, than to find people who can give you an informed opinion
1224 on your translation for "attribute value" in an XML query tool's
1225 interface.
1226
1228 I recommend reading all of these:
1229 Locale::Maketext::TPJ13 -- my The Perl Journal article about Maketext.
1231 It explains many important concepts underlying Locale::Maketext's
1232 design, and some insight into why Maketext is better than the plain old
1233 approach of having message catalogs that are just databases of sprintf
1234 formats.
1235 File::Findgrep is a sample application/module that uses
1237 Locale::Maketext to localize its messages. For a larger
1238 internationalized system, see also Apache::MP3.
1239 I18N::LangTags.
1241 Win32::Locale.
1243 RFC 3066, Tags for the Identification of Languages, as at
1245 http://sunsite.dk/RFC/rfc/rfc3066.html
1246 RFC 2277, IETF Policy on Character Sets and Languages is at
1248 http://sunsite.dk/RFC/rfc/rfc2277.html -- much of it is just things of
1249 interest to protocol designers, but it explains some basic concepts,
1250 like the distinction between locales and language-tags.
1251 The manual for GNU "gettext". The gettext dist is available in
1253 "ftp://prep.ai.mit.edu/pub/gnu/" -- get a recent gettext tarball and
1254 look in its "doc/" directory, there's an easily browsable HTML version
1255 in there. The gettext documentation asks lots of questions worth
1256 thinking about, even if some of their answers are sometimes wonky,
1257 particularly where they start talking about pluralization.
1258 The Locale/Maketext.pm source. Observe that the module is much shorter
1260 than its documentation!
1261
1263 Copyright (c) 1999-2004 Sean M. Burke. All rights reserved.
1264 This library is free software; you can redistribute it and/or modify it
1266 under the same terms as Perl itself.
1267 This program is distributed in the hope that it will be useful, but
1269 without any warranty; without even the implied warranty of
1270 merchantability or fitness for a particular purpose.
1271
1273 Sean M. Burke "sburke@cpan.org"
1274perl v5.34.0 2021-07-22 Locale::Maketext(3)