source: trunk/third/perl/pod/perllocale.pod @ 17035

Revision 17035, 38.6 KB checked in by zacheiss, 23 years ago (diff)
This commit was generated by cvs2svn to compensate for changes in r17034, which included commits to RCS files with non-trunk default branches.
Line 
1=head1 NAME
2
3perllocale - Perl locale handling (internationalization and localization)
4
5=head1 DESCRIPTION
6
7Perl supports language-specific notions of data such as "is this
8a letter", "what is the uppercase equivalent of this letter", and
9"which of these letters comes first".  These are important issues,
10especially for languages other than English--but also for English: it
11would be naE<iuml>ve to imagine that C<A-Za-z> defines all the "letters"
12needed to write in English. Perl is also aware that some character other
13than '.' may be preferred as a decimal point, and that output date
14representations may be language-specific.  The process of making an
15application take account of its users' preferences in such matters is
16called B<internationalization> (often abbreviated as B<i18n>); telling
17such an application about a particular set of preferences is known as
18B<localization> (B<l10n>).
19
20Perl can understand language-specific data via the standardized (ISO C,
21XPG4, POSIX 1.c) method called "the locale system". The locale system is
22controlled per application using one pragma, one function call, and
23several environment variables.
24
25B<NOTE>: This feature is new in Perl 5.004, and does not apply unless an
26application specifically requests it--see L<Backward compatibility>.
27The one exception is that write() now B<always> uses the current locale
28- see L<"NOTES">.
29
30=head1 PREPARING TO USE LOCALES
31
32If Perl applications are to understand and present your data
33correctly according a locale of your choice, B<all> of the following
34must be true:
35
36=over 4
37
38=item *
39
40B<Your operating system must support the locale system>.  If it does,
41you should find that the setlocale() function is a documented part of
42its C library.
43
44=item *
45
46B<Definitions for locales that you use must be installed>.  You, or
47your system administrator, must make sure that this is the case. The
48available locales, the location in which they are kept, and the manner
49in which they are installed all vary from system to system.  Some systems
50provide only a few, hard-wired locales and do not allow more to be
51added.  Others allow you to add "canned" locales provided by the system
52supplier.  Still others allow you or the system administrator to define
53and add arbitrary locales.  (You may have to ask your supplier to
54provide canned locales that are not delivered with your operating
55system.)  Read your system documentation for further illumination.
56
57=item *
58
59B<Perl must believe that the locale system is supported>.  If it does,
60C<perl -V:d_setlocale> will say that the value for C<d_setlocale> is
61C<define>.
62
63=back
64
65If you want a Perl application to process and present your data
66according to a particular locale, the application code should include
67the S<C<use locale>> pragma (see L<The use locale pragma>) where
68appropriate, and B<at least one> of the following must be true:
69
70=over 4
71
72=item *
73
74B<The locale-determining environment variables (see L<"ENVIRONMENT">)
75must be correctly set up> at the time the application is started, either
76by yourself or by whoever set up your system account.
77
78=item *
79
80B<The application must set its own locale> using the method described in
81L<The setlocale function>.
82
83=back
84
85=head1 USING LOCALES
86
87=head2 The use locale pragma
88
89By default, Perl ignores the current locale.  The S<C<use locale>>
90pragma tells Perl to use the current locale for some operations:
91
92=over 4
93
94=item *
95
96B<The comparison operators> (C<lt>, C<le>, C<cmp>, C<ge>, and C<gt>) and
97the POSIX string collation functions strcoll() and strxfrm() use
98C<LC_COLLATE>.  sort() is also affected if used without an
99explicit comparison function, because it uses C<cmp> by default.
100
101B<Note:> C<eq> and C<ne> are unaffected by locale: they always
102perform a byte-by-byte comparison of their scalar operands.  What's
103more, if C<cmp> finds that its operands are equal according to the
104collation sequence specified by the current locale, it goes on to
105perform a byte-by-byte comparison, and only returns I<0> (equal) if the
106operands are bit-for-bit identical.  If you really want to know whether
107two strings--which C<eq> and C<cmp> may consider different--are equal
108as far as collation in the locale is concerned, see the discussion in
109L<Category LC_COLLATE: Collation>.
110
111=item *
112
113B<Regular expressions and case-modification functions> (uc(), lc(),
114ucfirst(), and lcfirst()) use C<LC_CTYPE>
115
116=item *
117
118B<The formatting functions> (printf(), sprintf() and write()) use
119C<LC_NUMERIC>
120
121=item *
122
123B<The POSIX date formatting function> (strftime()) uses C<LC_TIME>.
124
125=back
126
127C<LC_COLLATE>, C<LC_CTYPE>, and so on, are discussed further in
128L<LOCALE CATEGORIES>.
129
130The default behavior is restored with the S<C<no locale>> pragma, or
131upon reaching the end of block enclosing C<use locale>.
132
133The string result of any operation that uses locale
134information is tainted, as it is possible for a locale to be
135untrustworthy.  See L<"SECURITY">.
136
137=head2 The setlocale function
138
139You can switch locales as often as you wish at run time with the
140POSIX::setlocale() function:
141
142        # This functionality not usable prior to Perl 5.004
143        require 5.004;
144
145        # Import locale-handling tool set from POSIX module.
146        # This example uses: setlocale -- the function call
147        #                    LC_CTYPE -- explained below
148        use POSIX qw(locale_h);
149
150        # query and save the old locale
151        $old_locale = setlocale(LC_CTYPE);
152
153        setlocale(LC_CTYPE, "fr_CA.ISO8859-1");
154        # LC_CTYPE now in locale "French, Canada, codeset ISO 8859-1"
155
156        setlocale(LC_CTYPE, "");
157        # LC_CTYPE now reset to default defined by LC_ALL/LC_CTYPE/LANG
158        # environment variables.  See below for documentation.
159
160        # restore the old locale
161        setlocale(LC_CTYPE, $old_locale);
162
163The first argument of setlocale() gives the B<category>, the second the
164B<locale>.  The category tells in what aspect of data processing you
165want to apply locale-specific rules.  Category names are discussed in
166L<LOCALE CATEGORIES> and L<"ENVIRONMENT">.  The locale is the name of a
167collection of customization information corresponding to a particular
168combination of language, country or territory, and codeset.  Read on for
169hints on the naming of locales: not all systems name locales as in the
170example.
171
172If no second argument is provided and the category is something else
173than LC_ALL, the function returns a string naming the current locale
174for the category.  You can use this value as the second argument in a
175subsequent call to setlocale().
176
177If no second argument is provided and the category is LC_ALL, the
178result is implementation-dependent.  It may be a string of
179concatenated locales names (separator also implementation-dependent)
180or a single locale name.  Please consult your L<setlocale(3)> for
181details.
182
183If a second argument is given and it corresponds to a valid locale,
184the locale for the category is set to that value, and the function
185returns the now-current locale value.  You can then use this in yet
186another call to setlocale().  (In some implementations, the return
187value may sometimes differ from the value you gave as the second
188argument--think of it as an alias for the value you gave.)
189
190As the example shows, if the second argument is an empty string, the
191category's locale is returned to the default specified by the
192corresponding environment variables.  Generally, this results in a
193return to the default that was in force when Perl started up: changes
194to the environment made by the application after startup may or may not
195be noticed, depending on your system's C library.
196
197If the second argument does not correspond to a valid locale, the locale
198for the category is not changed, and the function returns I<undef>.
199
200For further information about the categories, consult L<setlocale(3)>.
201
202=head2 Finding locales
203
204For locales available in your system, consult also L<setlocale(3)> to
205see whether it leads to the list of available locales (search for the
206I<SEE ALSO> section).  If that fails, try the following command lines:
207
208        locale -a
209
210        nlsinfo
211
212        ls /usr/lib/nls/loc
213
214        ls /usr/lib/locale
215
216        ls /usr/lib/nls
217
218        ls /usr/share/locale
219
220and see whether they list something resembling these
221
222        en_US.ISO8859-1     de_DE.ISO8859-1     ru_RU.ISO8859-5
223        en_US.iso88591      de_DE.iso88591      ru_RU.iso88595
224        en_US               de_DE               ru_RU
225        en                  de                  ru
226        english             german              russian
227        english.iso88591    german.iso88591     russian.iso88595
228        english.roman8                          russian.koi8r
229
230Sadly, even though the calling interface for setlocale() has been
231standardized, names of locales and the directories where the
232configuration resides have not been.  The basic form of the name is
233I<language_territory>B<.>I<codeset>, but the latter parts after
234I<language> are not always present.  The I<language> and I<country>
235are usually from the standards B<ISO 3166> and B<ISO 639>, the
236two-letter abbreviations for the countries and the languages of the
237world, respectively.  The I<codeset> part often mentions some B<ISO
2388859> character set, the Latin codesets.  For example, C<ISO 8859-1>
239is the so-called "Western European codeset" that can be used to encode
240most Western European languages adequately.  Again, there are several
241ways to write even the name of that one standard.  Lamentably.
242
243Two special locales are worth particular mention: "C" and "POSIX".
244Currently these are effectively the same locale: the difference is
245mainly that the first one is defined by the C standard, the second by
246the POSIX standard.  They define the B<default locale> in which
247every program starts in the absence of locale information in its
248environment.  (The I<default> default locale, if you will.)  Its language
249is (American) English and its character codeset ASCII.
250
251B<NOTE>: Not all systems have the "POSIX" locale (not all systems are
252POSIX-conformant), so use "C" when you need explicitly to specify this
253default locale.
254
255=head2 LOCALE PROBLEMS
256
257You may encounter the following warning message at Perl startup:
258
259        perl: warning: Setting locale failed.
260        perl: warning: Please check that your locale settings:
261                LC_ALL = "En_US",
262                LANG = (unset)
263            are supported and installed on your system.
264        perl: warning: Falling back to the standard locale ("C").
265
266This means that your locale settings had LC_ALL set to "En_US" and
267LANG exists but has no value.  Perl tried to believe you but could not.
268Instead, Perl gave up and fell back to the "C" locale, the default locale
269that is supposed to work no matter what.  This usually means your locale
270settings were wrong, they mention locales your system has never heard
271of, or the locale installation in your system has problems (for example,
272some system files are broken or missing).  There are quick and temporary
273fixes to these problems, as well as more thorough and lasting fixes.
274
275=head2 Temporarily fixing locale problems
276
277The two quickest fixes are either to render Perl silent about any
278locale inconsistencies or to run Perl under the default locale "C".
279
280Perl's moaning about locale problems can be silenced by setting the
281environment variable PERL_BADLANG to a zero value, for example "0".
282This method really just sweeps the problem under the carpet: you tell
283Perl to shut up even when Perl sees that something is wrong.  Do not
284be surprised if later something locale-dependent misbehaves.
285
286Perl can be run under the "C" locale by setting the environment
287variable LC_ALL to "C".  This method is perhaps a bit more civilized
288than the PERL_BADLANG approach, but setting LC_ALL (or
289other locale variables) may affect other programs as well, not just
290Perl.  In particular, external programs run from within Perl will see
291these changes.  If you make the new settings permanent (read on), all
292programs you run see the changes.  See L<ENVIRONMENT> for
293the full list of relevant environment variables and L<USING LOCALES>
294for their effects in Perl.  Effects in other programs are
295easily deducible.  For example, the variable LC_COLLATE may well affect
296your B<sort> program (or whatever the program that arranges `records'
297alphabetically in your system is called).
298
299You can test out changing these variables temporarily, and if the
300new settings seem to help, put those settings into your shell startup
301files.  Consult your local documentation for the exact details.  For in
302Bourne-like shells (B<sh>, B<ksh>, B<bash>, B<zsh>):
303
304        LC_ALL=en_US.ISO8859-1
305        export LC_ALL
306
307This assumes that we saw the locale "en_US.ISO8859-1" using the commands
308discussed above.  We decided to try that instead of the above faulty
309locale "En_US"--and in Cshish shells (B<csh>, B<tcsh>)
310
311        setenv LC_ALL en_US.ISO8859-1
312
313If you do not know what shell you have, consult your local
314helpdesk or the equivalent.
315
316=head2 Permanently fixing locale problems
317
318The slower but superior fixes are when you may be able to yourself
319fix the misconfiguration of your own environment variables.  The
320mis(sing)configuration of the whole system's locales usually requires
321the help of your friendly system administrator.
322
323First, see earlier in this document about L<Finding locales>.  That tells
324how to find which locales are really supported--and more importantly,
325installed--on your system.  In our example error message, environment
326variables affecting the locale are listed in the order of decreasing
327importance (and unset variables do not matter).  Therefore, having
328LC_ALL set to "En_US" must have been the bad choice, as shown by the
329error message.  First try fixing locale settings listed first.
330
331Second, if using the listed commands you see something B<exactly>
332(prefix matches do not count and case usually counts) like "En_US"
333without the quotes, then you should be okay because you are using a
334locale name that should be installed and available in your system.
335In this case, see L<Permanently fixing your system's locale configuration>.
336
337=head2 Permanently fixing your system's locale configuration
338
339This is when you see something like:
340
341        perl: warning: Please check that your locale settings:
342                LC_ALL = "En_US",
343                LANG = (unset)
344            are supported and installed on your system.
345
346but then cannot see that "En_US" listed by the above-mentioned
347commands.  You may see things like "en_US.ISO8859-1", but that isn't
348the same.  In this case, try running under a locale
349that you can list and which somehow matches what you tried.  The
350rules for matching locale names are a bit vague because
351standardization is weak in this area.  See again the
352L<Finding locales> about general rules.
353
354=head2 Fixing system locale configuration
355
356Contact a system administrator (preferably your own) and report the exact
357error message you get, and ask them to read this same documentation you
358are now reading.  They should be able to check whether there is something
359wrong with the locale configuration of the system.  The L<Finding locales>
360section is unfortunately a bit vague about the exact commands and places
361because these things are not that standardized.
362
363=head2 The localeconv function
364
365The POSIX::localeconv() function allows you to get particulars of the
366locale-dependent numeric formatting information specified by the current
367C<LC_NUMERIC> and C<LC_MONETARY> locales.  (If you just want the name of
368the current locale for a particular category, use POSIX::setlocale()
369with a single parameter--see L<The setlocale function>.)
370
371        use POSIX qw(locale_h);
372
373        # Get a reference to a hash of locale-dependent info
374        $locale_values = localeconv();
375
376        # Output sorted list of the values
377        for (sort keys %$locale_values) {
378            printf "%-20s = %s\n", $_, $locale_values->{$_}
379        }
380
381localeconv() takes no arguments, and returns B<a reference to> a hash.
382The keys of this hash are variable names for formatting, such as
383C<decimal_point> and C<thousands_sep>.  The values are the
384corresponding, er, values.  See L<POSIX/localeconv> for a longer
385example listing the categories an implementation might be expected to
386provide; some provide more and others fewer.  You don't need an
387explicit C<use locale>, because localeconv() always observes the
388current locale.
389
390Here's a simple-minded example program that rewrites its command-line
391parameters as integers correctly formatted in the current locale:
392
393        # See comments in previous example
394        require 5.004;
395        use POSIX qw(locale_h);
396
397        # Get some of locale's numeric formatting parameters
398        my ($thousands_sep, $grouping) =
399             @{localeconv()}{'thousands_sep', 'grouping'};
400
401        # Apply defaults if values are missing
402        $thousands_sep = ',' unless $thousands_sep;
403
404        # grouping and mon_grouping are packed lists
405        # of small integers (characters) telling the
406        # grouping (thousand_seps and mon_thousand_seps
407        # being the group dividers) of numbers and
408        # monetary quantities.  The integers' meanings:
409        # 255 means no more grouping, 0 means repeat
410        # the previous grouping, 1-254 means use that
411        # as the current grouping.  Grouping goes from
412        # right to left (low to high digits).  In the
413        # below we cheat slightly by never using anything
414        # else than the first grouping (whatever that is).
415        if ($grouping) {
416            @grouping = unpack("C*", $grouping);
417        } else {
418            @grouping = (3);
419        }
420
421        # Format command line params for current locale
422        for (@ARGV) {
423            $_ = int;    # Chop non-integer part
424            1 while
425            s/(\d)(\d{$grouping[0]}($|$thousands_sep))/$1$thousands_sep$2/;
426            print "$_";
427        }
428        print "\n";
429
430=head1 LOCALE CATEGORIES
431
432The following subsections describe basic locale categories.  Beyond these,
433some combination categories allow manipulation of more than one
434basic category at a time.  See L<"ENVIRONMENT"> for a discussion of these.
435
436=head2 Category LC_COLLATE: Collation
437
438In the scope of S<C<use locale>>, Perl looks to the C<LC_COLLATE>
439environment variable to determine the application's notions on collation
440(ordering) of characters.  For example, 'b' follows 'a' in Latin
441alphabets, but where do 'E<aacute>' and 'E<aring>' belong?  And while
442'color' follows 'chocolate' in English, what about in Spanish?
443
444The following collations all make sense and you may meet any of them
445if you "use locale".
446
447        A B C D E a b c d e
448        A a B b C c D d E e
449        a A b B c C d D e E
450        a b c d e A B C D E
451
452Here is a code snippet to tell what "word"
453characters are in the current locale, in that locale's order:
454
455        use locale;
456        print +(sort grep /\w/, map { chr } 0..255), "\n";
457
458Compare this with the characters that you see and their order if you
459state explicitly that the locale should be ignored:
460
461        no locale;
462        print +(sort grep /\w/, map { chr } 0..255), "\n";
463
464This machine-native collation (which is what you get unless S<C<use
465locale>> has appeared earlier in the same block) must be used for
466sorting raw binary data, whereas the locale-dependent collation of the
467first example is useful for natural text.
468
469As noted in L<USING LOCALES>, C<cmp> compares according to the current
470collation locale when C<use locale> is in effect, but falls back to a
471byte-by-byte comparison for strings that the locale says are equal. You
472can use POSIX::strcoll() if you don't want this fall-back:
473
474        use POSIX qw(strcoll);
475        $equal_in_locale =
476            !strcoll("space and case ignored", "SpaceAndCaseIgnored");
477
478$equal_in_locale will be true if the collation locale specifies a
479dictionary-like ordering that ignores space characters completely and
480which folds case.
481
482If you have a single string that you want to check for "equality in
483locale" against several others, you might think you could gain a little
484efficiency by using POSIX::strxfrm() in conjunction with C<eq>:
485
486        use POSIX qw(strxfrm);
487        $xfrm_string = strxfrm("Mixed-case string");
488        print "locale collation ignores spaces\n"
489            if $xfrm_string eq strxfrm("Mixed-casestring");
490        print "locale collation ignores hyphens\n"
491            if $xfrm_string eq strxfrm("Mixedcase string");
492        print "locale collation ignores case\n"
493            if $xfrm_string eq strxfrm("mixed-case string");
494
495strxfrm() takes a string and maps it into a transformed string for use
496in byte-by-byte comparisons against other transformed strings during
497collation.  "Under the hood", locale-affected Perl comparison operators
498call strxfrm() for both operands, then do a byte-by-byte
499comparison of the transformed strings.  By calling strxfrm() explicitly
500and using a non locale-affected comparison, the example attempts to save
501a couple of transformations.  But in fact, it doesn't save anything: Perl
502magic (see L<perlguts/Magic Variables>) creates the transformed version of a
503string the first time it's needed in a comparison, then keeps this version around
504in case it's needed again.  An example rewritten the easy way with
505C<cmp> runs just about as fast.  It also copes with null characters
506embedded in strings; if you call strxfrm() directly, it treats the first
507null it finds as a terminator.  don't expect the transformed strings
508it produces to be portable across systems--or even from one revision
509of your operating system to the next.  In short, don't call strxfrm()
510directly: let Perl do it for you.
511
512Note: C<use locale> isn't shown in some of these examples because it isn't
513needed: strcoll() and strxfrm() exist only to generate locale-dependent
514results, and so always obey the current C<LC_COLLATE> locale.
515
516=head2 Category LC_CTYPE: Character Types
517
518In the scope of S<C<use locale>>, Perl obeys the C<LC_CTYPE> locale
519setting.  This controls the application's notion of which characters are
520alphabetic.  This affects Perl's C<\w> regular expression metanotation,
521which stands for alphanumeric characters--that is, alphabetic,
522numeric, and including other special characters such as the underscore or
523hyphen.  (Consult L<perlre> for more information about
524regular expressions.)  Thanks to C<LC_CTYPE>, depending on your locale
525setting, characters like 'E<aelig>', 'E<eth>', 'E<szlig>', and
526'E<oslash>' may be understood as C<\w> characters.
527
528The C<LC_CTYPE> locale also provides the map used in transliterating
529characters between lower and uppercase.  This affects the case-mapping
530functions--lc(), lcfirst, uc(), and ucfirst(); case-mapping
531interpolation with C<\l>, C<\L>, C<\u>, or C<\U> in double-quoted strings
532and C<s///> substitutions; and case-independent regular expression
533pattern matching using the C<i> modifier.
534
535Finally, C<LC_CTYPE> affects the POSIX character-class test
536functions--isalpha(), islower(), and so on.  For example, if you move
537from the "C" locale to a 7-bit Scandinavian one, you may find--possibly
538to your surprise--that "|" moves from the ispunct() class to isalpha().
539
540B<Note:> A broken or malicious C<LC_CTYPE> locale definition may result
541in clearly ineligible characters being considered to be alphanumeric by
542your application.  For strict matching of (mundane) letters and
543digits--for example, in command strings--locale-aware applications
544should use C<\w> inside a C<no locale> block.  See L<"SECURITY">.
545
546=head2 Category LC_NUMERIC: Numeric Formatting
547
548In the scope of S<C<use locale>>, Perl obeys the C<LC_NUMERIC> locale
549information, which controls an application's idea of how numbers should
550be formatted for human readability by the printf(), sprintf(), and
551write() functions.  String-to-numeric conversion by the POSIX::strtod()
552function is also affected.  In most implementations the only effect is to
553change the character used for the decimal point--perhaps from '.'  to ','.
554These functions aren't aware of such niceties as thousands separation and
555so on.  (See L<The localeconv function> if you care about these things.)
556
557Output produced by print() is also affected by the current locale: it
558depends on whether C<use locale> or C<no locale> is in effect, and
559corresponds to what you'd get from printf() in the "C" locale.  The
560same is true for Perl's internal conversions between numeric and
561string formats:
562
563        use POSIX qw(strtod);
564        use locale;
565
566        $n = 5/2;   # Assign numeric 2.5 to $n
567
568        $a = " $n"; # Locale-dependent conversion to string
569
570        print "half five is $n\n";       # Locale-dependent output
571
572        printf "half five is %g\n", $n;  # Locale-dependent output
573
574        print "DECIMAL POINT IS COMMA\n"
575            if $n == (strtod("2,5"))[0]; # Locale-dependent conversion
576
577=head2 Category LC_MONETARY: Formatting of monetary amounts
578
579The C standard defines the C<LC_MONETARY> category, but no function
580that is affected by its contents.  (Those with experience of standards
581committees will recognize that the working group decided to punt on the
582issue.)  Consequently, Perl takes no notice of it.  If you really want
583to use C<LC_MONETARY>, you can query its contents--see
584L<The localeconv function>--and use the information that it returns in your
585application's own formatting of currency amounts.  However, you may well
586find that the information, voluminous and complex though it may be, still
587does not quite meet your requirements: currency formatting is a hard nut
588to crack.
589
590=head2 LC_TIME
591
592Output produced by POSIX::strftime(), which builds a formatted
593human-readable date/time string, is affected by the current C<LC_TIME>
594locale.  Thus, in a French locale, the output produced by the C<%B>
595format element (full month name) for the first month of the year would
596be "janvier".  Here's how to get a list of long month names in the
597current locale:
598
599        use POSIX qw(strftime);
600        for (0..11) {
601            $long_month_name[$_] =
602                strftime("%B", 0, 0, 0, 1, $_, 96);
603        }
604
605Note: C<use locale> isn't needed in this example: as a function that
606exists only to generate locale-dependent results, strftime() always
607obeys the current C<LC_TIME> locale.
608
609=head2 Other categories
610
611The remaining locale category, C<LC_MESSAGES> (possibly supplemented
612by others in particular implementations) is not currently used by
613Perl--except possibly to affect the behavior of library functions
614called by extensions outside the standard Perl distribution and by the
615operating system and its utilities.  Note especially that the string
616value of C<$!> and the error messages given by external utilities may
617be changed by C<LC_MESSAGES>.  If you want to have portable error
618codes, use C<%!>.  See L<Errno>.
619
620=head1 SECURITY
621
622Although the main discussion of Perl security issues can be found in
623L<perlsec>, a discussion of Perl's locale handling would be incomplete
624if it did not draw your attention to locale-dependent security issues.
625Locales--particularly on systems that allow unprivileged users to
626build their own locales--are untrustworthy.  A malicious (or just plain
627broken) locale can make a locale-aware application give unexpected
628results.  Here are a few possibilities:
629
630=over 4
631
632=item *
633
634Regular expression checks for safe file names or mail addresses using
635C<\w> may be spoofed by an C<LC_CTYPE> locale that claims that
636characters such as "E<gt>" and "|" are alphanumeric.
637
638=item *
639
640String interpolation with case-mapping, as in, say, C<$dest =
641"C:\U$name.$ext">, may produce dangerous results if a bogus LC_CTYPE
642case-mapping table is in effect.
643
644=item *
645
646A sneaky C<LC_COLLATE> locale could result in the names of students with
647"D" grades appearing ahead of those with "A"s.
648
649=item *
650
651An application that takes the trouble to use information in
652C<LC_MONETARY> may format debits as if they were credits and vice versa
653if that locale has been subverted.  Or it might make payments in US
654dollars instead of Hong Kong dollars.
655
656=item *
657
658The date and day names in dates formatted by strftime() could be
659manipulated to advantage by a malicious user able to subvert the
660C<LC_DATE> locale.  ("Look--it says I wasn't in the building on
661Sunday.")
662
663=back
664
665Such dangers are not peculiar to the locale system: any aspect of an
666application's environment which may be modified maliciously presents
667similar challenges.  Similarly, they are not specific to Perl: any
668programming language that allows you to write programs that take
669account of their environment exposes you to these issues.
670
671Perl cannot protect you from all possibilities shown in the
672examples--there is no substitute for your own vigilance--but, when
673C<use locale> is in effect, Perl uses the tainting mechanism (see
674L<perlsec>) to mark string results that become locale-dependent, and
675which may be untrustworthy in consequence.  Here is a summary of the
676tainting behavior of operators and functions that may be affected by
677the locale:
678
679=over 4
680
681=item  *
682
683B<Comparison operators> (C<lt>, C<le>, C<ge>, C<gt> and C<cmp>):
684
685Scalar true/false (or less/equal/greater) result is never tainted.
686
687=item  *
688
689B<Case-mapping interpolation> (with C<\l>, C<\L>, C<\u> or C<\U>)
690
691Result string containing interpolated material is tainted if
692C<use locale> is in effect.
693
694=item  *
695
696B<Matching operator> (C<m//>):
697
698Scalar true/false result never tainted.
699
700Subpatterns, either delivered as a list-context result or as $1 etc.
701are tainted if C<use locale> is in effect, and the subpattern regular
702expression contains C<\w> (to match an alphanumeric character), C<\W>
703(non-alphanumeric character), C<\s> (white-space character), or C<\S>
704(non white-space character).  The matched-pattern variable, $&, $`
705(pre-match), $' (post-match), and $+ (last match) are also tainted if
706C<use locale> is in effect and the regular expression contains C<\w>,
707C<\W>, C<\s>, or C<\S>.
708
709=item  *
710
711B<Substitution operator> (C<s///>):
712
713Has the same behavior as the match operator.  Also, the left
714operand of C<=~> becomes tainted when C<use locale> in effect
715if modified as a result of a substitution based on a regular
716expression match involving C<\w>, C<\W>, C<\s>, or C<\S>; or of
717case-mapping with C<\l>, C<\L>,C<\u> or C<\U>.
718
719=item  *
720
721B<Output formatting functions> (printf() and write()):
722
723Results are never tainted because otherwise even output from print,
724for example C<print(1/7)>, should be tainted if C<use locale> is in
725effect.
726
727=item  *
728
729B<Case-mapping functions> (lc(), lcfirst(), uc(), ucfirst()):
730
731Results are tainted if C<use locale> is in effect.
732
733=item  *
734
735B<POSIX locale-dependent functions> (localeconv(), strcoll(),
736strftime(), strxfrm()):
737
738Results are never tainted.
739
740=item  *
741
742B<POSIX character class tests> (isalnum(), isalpha(), isdigit(),
743isgraph(), islower(), isprint(), ispunct(), isspace(), isupper(),
744isxdigit()):
745
746True/false results are never tainted.
747
748=back
749
750Three examples illustrate locale-dependent tainting.
751The first program, which ignores its locale, won't run: a value taken
752directly from the command line may not be used to name an output file
753when taint checks are enabled.
754
755        #/usr/local/bin/perl -T
756        # Run with taint checking
757
758        # Command line sanity check omitted...
759        $tainted_output_file = shift;
760
761        open(F, ">$tainted_output_file")
762            or warn "Open of $untainted_output_file failed: $!\n";
763
764The program can be made to run by "laundering" the tainted value through
765a regular expression: the second example--which still ignores locale
766information--runs, creating the file named on its command line
767if it can.
768
769        #/usr/local/bin/perl -T
770
771        $tainted_output_file = shift;
772        $tainted_output_file =~ m%[\w/]+%;
773        $untainted_output_file = $&;
774
775        open(F, ">$untainted_output_file")
776            or warn "Open of $untainted_output_file failed: $!\n";
777
778Compare this with a similar but locale-aware program:
779
780        #/usr/local/bin/perl -T
781
782        $tainted_output_file = shift;
783        use locale;
784        $tainted_output_file =~ m%[\w/]+%;
785        $localized_output_file = $&;
786
787        open(F, ">$localized_output_file")
788            or warn "Open of $localized_output_file failed: $!\n";
789
790This third program fails to run because $& is tainted: it is the result
791of a match involving C<\w> while C<use locale> is in effect.
792
793=head1 ENVIRONMENT
794
795=over 12
796
797=item PERL_BADLANG
798
799A string that can suppress Perl's warning about failed locale settings
800at startup.  Failure can occur if the locale support in the operating
801system is lacking (broken) in some way--or if you mistyped the name of
802a locale when you set up your environment.  If this environment
803variable is absent, or has a value that does not evaluate to integer
804zero--that is, "0" or ""-- Perl will complain about locale setting
805failures.
806
807B<NOTE>: PERL_BADLANG only gives you a way to hide the warning message.
808The message tells about some problem in your system's locale support,
809and you should investigate what the problem is.
810
811=back
812
813The following environment variables are not specific to Perl: They are
814part of the standardized (ISO C, XPG4, POSIX 1.c) setlocale() method
815for controlling an application's opinion on data.
816
817=over 12
818
819=item LC_ALL
820
821C<LC_ALL> is the "override-all" locale environment variable. If
822set, it overrides all the rest of the locale environment variables.
823
824=item LANGUAGE
825
826B<NOTE>: C<LANGUAGE> is a GNU extension, it affects you only if you
827are using the GNU libc.  This is the case if you are using e.g. Linux.
828If you are using "commercial" UNIXes you are most probably I<not>
829using GNU libc and you can ignore C<LANGUAGE>.
830
831However, in the case you are using C<LANGUAGE>: it affects the
832language of informational, warning, and error messages output by
833commands (in other words, it's like C<LC_MESSAGES>) but it has higher
834priority than L<LC_ALL>.  Moreover, it's not a single value but
835instead a "path" (":"-separated list) of I<languages> (not locales).
836See the GNU C<gettext> library documentation for more information.
837
838=item LC_CTYPE
839
840In the absence of C<LC_ALL>, C<LC_CTYPE> chooses the character type
841locale.  In the absence of both C<LC_ALL> and C<LC_CTYPE>, C<LANG>
842chooses the character type locale.
843
844=item LC_COLLATE
845
846In the absence of C<LC_ALL>, C<LC_COLLATE> chooses the collation
847(sorting) locale.  In the absence of both C<LC_ALL> and C<LC_COLLATE>,
848C<LANG> chooses the collation locale.
849
850=item LC_MONETARY
851
852In the absence of C<LC_ALL>, C<LC_MONETARY> chooses the monetary
853formatting locale.  In the absence of both C<LC_ALL> and C<LC_MONETARY>,
854C<LANG> chooses the monetary formatting locale.
855
856=item LC_NUMERIC
857
858In the absence of C<LC_ALL>, C<LC_NUMERIC> chooses the numeric format
859locale.  In the absence of both C<LC_ALL> and C<LC_NUMERIC>, C<LANG>
860chooses the numeric format.
861
862=item LC_TIME
863
864In the absence of C<LC_ALL>, C<LC_TIME> chooses the date and time
865formatting locale.  In the absence of both C<LC_ALL> and C<LC_TIME>,
866C<LANG> chooses the date and time formatting locale.
867
868=item LANG
869
870C<LANG> is the "catch-all" locale environment variable. If it is set, it
871is used as the last resort after the overall C<LC_ALL> and the
872category-specific C<LC_...>.
873
874=back
875
876=head1 NOTES
877
878=head2 Backward compatibility
879
880Versions of Perl prior to 5.004 B<mostly> ignored locale information,
881generally behaving as if something similar to the C<"C"> locale were
882always in force, even if the program environment suggested otherwise
883(see L<The setlocale function>).  By default, Perl still behaves this
884way for backward compatibility.  If you want a Perl application to pay
885attention to locale information, you B<must> use the S<C<use locale>>
886pragma (see L<The use locale pragma>) to instruct it to do so.
887
888Versions of Perl from 5.002 to 5.003 did use the C<LC_CTYPE>
889information if available; that is, C<\w> did understand what
890were the letters according to the locale environment variables.
891The problem was that the user had no control over the feature:
892if the C library supported locales, Perl used them.
893
894=head2 I18N:Collate obsolete
895
896In versions of Perl prior to 5.004, per-locale collation was possible
897using the C<I18N::Collate> library module.  This module is now mildly
898obsolete and should be avoided in new applications.  The C<LC_COLLATE>
899functionality is now integrated into the Perl core language: One can
900use locale-specific scalar data completely normally with C<use locale>,
901so there is no longer any need to juggle with the scalar references of
902C<I18N::Collate>.
903
904=head2 Sort speed and memory use impacts
905
906Comparing and sorting by locale is usually slower than the default
907sorting; slow-downs of two to four times have been observed.  It will
908also consume more memory: once a Perl scalar variable has participated
909in any string comparison or sorting operation obeying the locale
910collation rules, it will take 3-15 times more memory than before.  (The
911exact multiplier depends on the string's contents, the operating system
912and the locale.) These downsides are dictated more by the operating
913system's implementation of the locale system than by Perl.
914
915=head2 write() and LC_NUMERIC
916
917Formats are the only part of Perl that unconditionally use information
918from a program's locale; if a program's environment specifies an
919LC_NUMERIC locale, it is always used to specify the decimal point
920character in formatted output.  Formatted output cannot be controlled by
921C<use locale> because the pragma is tied to the block structure of the
922program, and, for historical reasons, formats exist outside that block
923structure.
924
925=head2 Freely available locale definitions
926
927There is a large collection of locale definitions at
928C<ftp://dkuug.dk/i18n/WG15-collection>.  You should be aware that it is
929unsupported, and is not claimed to be fit for any purpose.  If your
930system allows installation of arbitrary locales, you may find the
931definitions useful as they are, or as a basis for the development of
932your own locales.
933
934=head2 I18n and l10n
935
936"Internationalization" is often abbreviated as B<i18n> because its first
937and last letters are separated by eighteen others.  (You may guess why
938the internalin ... internaliti ... i18n tends to get abbreviated.)  In
939the same way, "localization" is often abbreviated to B<l10n>.
940
941=head2 An imperfect standard
942
943Internationalization, as defined in the C and POSIX standards, can be
944criticized as incomplete, ungainly, and having too large a granularity.
945(Locales apply to a whole process, when it would arguably be more useful
946to have them apply to a single thread, window group, or whatever.)  They
947also have a tendency, like standards groups, to divide the world into
948nations, when we all know that the world can equally well be divided
949into bankers, bikers, gamers, and so on.  But, for now, it's the only
950standard we've got.  This may be construed as a bug.
951
952=head1 BUGS
953
954=head2 Broken systems
955
956In certain systems, the operating system's locale support
957is broken and cannot be fixed or used by Perl.  Such deficiencies can
958and will result in mysterious hangs and/or Perl core dumps when the
959C<use locale> is in effect.  When confronted with such a system,
960please report in excruciating detail to <F<perlbug@perl.org>>, and
961complain to your vendor: bug fixes may exist for these problems
962in your operating system.  Sometimes such bug fixes are called an
963operating system upgrade.
964
965=head1 SEE ALSO
966
967L<POSIX/isalnum>, L<POSIX/isalpha>, L<POSIX/isdigit>,
968L<POSIX/isgraph>, L<POSIX/islower>, L<POSIX/isprint>,
969L<POSIX/ispunct>, L<POSIX/isspace>, L<POSIX/isupper>,
970L<POSIX/isxdigit>, L<POSIX/localeconv>, L<POSIX/setlocale>,
971L<POSIX/strcoll>, L<POSIX/strftime>, L<POSIX/strtod>,
972L<POSIX/strxfrm>.
973
974=head1 HISTORY
975
976Jarkko Hietaniemi's original F<perli18n.pod> heavily hacked by Dominic
977Dunlop, assisted by the perl5-porters.  Prose worked over a bit by
978Tom Christiansen.
979
980Last update: Thu Jun 11 08:44:13 MDT 1998
Note: See TracBrowser for help on using the repository browser.