1LIBHUGETLBFS(7)        Miscellaneous Information Manual        LIBHUGETLBFS(7)


6       libhugetlbfs  -  preload library to back text, data, malloc() or shared
7       memory with hugepages


10       export [environment options]
11       [LD_PRELOAD=libhugetlbfs.so] target_application


14       libhugetlbfs is a library that can back application  text,  data,  mal‐
15       loc()  and shared memory with hugepages. This is of benefit to applica‐
16       tions that use large amounts of address space and suffer a  performance
17       hit  due  to  TLB  misses.  Wall-clock  time or oprofile can be used to
18       determine if there is a performance benefit from using libhugetlbfs  or
19       not.   In all cases but shared memory, a hugetlbfs mount must exist and
20       a hugepage pool defined for hugepages to be used.
22       Some limited functionality  is  available  for  unmodified  dynamically
23       linked  applications.  By  preloading the library, the library can back
24       malloc() and shared memory, and text and data segments can be partially
25       backed if they are large enough.
27       For  the effective backing of text and data with huge pages, the appli‐
28       cation must be linked to the library and  the  ELF  segments  correctly
29       aligned  using the ld helpers. Once linked, malloc or shared memory can
30       still be backed but no pre-loading is required. See /usr/share/doc/lib‐
31       hugetlbfs/HOWTO   and  ld.hugetlbfs(1)  for  detailed  instructions  on
32       relinking applications.
34       For applications that are hugepage-aware  and  linked  to  the  library
35       get_huge_pages()  can  be  used  for the direct allocation of hugepage-
36       backed regions.
38       Unless otherwise specified, libhugetlbfs will use the default  hugepage
39       size to back memory regions. The default size is the value of Hugepage‐
40       size displayed in /proc/meminfo. The size can be specified in bytes  or
41       in  kilobytes,  megabytes, or gigabytes by appending K, M, or G respec‐
42       tively. It is an error to specify a invalid, unsupported, or  otherwise
43       unconfigured  huge  page  size.  Kernel  2.6.27 or later is required to
44       specify any pagesize other than the default.
46       See /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on how
47       the  library  should  be used, particularly when relinking the applica‐
48       tion.  This manual page provides a brief synopsis  of  the  environment
49       variables as a quick reference.
51       The  following  variables  affect  what  memory  regions  are backed by
52       hugepages. In all cases, the environment being unset implies  the  fea‐
53       ture should remain disabled.
56       HUGETLB_DEFAULT_PAGE_SIZE=<pagesize>
57              This  sets the default hugepage size to be used by libhugetlbfs.
58              If not set, libhugetlbfs will use the kernel's default  hugepage
59              size.
62       HUGETLB_MORECORE=[yes|<pagesize>]
63              This  enables  the  hugepage  malloc() feature, instructing lib‐
64              hugetlbfs to override glibc's normal morecore() function with  a
65              hugepage  version and use it for malloc().  All application mal‐
66              loc() memory should come from hugepage memory until it runs out,
67              it  will  then  fallback  to base pages.  Note that applications
68              that use custom allocators may not be able to back  their  heaps
69              using  hugepages and this environment variable. It may be neces‐
70              sary to modify the custom allocator to use get_huge_pages().
73       HUGETLB_SHM=yes
74              When this environment variable is set, the SHM_HUGETLB  flag  is
75              added  to the shmget() call and the size parameter is aligned to
76              back the shared memory segment  with  hugepages.  In  the  event
77              hugepages  cannot be used, base pages will be used instead and a
78              warning will be printed to explain  the  failure.  The  pagesize
79              cannot  be  specified with this parameter. To change the kernels
80              default hugepage size, use the pagesize= kernel  boot  parameter
81              (2.6.26 or later required).
84       HUGETLB_ELFMAP=[no|[R[<=pagesize>]:[W[<=pagesize>]]]
85              If the application has been relinked (see the HOWTO for instruc‐
86              tions), this environment variable determines whether  read-only,
87              read-write, both or no segments are backed by hugepages and what
88              pagesize should be used. If the recommended relinking method has
89              been  used,  then hugeedit can be used to automatically back the
90              text or data by default.
94              Force the use of hugepages for text and data  segments  even  if
95              the  application has not been relinked to align the ELF segments
96              on a hugepage boundary.  Partial segment remapping is not  guar‐
97              anteed  to work and the segments must be large enough to contain
98              at least one hugepage for the remapping to occur.
101       The following options affect how libhugetlbfs behaves.
104       HUGETLB_RESTRICT_EXE=e1:e2:...:eN
105              By default, libhugetlbfs will act on  any  program  that  it  is
106              loaded with, either via LD_PRELOAD or by explicitly linking with
107              -lhugetlbfs.
109              There are situations in which it is desirable to  restrict  lib‐
110              hugetlbfs'  actions to specific programs.  For example, some ISV
111              applications are wrapped in a  series  of  scripts  that  invoke
112              bash,  python,  and/or  perl.   It is more convenient to set the
113              environment variables related to  libhugetlbfs  before  invoking
114              the wrapper scripts, yet this has the unintended and undesirable
115              consequence of causing the script interpreters to use  and  con‐
116              sume  hugepages.   There  is  no  obvious benefit to causing the
117              script interpreters to use hugepages, and there is a clear  dis‐
118              advantage:  fewer hugepages are available to the actual applica‐
119              tion.
121              To address this scenario, set HUGETLB_RESTRICT_EXE to  a  colon-
122              separated list of programs to which the other libhugetlbfs envi‐
123              ronment variables should apply.  (If not set, libhugetlbfs  will
124              attempt  to  apply  the requested actions to all programs.)  For
125              example,
127                  HUGETLB_RESTRICT_EXE=hpcc:long_hpcc
129              will  restrict   libhugetlbfs'   actions   to   programs   named
130              /home/fred/hpcc and /bench/long_hpcc but not /bin/hpcc_no.
135              By  default,  the  hugepage  heap  does not shrink. Shrinking is
136              enabled by setting this environment variable. It is disabled  by
137              default  as  glibc occasionally exhibits strange behaviour if it
138              mistakes the heap returned by libhugetlbfs as a foreign brk().
142              By default libhugetlbfs will  prefault  regions  it  creates  to
143              ensure  they  can  be referenced without receiving a SIGKILL. On
144              kernels older than 2.6.27, this was necessary as the system  did
145              not guarantee that future faults would succeed on regions mapped
146              MAP_PRIVATE.  Prefaulting impacts the  performance  of  malloc()
147              and can result in poor placement on NUMA systems. If it is known
148              the hugepage pool is large enough to run the application or  the
149              kernel  is  2.6.27 or later, this environment variable should be
150              set.
153       HUGETLB_NO_RESERVE=yes
155              By default, the kernel will reserve huge pages at mmap() time to
156              ensure  that  future faults will succeed. This avoids unexpected
157              application failure at fault time but some  applications  depend
158              on  memory  overcommit to create large sparse mappings. For this
159              type of application, setting this environment variable will cre‐
160              ate  huge  page  backed mappings without a reservation. Use this
161              option with extreme care as in the  event  huge  pages  are  not
162              available  when  the  mapping  is  used, the application will be
163              killed. On older kernels, the use of this  feature  can  trigger
164              the OOM killer. Hence, even with this variable set, reservations
165              may still be used for safety.
169              libhugetlbfs normally picks an address to use as the base of the
170              heap for malloc() automatically. This environment variable fixes
171              which address is used.
174       HUGETLB_PATH=<path>
175              The path to the hugetlbfs mount is automatically  determined  at
176              run-time.  In  the event there are multiple mounts and the wrong
177              one is being selected, use this option  to  select  the  correct
178              one.  This may be the case if an application-specific mount with
179              a fixed quota has been created for example.
182       HUGETLB_SHARE=1
183              By default, libhugetlbfs uses unlinked hugetlbfs files to  store
184              remapped  program  segment  data. If the same program is started
185              multiple times using hugepage segments, multiple hugepages  will
186              be used to store the same program data. The reduce this wastage,
187              setting this environment variable will share read-only  segments
188              between  multiple  invocations  of  a program at the cost of the
189              memory being used whether the applications are running  or  not.
190              It  is also possible that a malicious application inferfere with
191              other applications executable  code.  See  the  HOWTO  for  more
192              detailed information on this topic.
195       The following options control the verbosity of libhugetlbfs.
198       HUGETLB_VERBOSE=<level>
199              The  default  value  for this is 1 and the range of the value is
200              from 0 to 99. The higher the value, the more verbose the  output
201              is. 0 is quiet and 3 will output much debugging information.
205              Once set, this will give very detailed output on what is happen‐
206              ing in the library and run extra diagnostics.


210       [DESTDIR|/usr/share]/doc/libhugetlbfs/HOWTO


214       oprofile(1),  ld.hugetlbfs(1),  hugectl(8),  hugeedit(8),  gethugepage‐
215       size(3),  gethugepagesizes(3), getpagesizes(3), hugetlbfs_test_path(3),
216       hugetlbfs_find_path(3),                hugetlbfs_find_path_for_size(3),
217       hugetlbfs_test_path(3),                hugetlbfs_test_path_for_size(3),
218       hugetlbfs_unlinked_fd(3),            hugetlbfs_unlinked_fd_for_size(3),
219       get_huge_pages(3), free_huge_pages(3)


222       libhugetlbfs  was  written  by various people on the libhugetlbfs-devel
223       mailing list.
227                              September 27, 2008               LIBHUGETLBFS(7)