1LIBHUGETLBFS(7) Miscellaneous Information Manual LIBHUGETLBFS(7)
2
6 libhugetlbfs - preload library to back text, data, malloc() or shared
7 memory with hugepages
8
10 export [environment options]
11 [LD_PRELOAD=libhugetlbfs.so] target_application
12
14 libhugetlbfs is a library that can back application text, data, mal‐
15 loc() and shared memory with hugepages. This is of benefit to applica‐
16 tions that use large amounts of address space and suffer a performance
17 hit due to TLB misses. Wall-clock time or oprofile can be used to
18 determine if there is a performance benefit from using libhugetlbfs or
19 not. In all cases but shared memory, a hugetlbfs mount must exist and
20 a hugepage pool defined for hugepages to be used.
21 Some limited functionality is available for unmodified dynamically
23 linked applications. By preloading the library, the library can back
24 malloc() and shared memory, and text and data segments can be partially
25 backed if they are large enough.
26 For the effective backing of text and data with huge pages, the appli‐
28 cation must be linked to the library and the ELF segments correctly
29 aligned using the ld helpers. Once linked, malloc or shared memory can
30 still be backed but no pre-loading is required. See
31 /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on relink‐
32 ing applications.
33 For applications that are hugepage-aware and linked to the library
35 get_huge_pages() can be used for the direct allocation of hugepage-
36 backed regions.
37 Unless otherwise specified, libhugetlbfs will use the default hugepage
39 size to back memory regions. The default size is the value of Hugepage‐
40 size displayed in /proc/meminfo. The size can be specified in bytes or
41 in kilobytes, megabytes, or gigabytes by appending K, M, or G respec‐
42 tively. It is an error to specify a invalid, unsupported, or otherwise
43 unconfigured huge page size. Kernel 2.6.27 or later is required to
44 specify any pagesize other than the default.
45 See /usr/share/docs/libhugetlbfs/HOWTO for detailed instructions on how
47 the library should be used, particularly when relinking the applica‐
48 tion. This manual page provides a brief synopsis of the environment
49 variables as a quick reference.
50 The following variables affect what memory regions are backed by
52 hugepages. In all cases, the environment being unset implies the fea‐
53 ture should remain disabled.
54 HUGETLB_MORECORE=[yes|<pagesize>]
57 This enables the hugepage malloc() feature, instructing lib‐
58 hugetlbfs to override glibc's normal morecore() function with a
59 hugepage version and use it for malloc(). All application mal‐
60 loc() memory should come from hugepage memory until it runs out,
61 it will then fallback to base pages. Note that applications
62 that use custom allocators may not be able to back their heaps
63 using hugepages and this environment variable. It may be neces‐
64 sary to modify the custom allocator to use get_huge_pages().
65 HUGETLB_SHM=yes
68 When this environment variable is set, the SHM_HUGETLB flag is
69 added to the shmget() call and the size parameter is aligned to
70 back the shared memory segment with hugepages. In the event
71 hugepages cannot be used, base pages will be used instead and a
72 warning will be printed to explain the failure. The pagesize
73 cannot be specified with this parameter. To change the kernels
74 default hugepage size, use the pagesize= kernel boot parameter
75 (2.6.26 or later required).
76 HUGETLB_ELFMAP=[no|[R[<=pagesize>]:[W[<=pagesize>]]]
79 If the application has been relinked (see the HOWTO for instruc‐
80 tions), this environment variable determines whether read-only,
81 read-write, both or no segments are backed by hugepages and what
82 pagesize should be used. If the recommended relinking method has
83 been used, then hugeedit can be used to automatically back the
84 text or data by default.
85 HUGETLB_FORCE_ELFMAP=yes
88 Force the use of hugepages for text and data segments even if
89 the application has not been relinked to align the ELF segments
90 on a hugepage boundary. Partial segment remapping is not guar‐
91 anteed to work and the segments must be large enough to contain
92 at least one hugepage for the remapping to occur.
93 The following options affect how libhugetlbfs behaves.
96 HUGETLB_MORECORE_SHRINK=yes
99 By default, the hugepage heap does not shrink. Shrinking is
100 enabled by setting this environment variable. It is disabled by
101 default as glibc occasionally exhibits strange behaviour if it
102 mistakes the heap returned by libhugetlbfs as a foreign brk().
103 HUGETLB_NO_PREFAULT
106 By default libhugetlbfs will prefault regions it creates to
107 ensure they can be referenced without receiving a SIGKILL. On
108 kernels older than 2.6.27, this was necessary as the system did
109 not guarantee that future faults would succeed on regions mapped
110 MAP_PRIVATE. Prefaulting impacts the performance of malloc()
111 and can result in poor placement on NUMA systems. If it is known
112 the hugepage pool is large enough to run the application or the
113 kernel is 2.6.27 or later, this environment variable should be
114 set.
115 HUGETLB_NO_RESERVE=yes
118 By default, the kernel will reserve huge pages at mmap() time to
120 ensure that future faults will succeed. This avoids unexpected
121 application failure at fault time but some applications depend
122 on memory overcommit to create large sparse mappings. For this
123 type of application, setting this environment variable will cre‐
124 ate huge page backed mappings without a reservation. Use this
125 option with extreme care as in the event huge pages are not
126 available when the mapping is used, the application will be
127 killed. On older kernels, the use of this feature can trigger
128 the OOM killer. Hence, even with this variable set, reservations
129 may still be used for safety.
130 HUGETLB_MORECORE_HEAPBASE=address
133 libhugetlbfs normally picks an address to use as the base of the
134 heap for malloc() automatically. This environment variable fixes
135 which address is used.
136 HUGETLB_PATH=<path>
139 The path to the hugetlbfs mount is automatically determined at
140 run-time. In the event there are multiple mounts and the wrong
141 one is being selected, use this option to select the correct
142 one. This may be the case if an application-specific mount with
143 a fixed quota has been created for example.
144 HUGETLB_SHARE=1
147 By default, libhugetlbfs uses unlinked hugetlbfs files to store
148 remapped program segment data. If the same program is started
149 multiple times using hugepage segments, multiple hugepages will
150 be used to store the same program data. The reduce this wastage,
151 setting this environment variable will share read-only segments
152 between multiple invocations of a program at the cost of the
153 memory being used whether the applications are running or not.
154 It is also possible that a malicious application inferfere with
155 other applications executable code. See the HOWTO for more
156 detailed information on this topic.
157 The following options control the verbosity of libhugetlbfs.
160 HUGETLB_VERBOSE=<level>
163 The default value for this is 1 and the range of the value is
164 from 0 to 99. The higher the value, the more verbose the output
165 is. 0 is quiet and 3 will output much debugging information.
166 HUGETLB_DEBUG
169 Once set, this will give very detailed output on what is happen‐
170 ing in the library and run extra diagnostics.
171
174 oprofile(1), hugectl(8), hugeedit(8), get_huge_pages(3),
175 free_huge_pages(3)
176
178 libhugetlbfs was written by various people on the libhugetlbfs-devel
179 mailing list.
180 September 27, 2008 LIBHUGETLBFS(7)