1HBWMALLOC(3) HBWMALLOC HBWMALLOC(3)
2
6 hbwmalloc - The high bandwidth memory interface
7 Note: hbwmalloc.h functionality is considered as stable API (STANDARD
8 API).
9
11 #include <hbwmalloc.h>
12 Link with -lmemkind
14 int hbw_check_available(void);
16 void* hbw_malloc(size_t size);
17 void* hbw_calloc(size_t nmemb, size_t size);
18 void* hbw_realloc (void *ptr, size_t size);
19 void hbw_free(void *ptr);
20 int hbw_posix_memalign(void **memptr, size_t alignment, size_t size);
21 int hbw_posix_memalign_psize(void **memptr, size_t alignment, size_t size, hbw_pagesize_t pagesize);
22 hbw_policy_t hbw_get_policy(void);
23 int hbw_set_policy(hbw_policy_t mode);
24 int hbw_verify_memory_region(void *addr, size_t size, int flags);
25
27 hbw_check_available() returns zero if high bandwidth memory is avail‐
28 able or an error code described in the ERRORS section if not.
29 hbw_malloc() allocates size bytes of uninitialized high bandwidth mem‐
31 ory. The allocated space is suitably aligned (after possible pointer
32 coercion) for storage of any type of object. If size is zero then
33 hbw_malloc() returns NULL.
34 hbw_calloc() allocates space for nmemb objects in high bandwidth mem‐
36 ory, each size bytes in length. The result is identical to calling
37 hbw_malloc() with an argument of nmemb * size, with the exception that
38 the allocated memory is explicitly initialized to zero bytes. If nmemb
39 or size is 0, then hbw_calloc() returns NULL.
40 hbw_realloc() changes the size of the previously allocated high band‐
42 width memory referenced by ptr to size bytes. The contents of the mem‐
43 ory remain unchanged up to the lesser of the new and old sizes. If the
44 new size is larger, the contents of the newly allocated portion of the
45 memory are undefined. Upon success, the memory referenced by ptr is
46 freed and a pointer to the newly allocated high bandwidth memory is
47 returned.
48 Note: hbw_realloc() may move the memory allocation, resulting in a dif‐
50 ferent return value than ptr.
51 If ptr is NULL, the hbw_realloc() function behaves identically to
53 hbw_malloc() for the specified size. If size is equal to zero, and ptr
54 is not NULL, then the call is equivalent to hbw_free(ptr) and NULL is
55 returned. The address ptr, if not NULL, was returned by a previous call
56 to hbw_malloc(), hbw_calloc(), hbw_realloc() or hbw_posix_memalign().
57 Otherwise, or if hbw_free(ptr) was called before, undefined behavior
58 occurs.
59 Note: hbw_realloc() cannot be used with a pointer returned by
61 hbw_posix_memalign_psize().
62 hbw_free() causes the allocated memory referenced by ptr to be made
65 available for future allocations. If ptr is NULL, no action occurs.
66 The address ptr, if not NULL, must have been returned by a previous
67 call to hbw_malloc(), hbw_calloc(), hbw_realloc(), hbw_posix_memalign()
68 or hbw_posix_memalign_psize(). Otherwise, if hbw_free(ptr) was called
69 before, undefined behavior occurs.
70 hbw_posix_memalign() allocates size bytes of high bandwidth memory such
72 that the allocation's base address is an even multiple of alignment,
73 and returns the allocation in the value pointed to by memptr. The
74 requested alignment must be a power of 2 at least as large as
75 sizeof(void*). If size is 0, then hbw_posix_memalign() returns 0, with
76 a NULL returned in memptr.
77 hbw_posix_memalign_psize() allocates size bytes of high bandwidth mem‐
79 ory such that the allocation's base address is an even multiple of
80 alignment, and returns the allocation in the value pointed to by
81 memptr. The requested alignment must be a power of 2 at least as large
82 as sizeof(void*). The memory will be allocated using pages determined
83 by the pagesize variable which may be one of the following enumerated
84 values:
85 HBW_PAGESIZE_4KB
87 The four kilobyte page size option. Note that with transparent
88 huge pages enabled these allocations may be promoted by the
89 operating system to two megabyte pages.
90 HBW_PAGESIZE_2MB
92 The two megabyte page size option. Note: This page size
93 requires huge pages configuration described in SYSTEM CONFIGURA‐
94 TION section.
95 HBW_PAGESIZE_1GB (DEPRECATED)
97 This option allows the user to specify arbitrary sizes backed by
98 1GB chunks of huge pages. Huge pages are allocated even if the
99 size is not a modulo of 1GB. Note: This page size requires huge
100 pages configuration described in SYSTEM CONFIGURATION section.
101 HBW_PAGESIZE_1GB_STRICT (DEPRECATED)
103 The total size of the allocation must be a multiple of 1GB with
104 this option, otherwise the allocation will fail. Note: This
105 page size requires huge pages configuration described in SYSTEM
106 CONFIGURATION section.
107 Note: HBW_PAGESIZE_2MB, HBW_PAGESIZE_1GB and HBW_PAGESIZE_1GB_STRICT
109 options are not supported with HBW_POLICY_INTERLEAVE policy.
110 hbw_get_policy() returns the current fallback policy when insufficient
112 high bandwidth memory is available.
113 hbw_set_policy() sets the current fallback policy. The policy can be
115 modified only once in the lifetime of an application and before calling
116 hbw_malloc(), hbw_calloc(), hbw_realloc(), hbw_posix_memalign(), or
117 hbw_posix_memalign_psize() function.
118 Note: If the policy is not set, than HBW_POLICY_PREFERRED will be used
119 by default.
120 HBW_POLICY_BIND
122 If insufficient high bandwidth memory from the nearest NUMA node
123 is available to satisfy a request, the allocated pointer is set
124 to NULL and errno is set to ENOMEM. If insufficient high band‐
125 width memory pages are available at fault time the Out Of Memory
126 (OOM) Killer is triggered. Note that pages are faulted exclu‐
127 sively from the high bandwidth NUMA node nearest at time of
128 allocation, not at time of fault.
129 HBW_POLICY_BIND_ALL
131 If insufficient high bandwidth memory is available to satisfy a
132 request, the allocated pointer is set to NULL and errno is set
133 to ENOMEM. If insufficient high bandwidth memory pages are
134 available at fault time the Out Of Memory (OOM) Killer is trig‐
135 gered. Note that pages are faulted from the high bandwidth NUMA
136 nodes. Nearest NUMA node is selected at time of page fault.
137 HBW_POLICY_PREFERRED
139 If insufficient memory is available from the high bandwidth NUMA
140 node closest at allocation time, fall back to standard memory
141 (default) with the smallest NUMA distance.
142 HBW_POLICY_INTERLEAVE
144 Interleave faulted pages from across all high bandwidth NUMA
145 nodes using standard size pages (the Transparent Huge Page fea‐
146 ture is disabled).
147 hbw_verify_memory_region() verifies if memory region fully falls into
149 high bandwidth memory. Returns 0 if memory address range from addr to
150 addr + size is allocated in high bandwidth memory, -1 if any fragment
151 of memory was not backed by high bandwidth memory (e.g. when memory is
152 not initialized) or one of error codes described in ERRORS section.
153 Using this function in production code may result in serious perfor‐
155 mance penalty.
156 The Flags argument may include optional flags that modify function
158 behavior:
159 HBW_TOUCH_PAGES
161 Before checking pages, function will touch first byte of all
162 pages in address range starting from addr to addr + size by read
163 and write (so the content will be overwritten by the same data
164 as it was read). Using this option may trigger Out Of Memory
165 Killer.
166
168 hbw_get_policy() returns HBW_POLICY_BIND, HBW_POLICY_BIND_ALL, HBW_POL‐
169 ICY_PREFERRED or HBW_POLICY_INTERLEAVE which represents the current
170 high bandwidth policy. hbw_free() do not have return value. hbw_mal‐
171 loc() hbw_calloc() and hbw_realloc() return the pointer to the allo‐
172 cated memory, or NULL if the request fails. hbw_posix_memalign(),
173 hbw_posix_memalign_psize() and hbw_set_policy() return zero on success
174 and return an error code as described in the ERRORS section below on
175 failure.
176
178 Error codes described here are the POSIX standard error codes as
179 defined in
180 <errno.h>
181 hbw_check_available()
183 returns ENODEV if high-bandwidth memory is unavailable.
184 hbw_posix_memalign() and hbw_posix_memalign_psize()
186 If the alignment parameter is not a power of two, or was not a
187 multiple of sizeof(void*), then EINVAL is returned. If the pol‐
188 icy and pagesize combination is unsupported then EINVAL is
189 returned. If there was insufficient memory to satisfy the
190 request then ENOMEM is returned.
191 hbw_set_policy()
193 returns EPERM if hbw_set_policy() was called more than once, or
194 EINVAL if mode argument was neither HBW_POLICY_PREFERRED,
195 HBW_POLICY_BIND, HBW_POLICY_BIND_ALL nor HBW_POLICY_INTERLEAVE.
196 hbw_verify_memory_region()
198 returns EINVAL if addr is NULL, size equals 0 or flags contained
199 unsupported bit set. If memory pointed by addr could not be ver‐
200 ified then EFAULT is returned.
201
203 The <hbwmalloc.h> file defines the external functions and enumerations
204 for the hbwmalloc library. These interfaces define a heap manager that
205 targets high bandwidth memory numa nodes.
206
208 /usr/bin/memkind-hbw-nodes
209 Prints a comma separated list of high bandwidth nodes.
210
212 MEMKIND_HBW_NODES
213 This environment variable is a comma separated list of NUMA
214 nodes that are treated as high bandwidth. Uses the libnuma rou‐
215 tine numa_parse_nodestring() for parsing, so the syntax
216 described in the numa(3) man page for this routine applies for
217 example: 1-3,5 is a valid setting.
218 MEMKIND_ARENA_NUM_PER_KIND
220 This environment variable allows leveraging internal mechanism
221 of the library for setting number of arenas per kind. Value
222 should be a positive integer (not greater than INT_MAX defined
223 in <limits.h>). The user should set the value based on the
224 characteristics of application that is using the library. Higher
225 value can provide better performance in extremely multithreaded
226 applications at the cost of memory overhead. See section IMPLE‐
227 MENTATION NOTES of jemalloc(3) for more details about arenas.
228 MEMKIND_HEAP_MANAGER
230 Controls heap management behavior in memkind library by switch‐
231 ing to one of the available heap managers.
232 Values:
233 JEMALLOC - sets the jemalloc heap manager
234 TBB - sets the Intel Threading Building Blocks heap manager.
235 This option requires installed
236 Intel Threading Building Blocks library.
237 Note: If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap
239 manager will be used by default.
240
242 Interfaces for obtaining 2MB (HUGETLB) need allocated huge pages in the
243 kernel's huge page pool.
244 HUGETLB (huge pages)
246 Current number of "persistent" huge pages can be read from
247 /proc/sys/vm/nr_hugepages file. Proposed way of setting
248 hugepages is: sudo sysctl vm.nr_hugepages=<number_of_hugepages>.
249 More information can be found here:
250 ⟨https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt⟩
251
253 HUGETLB (huge pages)
254 There might be some overhead in huge pages consumption caused by
255 heap management. If your allocation fails because of OOM,
256 please try to allocate extra huge pages (e.g. 8 huge pages).
257
259 Copyright (C) 2014 - 2019 Intel Corporation. All rights reserved.
260
262 malloc(3), numa(3), numactl(8), mbind(2), mmap(2), move_pages(2),
263 jemalloc(3), memkind(3)
264Intel Corporation 2015-03-31 HBWMALLOC(3)