1MEMKIND(3) MEMKIND MEMKIND(3)
2
6 memkind - Heap manager that enables allocations to memory with differ‐
7 ent properties.
8 This header expose EXPERIMENTAL API in except of STANDARD API placed in
9 section LIBRARY VERSION. API Standards are described below in this man
10 page.
11
13 #include <memkind.h>
14 Link with -lmemkind
16 ERROR HANDLING:
18 void memkind_error_message(int err, char *msg, size_t size);
19 HEAP MANAGEMENT:
21 void *memkind_malloc(memkind_t kind, size_t size);
22 void *memkind_calloc(memkind_t kind, size_t num, size_t size);
23 void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
24 int memkind_posix_memalign(memkind_t kind, void **memptr, size_t alignment, size_t size);
25 void memkind_free(memkind_t kind, void *ptr);
26 KIND MANAGMENT:
28 int memkind_create_pmem(const char *dir, size_t max_size, memkind_t *kind);
29 int memkind_check_available(memkind_t kind);
30 DECORATORS:
32 void memkind_malloc_pre(memkind_t *kind, size_t *size);
33 void memkind_malloc_post(memkind_t kind, size_t size, void **result);
34 void memkind_calloc_pre(memkind_t *kind, size_t *nmemb, size_t *size);
35 void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size, void **result);
36 void memkind_posix_memalign_pre(memkind_t *kind, void **memptr, size_t *alignment, size_t *size);
37 void memkind_posix_memalign_post(memkind_t kind, void **memptr, size_t alignment, size_t size, int *err);
38 void memkind_realloc_pre(memkind_t *kind, void **ptr, size_t *size);
39 void memkind_realloc_post(memkind_t kind, void *ptr, size_t size, void **result);
40 void memkind_free_pre(memkind_t *kind, void **ptr);
41 void memkind_free_post(memkind_t kind, void *ptr);
42 LIBRARY VERSION:
44 int memkind_get_version();
45
48 memkind_error_message() converts an error number err returned by a mem‐
49 ber of the memkind interface to an error message msg where the maximum
50 size of the message is passed by the size parameter.
51 HEAP MANAGMENT:
53 The functions described in this section define a heap manager with an
54 interface modeled on the ISO C standard API's, except that the user
55 must specify the kind of memory with the first argument to each func‐
56 tion. See the KINDS section below for a full description of the imple‐
57 mented kinds.
58 memkind_malloc() allocates size bytes of uninitialized memory of the
60 specified kind. The allocated space is suitably aligned (after possi‐
61 ble pointer coercion) for storage of any type of object. If size is 0,
62 then memkind_malloc() returns NULL.
63 memkind_calloc() allocates space for num objects each size bytes in
65 length in memory of the specified kind. The result is identical to
66 calling memkind_malloc() with an argument of num*size, with the excep‐
67 tion that the allocated memory is explicitly initialized to zero bytes.
68 If num or size is 0, then memkind_calloc() returns NULL.
69 memkind_realloc() changes the size of the previously allocated memory
71 referenced by ptr to size bytes of the specified kind. The contents of
72 the memory are unchanged up to the lesser of the new and old sizes. If
73 the new size is larger, the contents of the newly allocated portion of
74 the memory are undefined. Upon success, the memory referenced by ptr is
75 freed and a pointer to the newly allocated high bandwidth memory is
76 returned.
77 Note: memkind_realloc() may move the memory allocation, resulting in a
79 different return value than ptr.
80 If ptr is NULL, the memkind_realloc() function behaves identically to
82 memkind_malloc() for the specified size. The address ptr, if not NULL,
83 must have been returned by a previous call to memkind_malloc(),
84 memkind_calloc(), memkind_realloc(), or memkind_posix_memalign() with
85 the same kind as specified to the call to memkind_realloc(). Other‐
86 wise, if memkind_free(kind, ptr) was called before, undefined behavior
87 occurs.
88 memkind_posix_memalign() allocates size bytes of memory of a specified
90 kind such that the allocation's base address is an even multiple of
91 alignment, and returns the allocation in the value pointed to by
92 memptr. The requested alignment must be a power of 2 at least as large
93 as sizeof(void *). If size is 0, then memkind_posix_memalign() returns
94 NULL.
95 memkind_free() causes the allocated memory referenced by ptr to be made
97 available for future allocations. This pointer must have been returned
98 by a previous call to memkind_malloc(), memkind_calloc(), memkind_real‐
99 loc(), or memkind_posix_memalign(). Otherwise, if memkind_free(kind,
100 ptr) was already called before, undefined behavior occurs. If ptr is
101 NULL, no operation is performed. The value of MEMKIND_DEFAULT can be
102 given as the kind for all buffers allocated by a kind that leverages
103 the jemalloc allocator. In cases where the kind is unknown in the con‐
104 text of the call to memkind_free() 0 can be given as the kind specified
105 to memkind_free() but this will require a look up that can be bypassed
106 by specifying a non-zero value.
107 KIND MANAGEMENT:
109 There are built-in kinds that are always available, and these are enu‐
110 merated in the KINDS section. The user can also create their own kinds
111 of memory. This section describes the API's that enable the tracking
112 of the different kinds of memory and determining their properties.
113 memkind_create_pmem() is a convenience function used to create a file-
115 backed kind of memory. It allocates a temporary file in the given
116 directory dir. The file is created in a fashion similar to tmpfile(3),
117 so that the file name does not appear when the directory is listed and
118 the space is automatically freed when the program terminates. The file
119 is truncated to a size of max_size bytes and the resulting space is
120 memory-mapped.
121 Note that the actual file system space is not allocated immediately,
122 but only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This
123 allows to create a pmem memkind of a pretty large size without the need
124 to reserve in advance the corresponding file system space for the
125 entire heap. The minimum max_size value allowed by the library is
126 defined in <memkind_pmem.h> as MEMKIND_PMEM_MIN_SIZE. Calling
127 memkind_create_pmem() with a size smaller than that will return an
128 error. The maximum allowed size is not limited by memkind, but by the
129 file system specified by the dir argument. The max_size passed in is
130 the raw size of the memory pool and jemalloc will use some of that
131 space for its own metadata.
132 memkind_check_available() Returns a zero if the specified kind is
134 available or an error code from the ERRORS section if it is not.
135 DECORATORS:
137 The memkind library enables the user to define decorator functions that
138 can be called before and after each memkind heap management API. The
139 decorators that are called at the beginning of the function end are
140 named after that function with _pre appended to the name, and those
141 that are called at the end of the function are named after that func‐
142 tion with _post appended to the name. These are weak symbols, and if
143 they are not present at link time they are not called. The memkind
144 library does not define these symbols which are reserved for user defi‐
145 nition. These decorators can be used to track calls to the heap man‐
146 agement interface or to modify parameters. The decorators that are
147 called at the beginning of the allocator pass all inputs by reference,
148 and the decorators that are called at the end of the allocator pass the
149 output by reference. This enables the modification of the input and
150 output of each heap management function by the decorators.
151 LIBRARY VERSION
153 The memkind library version scheme consist major, minor and patch num‐
154 bers separated by dot. Combining those numbers, we got the following
155 representation:
156 major.minor.patch, where:
157 -major number is incremented whenever API is changed (loss of
158 backward compatibility),
159 -minor number is incremented whenever additional extensions are
160 introduced, or behavior has been changed,
161 -patch number is incremented whenever small bug fixes are added.
162 memkind library provide numeric representation of the version by expos‐
164 ing the following API:
165 int memkind_get_version() return version number represented by a single
166 integer number, obtained from the formula:
167 major * 1000000 + minor * 1000 + patch
168 Note: major < 1 means unstable API.
170 API standards:
172 -STANDARD API, API is considered as stable
173 -NON-STANDARD API, API is considered as stable, however this is not a
174 standard way to use memkind
175 -EXPERIMENTAL API, API is considered as unstable and the subject to
176 change
177
180 memkind_calloc(), memkind_malloc(), and memkind_realloc(), return the
181 pointer to the allocated memory, or NULL if the request fails.
182 memkind_free() and memkind_error_message() do not have return values.
183 All other memkind API's return 0 upon success, and an error code
184 defined in the ERRORS section upon failure. The memkind library avoids
185 setting errno directly, but calls to underlying libraries and system
186 calls may set errno.
187
189 The available kinds of memory
190 MEMKIND_DEFAULT
192 Default allocation using standard memory and default page size.
193 MEMKIND_HUGETLB
195 Allocate from standard memory using huge pages. Note: This kind
196 requires huge pages configuration described in SYSTEM CONFIGURA‐
197 TION section.
198 MEMKIND_GBTLB (DEPRECATED)
200 Allocate from standard memory using 1GB chunks backed by huge
201 pages. Note: This kind requires huge pages configuration
202 described in SYSTEM CONFIGURATION section.
203 MEMKIND_INTERLEAVE
205 Allocate pages interleaved across all NUMA nodes with transpar‐
206 ent huge pages disabled.
207 MEMKIND_HBW
209 Allocate from the closest high bandwidth memory NUMA node at
210 time of allocation. If there is not enough high bandwidth memory
211 to satisfy the request errno is set to ENOMEM and the allocated
212 pointer is set to NULL.
213 MEMKIND_HBW_ALL
215 Same as MEMKIND_HBW except decision regarding closest NUMA node
216 is postponed until the time of first write.
217 MEMKIND_HBW_HUGETLB
219 Same as MEMKIND_HBW except the allocation is backed by huge
220 pages. Note: This kind requires huge pages configuration
221 described in SYSTEM CONFIGURATION section.
222 MEMKIND_HBW_ALL_HUGETLB
224 Combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB proper‐
225 ties. Note: This kind requires huge pages configuration
226 described in SYSTEM CONFIGURATION section.
227 MEMKIND_HBW_PREFERRED
229 Same as MEMKIND_HBW except that if there is not enough high
230 bandwidth memory to satisfy the request, the allocation will
231 fall back on standard memory.
232 MEMKIND_HBW_PREFERRED_HUGETLB
234 Same as MEMKIND_HBW_PREFERRED except the allocation is backed by
235 huge pages. Note: This kind requires huge pages configuration
236 described in SYSTEM CONFIGURATION section.
237 MEMKIND_HBW_GBTLB (DEPRECATED)
239 Same as MEMKIND_HBW except the allocation is backed by 1GB
240 chunks of huge pages. Note that size can take on any value, but
241 full gigabyte pages will allocated for each request, so remain‐
242 der of the last page will be wasted. This kind requires huge
243 pages configuration described in SYSTEM CONFIGURATION section.
244 MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED)
246 Same as MEMKIND_HBW_GBTLB except that if there is not enough
247 high bandwidth memory to satisfy the request, the allocation
248 will fall back on standard memory. Note: This kind requires huge
249 pages configuration described in SYSTEM CONFIGURATION section.
250 MEMKIND_HBW_INTERLEAVE
252 Same as MEMKIND_HBW except that the pages that support the allo‐
253 cation are interleaved across all high bandwidth nodes and
254 transparent huge pages are disabled.
255 MEMKIND_REGULAR
257 Allocate from regular memory using the default page size. Regu‐
258 lar means general purpose memory from the NUMA nodes containing
259 CPUs.
260
262 memkind_posix_memalign()
263 returns the one of the POSIX standard error codes EINVAL or
264 ENOMEM as defined in <errno.h> if an error occurs (these have
265 positive values). If the alignment parameter is not a power of
266 two, or is not a multiple of sizoeof(void *), then EINVAL is
267 returned. If there is insufficient memory to satisfy the
268 request then ENOMEM is returned.
269 All functions other than memkind_posix_memalign() which have an integer
271 return type return one of the negative error codes as defined in
272 <memkind.h> and described below.
273 MEMKIND_ERROR_UNAVAILABLE
275 Requested memory kind is not available
276 MEMKIND_ERROR_MBIND
278 Call to mbind(2) failed
279 MEMKIND_ERROR_MMAP
281 Call to mmap(2) failed
282 MEMKIND_ERROR_MALLOC
284 Call to jemalloc's malloc() failed
285 MEMKIND_ERROR_ALLOCM
287 Call to jemalloc's allocm() failed
288 MEMKIND_ERROR_ENVIRON
290 Error parsing environment variable (MEMKIND_*)
291 MEMKIND_ERROR_INVALID
293 Invalid input arguments to memkind routine
294
296 /usr/bin/memkind-hbw-nodes
297 Prints a comma separated list of high bandwidth nodes.
298
300 MEMKIND_HBW_NODES
301 This environment variable is a comma separated list of NUMA
302 nodes that are treated as high bandwidth. Uses the libnuma rou‐
303 tine numa_parse_nodestring() for parsing, so the syntax
304 described in the numa(3) man page for this routine applies: e.g
305 1-3,5 is a valid setting.
306 MEMKIND_ARENA_NUM_PER_KIND
308 This environment variable allows leveraging internal mechanism
309 of the library for setting number of arenas per kind. Value
310 should be a positive integer (not greater than INT_MAX defined
311 in limits.h). The user should set the value based on the char‐
312 acteristics of application that is using the library. Higher
313 value can provide better performance in extremely multithreaded
314 applications at the cost of memory overhead. See section "IMPLE‐
315 MENTATION NOTES" of jemalloc(3) for more details about arenas.
316 MEMKIND_HOG_MEMORY
318 Controls behavior of memkind with regards to returning memory to
319 underlaying OS. Setting MEMKIND_HOG_MEMORY to "1" causes memkind
320 to not release memory to OS in anticipation of memory reuse
321 soon. This will improve latency of 'free' operations but
322 increase memory usage.
323 MEMKIND_DEBUG
325 Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to
326 "1" enables printing messages like errors and general informa‐
327 tions about environment to stderr.
328 MEMKIND_HEAP_MANAGER
330 Controls heap management behavior in memkind library by switch‐
331 ing to one of the available heap managers.
332 Values:
333 JEMALLOC – sets the jamalloc heap manager
334 TBB – sets the Intel Threading Building Blocks heap manager.
335 This option requires installed
336 Intel Threading Building Blocks library. If the
337 MEMKIND_HEAP_MANAGER is not set than the jemalloc heap manager
338 will be used by default.
339
341 Interfaces for obtaining 2MB (HUGETLB) need allocated huge pages in the
342 kernel's huge page pool.
343 HUGETLB (huge pages)
345 Current number of "persistent" huge pages can be read from
346 /proc/sys/vm/nr_hugepages file. Proposed way of setting
347 hugepages is: "sudo sysctl vm.nr_hugepages=<num‐
348 ber_of_hugepages>". More informations can be found here:
349 https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
350
352 When linking statically against memkind, libmemkind.a should be used
353 together with its dependencies libnuma and pthread. Pthread can be
354 linked by adding /usr/lib64/libpthread.a as a dependency (exact path
355 may vary). Typically libnuma will need to be compiled from sources to
356 use it as a static dependency. libnuma can be reached on github:
357 https://github.com/numactl/numactl
358
360 HUGETLB (huge pages)
361 There might be some overhead in huge pages consumption caused by
362 heap management. If your allocation fails because of OOM,
363 please try to allocate extra huge pages (e.g. 8 huge pages).
364
366 Copyright (C) 2014 - 2017 Intel Corporation. All rights reserved.
367
369 malloc(3), numa(3), numactl(8), mbind(2), mmap(2), move_pages(2),
370 jemalloc(3), memkind_default(3), memkind_arena(3), memkind_hbw(3),
371 memkind_hugetlb(3), memkind_pmem(3)
372Intel Corporation 2015-03-31 MEMKIND(3)