1MEMKIND(3) MEMKIND MEMKIND(3)
2
6 memkind - Heap manager that enables allocations to memory with differ‐
7 ent properties.
8 This header expose STANDARD and EXPERIMENTAL API. API Standards are
9 described below in this man page.
10
12 #include <memkind.h>
13 Link with -lmemkind
15 EXPERIMENTAL API:
17 HEAP MANAGEMENT:
18 int memkind_posix_memalign(memkind_t kind, void **memptr, size_t align‐
19 ment, size_t size);
20 KIND MANAGEMENT:
22 int memkind_create_kind(memkind_memtype_t memtype_flags, memkind_pol‐
23 icy_t policy, memkind_bits_t flags, memkind_t *kind);
24 STANDARD API:
27 ERROR HANDLING:
28 void memkind_error_message(int err, char *msg, size_t size);
29 LIBRARY VERSION:
31 int memkind_get_version();
32 HEAP MANAGEMENT:
34 void *memkind_malloc(memkind_t kind, size_t size);
35 void *memkind_calloc(memkind_t kind, size_t num, size_t size);
36 void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
37 void memkind_free(memkind_t kind, void *ptr);
38 size_t memkind_malloc_usable_size(memkind_t kind, void *ptr);
39 memkind_t memkind_detect_kind(void *ptr);
40 KIND CONFIGURATION MANAGEMENT:
42 struct memkind_config *memkind_config_new();
43 void memkind_config_delete(struct memkind_config *cfg);
44 void memkind_config_set_path(struct memkind_config *cfg, const char
45 *pmem_dir);
46 void memkind_config_set_size(struct memkind_config *cfg, size_t
47 pmem_size);
48 void memkind_config_set_memory_usage_policy(struct memkind_config *cfg,
49 memkind_mem_usage_policy policy);
50 KIND MANAGEMENT:
52 int memkind_create_pmem(const char *dir, size_t max_size, memkind_t
53 *kind);
54 int memkind_create_pmem_with_config(struct memkind_config *cfg,
55 memkind_t *kind);
56 int memkind_destroy_kind(memkind_t kind);
57 int memkind_check_available(memkind_t kind);
58 DECORATORS:
60 void memkind_malloc_pre(memkind_t *kind, size_t *size);
61 void memkind_malloc_post(memkind_t kind, size_t size, void **result);
62 void memkind_calloc_pre(memkind_t *kind, size_t *nmemb, size_t *size);
63 void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size,
64 void **result);
65 void memkind_posix_memalign_pre(memkind_t *kind, void **memptr, size_t
66 *alignment, size_t *size);
67 void memkind_posix_memalign_post(memkind_t kind, void **memptr, size_t
68 alignment, size_t size, int *err);
69 void memkind_realloc_pre(memkind_t *kind, void **ptr, size_t *size);
70 void memkind_realloc_post(memkind_t *kind, void *ptr, size_t size, void
71 **result);
72 void memkind_free_pre(memkind_t *kind, void **ptr);
73 void memkind_free_post(memkind_t kind, void *ptr);
74
78 memkind_error_message() converts an error number err returned by a mem‐
79 ber of the memkind interface to an error message msg where the maximum
80 size of the message is passed by the size parameter.
81 HEAP MANAGEMENT:
83 The functions described in this section define a heap manager with an
84 interface modeled on the ISO C standard API's, except that the user
85 must specify the kind of memory with the first argument to each func‐
86 tion. See the KINDS section below for a full description of the imple‐
87 mented kinds. For file-backed kind of memory see memkind_create_pmem()
88 or memkind_create_pmem_with_config().
89 memkind_malloc() allocates size bytes of uninitialized memory of the
91 specified kind. The allocated space is suitably aligned (after possi‐
92 ble pointer coercion) for storage of any type of object. If size is 0,
93 then memkind_malloc() returns NULL.
94 memkind_calloc() allocates space for num objects each size bytes in
96 length in memory of the specified kind. The result is identical to
97 calling memkind_malloc() with an argument of num * size, with the
98 exception that the allocated memory is explicitly initialized to zero
99 bytes. If num or size is 0, then memkind_calloc() returns NULL.
100 memkind_realloc() changes the size of the previously allocated memory
102 referenced by ptr to size bytes of the specified kind. The contents of
103 the memory remain unchanged up to the lesser of the new and old sizes.
104 If the new size is larger, the contents of the newly allocated portion
105 of the memory are undefined. Upon success, the memory referenced by ptr
106 is freed and a pointer to the newly allocated memory is returned.
107 Note: memkind_realloc() may move the memory allocation, resulting in a
109 different return value than ptr.
110 If ptr is NULL, the memkind_realloc() function behaves identically to
112 memkind_malloc() for the specified size. If size is equal to zero, and
113 ptr is not NULL, then the call is equivalent to memkind_free(kind, ptr)
114 and NULL is returned. The address ptr, if not NULL, must have been
115 returned by a previous call to memkind_malloc(), memkind_calloc(),
116 memkind_realloc() or memkind_posix_memalign() with the same kind as
117 specified to the call to memkind_realloc(). Otherwise, if
118 memkind_free(kind, ptr) was called before, undefined behavior occurs.
119 In cases where the kind is unknown in the context of the call to
120 memkind_realloc() NULL, can be given as the kind specified to
121 memkind_realloc(), but this will require a internal look up for correct
122 kind. Note: The look up for kind could result in serious performance
123 penalty, which can be avoided by specifying a correct kind. If kind is
124 NULL and ptr is NULL, then memkind_realloc() returns NULL and sets
125 errno to EINVAL.
126 memkind_posix_memalign() allocates size bytes of memory of a specified
128 kind such that the allocation's base address is an even multiple of
129 alignment, and returns the allocation in the value pointed to by
130 memptr. The requested alignment must be a power of 2 at least as large
131 as sizeof(void*). If size is 0, then memkind_posix_memalign() returns
132 0, with a NULL returned in memptr.
133 memkind_malloc_usable_size() function provides the same semantics as
135 malloc_usable_size(3), but operates on specified kind.
136 Note: In cases where the kind is unknown in the context of the call to
137 memkind_malloc_usable_size() NULL, can be given as the kind specified
138 to memkind_malloc_usable_size(), but this could require a internal look
139 up for correct kind. memkind_malloc_usable_size() is supported by TBB
140 heap manager described in ENVIRONMENT section since Intel TBB 2019
141 Update 4.
142 memkind_detect_kind() returns the kind associated with allocated memory
144 referenced by ptr. This pointer must have been returned by a previous
145 call to memkind_malloc(), memkind_calloc(), memkind_realloc() or
146 memkind_posix_memalign(). If ptr is NULL, then memkind_detect_kind()
147 returns NULL. Note: This function has non-trivial performance over‐
148 head.
149 memkind_free() causes the allocated memory referenced by ptr to be made
151 available for future allocations. This pointer must have been returned
152 by a previous call to memkind_malloc(), memkind_calloc(), memkind_real‐
153 loc() or memkind_posix_memalign(). Otherwise, if memkind_free(kind,
154 ptr) was already called before, undefined behavior occurs. If ptr is
155 NULL, no operation is performed. In cases where the kind is unknown in
156 the context of the call to memkind_free() NULL, can be given as the
157 kind specified to memkind_free(), but this will require a internal look
158 up for correct kind. Note: The look up for kind could result in seri‐
159 ous performance penalty, which can be avoided by specifying a correct
160 kind.
161 KIND CONFIGURATION MANAGEMENT:
163 The functions described in this section define a way to create, delete
164 and update kind specific configuration. Except of memkind_con‐
165 fig_new(), user must specify the memkind configuration with the first
166 argument to each function. API described here is most useful with
167 file-backed kind of memory, e.g. memkind_create_pmem_with_config()
168 method.
169 memkind_config_new() creates the memkind configuration.
171 memkind_config_delete() deletes previously created memkind configura‐
173 tion, which must have been returned by a previous call to memkind_con‐
174 fig_new().
175 memkind_config_set_path() updates the memkind pmem_dir configuration
177 parameter, which specifies directory path, where file-backed kind of
178 memory will be created. Note: This function does not validate that
179 pmem_dir specifies a valid path.
180 memkind_config_set_size() updates the memkind pmem_size configuration
182 parameter, which allows to limit the file-backed kind memory partition.
183 Note: This function does not validate that pmem_size is in valid range.
184 memkind_config_set_memory_usage_policy() updates the memkind policy
186 configuration parameter, which allows to tune up memory utilization.
187 The user should set the value based on the characteristics of applica‐
188 tion that is using the library (e.g. prioritize memory usage, CPU uti‐
189 lization), for more details about policy see the MEMORY USAGE POLICY
190 section below. Note: This function does not validate that policy is in
191 valid range.
192 KIND MANAGEMENT:
194 There are built-in kinds that are always available and these are enu‐
195 merated in the KINDS section. The user can also create their own kinds
196 of memory. This section describes the API's that enable the tracking of
197 the different kinds of memory and determining their properties.
198 memkind_create_pmem() is a convenient function used to create a file-
200 backed kind of memory. It allocates a temporary file in the given
201 directory dir. The file is created in a fashion similar to tmpfile(3),
202 so that the file name does not appear when the directory is listed and
203 the space is automatically freed when the program terminates. The file
204 is truncated to a size of max_size bytes and the resulting space is
205 memory-mapped.
206 Note that the actual file system space is not allocated immediately,
207 but only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This
208 allows to create a pmem memkind of a pretty large size without the need
209 to reserve in advance the corresponding file system space for the
210 entire heap. If the value of max_size equals 0, pmem memkind is only
211 limited by the capacity of the file system mounted under dir argument.
212 The minimum max_size value which allows to limit the size of kind by
213 the library is defined as MEMKIND_PMEM_MIN_SIZE. Calling memkind_cre‐
214 ate_pmem() with a size smaller than that and different than 0 will
215 return an error. The maximum allowed size is not limited by memkind,
216 but by the file system specified by the dir argument. The max_size
217 passed in is the raw size of the memory pool and jemalloc will use some
218 of that space for its own metadata. Returns zero if the pmem memkind
219 is created successfully or an error code from the ERRORS section if
220 not.
221 memkind_create_pmem_with_config() is a second function used to create a
223 file-backed kind of memory. Function behaves simillar to memkind_cre‐
224 ate_pmem() but instead of passing dir and max_size arguments, it uses
225 config param to specify characteristics of created file-backed kind of
226 memory (see KIND CONFIGURATION MANAGEMENT section).
227 memkind_create_kind() creates kind that allocates memory with specific
229 memory type, memory binding policy and flags (see MEMORY FLAGS sec‐
230 tion). The memtype_flags (see MEMORY TYPES section) determine memory
231 types to allocate, policy argument is policy for specifying page bind‐
232 ing to memory types selected by memtype_flags. Returns zero if the
233 specified kind is created successfully or an error code from the ERRORS
234 section if not.
235 memkind_destroy_kind() destroys previously created kind object, which
237 must have been returned by a previous call to memkind_create_pmem(),
238 memkind_create_pmem_with_config() or memkind_create_kind(). Otherwise,
239 or if memkind_destroy_kind(kind) was already called before, undefined
240 behavior occurs. Note that, when the kind was returned by memkind_cre‐
241 ate_kind() all allocated memory must be freed before kind is destroyed,
242 otherwise this will cause memory leak. When the kind was returned by
243 memkind_create_pmem() or memkind_create_pmem_with_config() all allo‐
244 cated memory will be freed after kind will be destroyed.
245 memkind_check_available() returns zero if the specified kind is avail‐
247 able or an error code from the ERRORS section if it is not.
248 MEMKIND_PMEM_MIN_SIZE The minimum size which allows to limit the file-
250 backed memory partition.
251 DECORATORS:
253 The memkind library enables the user to define decorator functions that
254 can be called before and after each memkind heap management API. The
255 decorators that are called at the beginning of the function end are
256 named after that function with _pre appended to the name and those that
257 are called at the end of the function are named after that function
258 with _post appended to the name. These are weak symbols and if they are
259 not present at link time they are not called. The memkind library does
260 not define these symbols which are reserved for user definition. These
261 decorators can be used to track calls to the heap management interface
262 or to modify parameters. The decorators that are called at the begin‐
263 ning of the allocator pass all inputs by reference and the decorators
264 that are called at the end of the allocator pass the output by refer‐
265 ence. This enables the modification of the input and output of each
266 heap management function by the decorators.
267 LIBRARY VERSION:
269 The memkind library version scheme consist major, minor and patch num‐
270 bers separated by dot. Combining those numbers, we got the following
271 representation:
272 major.minor.patch, where:
273 -major number is incremented whenever API is changed (loss of
274 backward compatibility),
275 -minor number is incremented whenever additional extensions are
276 introduced or behavior has been changed,
277 -patch number is incremented whenever small bug fixes are added.
278 memkind library provide numeric representation of the version by expos‐
280 ing the following API:
281 memkind_get_version() returns version number represented by a single
283 integer number, obtained from the formula:
284 major * 1000000 + minor * 1000 + patch
285 Note: major < 1 means unstable API.
287 API standards:
289 -STANDARD API, API is considered as stable
290 -NON-STANDARD API, API is considered as stable, however this is not a
291 standard way to use memkind
292 -EXPERIMENTAL API, API is considered as unstable and the subject to
293 change
294
297 memkind_calloc(), memkind_malloc() and memkind_realloc() returns the
298 pointer to the allocated memory or NULL if the request fails.
299 memkind_malloc_usable_size() returns the number of usable bytes in the
300 block of allocated memory pointed to by ptr, a pointer to a block of
301 memory allocated by memkind_malloc() or a related function. If ptr is
302 NULL, 0 is returned. memkind_free() and memkind_error_message() do not
303 have return values. All other memkind API's return 0 upon success and
304 an error code defined in the ERRORS section upon failure. The memkind
305 library avoids setting errno directly, but calls to underlying
306 libraries and system calls may set errno (e.g. memkind_create_pmem()).
307
309 The available kinds of memory:
310 MEMKIND_DEFAULT
312 Default allocation using standard memory and default page size.
313 MEMKIND_HUGETLB
315 Allocate from standard memory using huge pages. Note: This kind
316 requires huge pages configuration described in SYSTEM CONFIGURA‐
317 TION section.
318 MEMKIND_GBTLB (DEPRECATED)
320 Allocate from standard memory using 1GB chunks backed by huge
321 pages. Note: This kind requires huge pages configuration
322 described in SYSTEM CONFIGURATION section.
323 MEMKIND_INTERLEAVE
325 Allocate pages interleaved across all NUMA nodes with transpar‐
326 ent huge pages disabled.
327 MEMKIND_HBW
329 Allocate from the closest high bandwidth memory NUMA node at
330 time of allocation. If there is not enough high bandwidth memory
331 to satisfy the request errno is set to ENOMEM and the allocated
332 pointer is set to NULL.
333 MEMKIND_HBW_ALL
335 Same as MEMKIND_HBW except decision regarding closest NUMA node
336 is postponed until the time of first write.
337 MEMKIND_HBW_HUGETLB
339 Same as MEMKIND_HBW except the allocation is backed by huge
340 pages. Note: This kind requires huge pages configuration
341 described in SYSTEM CONFIGURATION section.
342 MEMKIND_HBW_ALL_HUGETLB
344 Combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB proper‐
345 ties. Note: This kind requires huge pages configuration
346 described in SYSTEM CONFIGURATION section.
347 MEMKIND_HBW_PREFERRED
349 Same as MEMKIND_HBW except that if there is not enough high
350 bandwidth memory to satisfy the request, the allocation will
351 fall back on standard memory.
352 MEMKIND_HBW_PREFERRED_HUGETLB
354 Same as MEMKIND_HBW_PREFERRED except the allocation is backed by
355 huge pages. Note: This kind requires huge pages configuration
356 described in SYSTEM CONFIGURATION section.
357 MEMKIND_HBW_GBTLB (DEPRECATED)
359 Same as MEMKIND_HBW except the allocation is backed by 1GB
360 chunks of huge pages. Note that size can take on any value, but
361 full gigabyte pages will allocated for each request, so remain‐
362 der of the last page will be wasted. This kind requires huge
363 pages configuration described in SYSTEM CONFIGURATION section.
364 MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED)
366 Same as MEMKIND_HBW_GBTLB except that if there is not enough
367 high bandwidth memory to satisfy the request, the allocation
368 will fall back on standard memory. Note: This kind requires
369 huge pages configuration described in SYSTEM CONFIGURATION sec‐
370 tion.
371 MEMKIND_HBW_INTERLEAVE
373 Same as MEMKIND_HBW except that the pages that support the allo‐
374 cation are interleaved across all high bandwidth nodes and
375 transparent huge pages are disabled.
376 MEMKIND_REGULAR
378 Allocate from regular memory using the default page size. Regu‐
379 lar means general purpose memory from the NUMA nodes containing
380 CPUs.
381
383 The available types of memory:
384 MEMKIND_MEMTYPE_DEFAULT
386 Standard memory, the same as process uses.
387 MEMKIND_MEMTYPE_HIGH_BANDWIDTH
389 High bandwidth memory (HBM). There must be at least two memory
390 types with different bandwidth to determine which is the HBM.
391
393 The available types of memory binding policy:
394 MEMKIND_POLICY_BIND_LOCAL
396 Allocate local memory. If there is not enough memory to satisfy
397 the request errno is set to ENOMEM and the allocated pointer is
398 set to NULL.
399 MEMKIND_POLICY_BIND_ALL
401 Memory locality is ignored. If there is not enough memory to
402 satisfy the request errno is set to ENOMEM and the allocated
403 pointer is set to NULL.
404 MEMKIND_POLICY_PREFERRED_LOCAL
406 Allocate preferred memory that is local. If there is not enough
407 preferred memory to satisfy the request or preferred memory is
408 not available, the allocation will fall back on any other mem‐
409 ory.
410 MEMKIND_POLICY_INTERLEAVE_LOCAL
412 Interleave allocation across local memory. For n memory types
413 the allocation will be interleaved across all of them.
414 MEMKIND_POLICY_INTERLEAVE_ALL
416 Interleave allocation. Locality is ignored. For n memory types
417 the allocation will be interleaved across all of them.
418 MEMKIND_POLICY_MAX_VALUE
420 Max policy value.
421
423 The available types of memory flags:
424 MEMKIND_MASK_PAGE_SIZE_2MB
426 Allocation backed by 2MB page size.
427
429 The available types of memory usage policy:
430 MEMKIND_MEM_USAGE_POLICY_DEFAULT
432 Default memory usage policy.
433 MEMKIND_MEM_USAGE_POLICY_CONSERVATIVE
435 Conservative memory usage policy - prioritize memory usage at
436 cost of performance. Note: Memory usage policies have no effect
437 for TBB heap manager described in ENVIRONMENT section.
438
440 memkind_posix_memalign()
441 returns the one of the POSIX standard error codes EINVAL or
442 ENOMEM as defined in <errno.h> if an error occurs (these have
443 positive values). If the alignment parameter is not a power of
444 two or is not a multiple of sizeof(void*), then EINVAL is
445 returned. If there is insufficient memory to satisfy the request
446 then ENOMEM is returned.
447 All functions other than memkind_posix_memalign() which have an integer
449 return type return one of the negative error codes as defined in
450 <memkind.h> and described below.
451 MEMKIND_ERROR_UNAVAILABLE
453 Requested memory kind is not available
454 MEMKIND_ERROR_MBIND
456 Call to mbind(2) failed
457 MEMKIND_ERROR_MMAP
459 Call to mmap(2) failed
460 MEMKIND_ERROR_MALLOC
462 Call to jemalloc's malloc() failed
463 MEMKIND_ERROR_ENVIRON
465 Error parsing environment variable MEMKIND_*
466 MEMKIND_ERROR_INVALID
468 Invalid input arguments to memkind routine
469 MEMKIND_ERROR_TOOMANY
471 Error trying to initialize more than maximum MEMKIND_MAX_KIND
472 number of kinds
473 MEMKIND_ERROR_BADOPS
475 Error memkind operation structure is missing or invalid
476 MEMKIND_ERROR_HUGETLB
478 Unable to allocate huge pages
479 MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE
481 Error requested memory type is not available
482 MEMKIND_ERROR_OPERATION_FAILED
484 Error memkind operation failed
485 MEMKIND_ERROR_ARENAS_CREATE
487 Call to jemalloc's arenas.create() failed
488 MEMKIND_ERROR_RUNTIME
490 Unspecified run-time error
491
493 /usr/bin/memkind-hbw-nodes
494 Prints a comma separated list of high bandwidth nodes.
495
497 MEMKIND_HBW_NODES
498 This environment variable is a comma separated list of NUMA
499 nodes that are treated as high bandwidth. Uses the libnuma rou‐
500 tine numa_parse_nodestring() for parsing, so the syntax
501 described in the numa(3) man page for this routine applies: e.g.
502 1-3,5 is a valid setting.
503 MEMKIND_ARENA_NUM_PER_KIND
505 This environment variable allows leveraging internal mechanism
506 of the library for setting number of arenas per kind. Value
507 should be a positive integer (not greater than INT_MAX defined
508 in <limits.h>). The user should set the value based on the
509 characteristics of application that is using the library. Higher
510 value can provide better performance in extremely multithreaded
511 applications at the cost of memory overhead. See section IMPLE‐
512 MENTATION NOTES of jemalloc(3) for more details about arenas.
513 MEMKIND_HOG_MEMORY
515 Controls behavior of memkind with regards to returning memory to
516 underlaying OS. Setting MEMKIND_HOG_MEMORY to 1 causes memkind
517 to not release memory to OS in anticipation of memory reuse
518 soon. This will improve latency of 'free' operations but
519 increase memory usage.
520 MEMKIND_DEBUG
522 Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to
523 1 enables printing messages like errors and general information
524 about environment to stderr.
525 MEMKIND_HEAP_MANAGER
527 Controls heap management behavior in memkind library by switch‐
528 ing to one of the available heap managers.
529 Values:
530 JEMALLOC - sets the jemalloc heap manager
531 TBB - sets the Intel Threading Building Blocks heap manager.
532 This option requires installed
533 Intel Threading Building Blocks library.
534 If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager
536 will be used by default.
537
539 Interfaces for obtaining 2MB (HUGETLB) need allocated huge pages in the
540 kernel's huge page pool.
541 HUGETLB (huge pages)
543 Current number of "persistent" huge pages can be read from
544 /proc/sys/vm/nr_hugepages file. Proposed way of setting
545 hugepages is: sudo sysctl vm.nr_hugepages=<number_of_hugepages>.
546 More information can be found here:
547 ⟨https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt⟩
548
550 When linking statically against memkind, libmemkind.a should be used
551 together with its dependencies libnuma and pthread. Pthread can be
552 linked by adding /usr/lib64/libpthread.a as a dependency (exact path
553 may vary). Typically libnuma will need to be compiled from sources to
554 use it as a static dependency. libnuma can be reached on GitHub:
555 ⟨https://github.com/numactl/numactl⟩
556
558 HUGETLB (huge pages)
559 There might be some overhead in huge pages consumption caused by
560 heap management. If your allocation fails because of OOM,
561 please try to allocate extra huge pages (e.g. 8 huge pages).
562
564 Copyright (C) 2014 - 2019 Intel Corporation. All rights reserved.
565
567 malloc(3), malloc_usable_size(3), numa(3), numactl(8), mbind(2),
568 mmap(2), move_pages(2), jemalloc(3), memkind_default(3),
569 memkind_arena(3), memkind_hbw(3), memkind_hugetlb(3), memkind_pmem(3)
570Intel Corporation 2015-03-31 MEMKIND(3)