1MEMKIND(3) MEMKIND MEMKIND(3)
2
6 memkind - Heap manager that enables allocations to memory with differ‐
7 ent properties.
8 This header expose STANDARD and EXPERIMENTAL API. API Standards are de‐
9 scribed below in this man page.
10
12 #include <memkind.h>
13 Link with -lmemkind
15 EXPERIMENTAL API:
17 HEAP MANAGEMENT:
18 int memkind_posix_memalign(memkind_t kind, void **memptr, size_t align‐
19 ment, size_t size);
20 KIND MANAGEMENT:
22 int memkind_create_kind(memkind_memtype_t memtype_flags, memkind_pol‐
23 icy_t policy, memkind_bits_t flags, memkind_t *kind);
24 STANDARD API:
27 ERROR HANDLING:
28 void memkind_error_message(int err, char *msg, size_t size);
29 LIBRARY VERSION:
31 int memkind_get_version();
32 HEAP MANAGEMENT:
34 void *memkind_malloc(memkind_t kind, size_t size);
35 void *memkind_calloc(memkind_t kind, size_t num, size_t size);
36 void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
37 void memkind_free(memkind_t kind, void *ptr);
38 size_t memkind_malloc_usable_size(memkind_t kind, void *ptr);
39 void *memkind_defrag_reallocate(memkind_t kind, void *ptr);
40 memkind_t memkind_detect_kind(void *ptr);
41 KIND CONFIGURATION MANAGEMENT:
43 struct memkind_config *memkind_config_new();
44 void memkind_config_delete(struct memkind_config *cfg);
45 void memkind_config_set_path(struct memkind_config *cfg, const char
46 *pmem_dir);
47 void memkind_config_set_size(struct memkind_config *cfg, size_t
48 pmem_size);
49 void memkind_config_set_memory_usage_policy(struct memkind_config *cfg,
50 memkind_mem_usage_policy policy);
51 KIND MANAGEMENT:
53 int memkind_create_fixed(void *addr, size_t size, memkind_t *kind);
54 int memkind_create_pmem(const char *dir, size_t max_size, memkind_t
55 *kind);
56 int memkind_create_pmem_with_config(struct memkind_config *cfg,
57 memkind_t *kind);
58 int memkind_destroy_kind(memkind_t kind);
59 int memkind_check_available(memkind_t kind);
60 ssize_t memkind_get_capacity(memkind_t kind);
61 int memkind_check_dax_path(const char *pmem_dir);
62 STATISTICS:
64 int memkind_update_cached_stats(void);
65 int memkind_get_stat(memkind_t kind, memkind_stat stat, size_t *value);
66 int memkind_stats_print(void (*write_cb) (void *, const char *), void
67 *cbopaque, memkind_stat_print_opt opts);
68 DECORATORS:
70 void memkind_malloc_pre(memkind_t *kind, size_t *size);
71 void memkind_malloc_post(memkind_t kind, size_t size, void **result);
72 void memkind_calloc_pre(memkind_t *kind, size_t *nmemb, size_t *size);
73 void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size,
74 void **result);
75 void memkind_posix_memalign_pre(memkind_t *kind, void **memptr, size_t
76 *alignment, size_t *size);
77 void memkind_posix_memalign_post(memkind_t kind, void **memptr, size_t
78 alignment, size_t size, int *err);
79 void memkind_realloc_pre(memkind_t *kind, void **ptr, size_t *size);
80 void memkind_realloc_post(memkind_t *kind, void *ptr, size_t size, void
81 **result);
82 void memkind_free_pre(memkind_t *kind, void **ptr);
83 void memkind_free_post(memkind_t kind, void *ptr);
84 ENHANCEMENTS:
86 int memkind_set_bg_threads(bool state);
87
91 memkind_error_message() converts an error number err returned by a mem‐
92 ber of the memkind interface to an error message msg where the maximum
93 size of the message is passed by the size parameter.
94 HEAP MANAGEMENT:
96 The functions described in this section define a heap manager with an
97 interface modeled on the ISO C standard API's, except that the user
98 must specify the kind of memory with the first argument to each func‐
99 tion. See the KINDS section below for a full description of the imple‐
100 mented kinds. For file-backed kind of memory see memkind_create_pmem()
101 or memkind_create_pmem_with_config(). For memory kind created on user-
102 specified area, please check memkind_create_fixed().
103 memkind_malloc() allocates size bytes of uninitialized memory of the
105 specified kind. The allocated space is suitably aligned (after possi‐
106 ble pointer coercion) for storage of any type of object. If size is 0,
107 then memkind_malloc() returns NULL.
108 memkind_calloc() allocates space for num objects each size bytes in
110 length in memory of the specified kind. The result is identical to
111 calling memkind_malloc() with an argument of num * size, with the ex‐
112 ception that the allocated memory is explicitly initialized to zero
113 bytes. If num or size is 0, then memkind_calloc() returns NULL.
114 memkind_realloc() changes the size of the previously allocated memory
116 referenced by ptr to size bytes of the specified kind. The contents of
117 the memory remain unchanged up to the lesser of the new and old sizes.
118 If the new size is larger, the contents of the newly allocated portion
119 of the memory are undefined. Upon success, the memory referenced by ptr
120 is freed and a pointer to the newly allocated memory is returned.
121 Note: memkind_realloc() may move the memory allocation, resulting in a
123 different return value than ptr.
124 If ptr is NULL, the memkind_realloc() function behaves identically to
126 memkind_malloc() for the specified size. If size is equal to zero, and
127 ptr is not NULL, then the call is equivalent to memkind_free(kind, ptr)
128 and NULL is returned. The address ptr, if not NULL, must have been re‐
129 turned by a previous call to memkind_malloc(), memkind_calloc(),
130 memkind_realloc(), memkind_defrag_reallocate() or memkind_posix_mema‐
131 lign() with the same kind as specified to the call to memkind_real‐
132 loc(). Otherwise, if memkind_free(kind, ptr) was called before, unde‐
133 fined behavior occurs. In cases where the kind is unknown in the con‐
134 text of the call to memkind_realloc() NULL can be given as the kind
135 specified to memkind_realloc(), but this will require an internal look
136 up for correct kind. Note: The look up for kind could result in seri‐
137 ous performance penalty, which can be avoided by specifying a correct
138 kind. If kind is NULL and ptr is NULL, then memkind_realloc() returns
139 NULL and sets errno to EINVAL.
140 memkind_posix_memalign() allocates size bytes of memory of a specified
142 kind such that the allocation's base address is an even multiple of
143 alignment, and returns the allocation in the value pointed to by
144 memptr. The requested alignment must be a power of 2 at least as large
145 as sizeof(void*). If size is 0, then memkind_posix_memalign() returns
146 0, with a NULL returned in memptr.
147 memkind_malloc_usable_size() function provides the same semantics as
149 malloc_usable_size(3), but operates on specified kind.
150 Note: In cases where the kind is unknown in the context of the call to
151 memkind_malloc_usable_size() NULL can be given as the kind specified to
152 memkind_malloc_usable_size(), but this could require a internal look up
153 for correct kind. memkind_malloc_usable_size() is supported by TBB
154 heap manager described in ENVIRONMENT section since Intel TBB 2019 Up‐
155 date 4.
156 memkind_defrag_reallocate() reallocates the object conditionally inside
158 specific kind. Function determines if it's worthwhile to move alloca‐
159 tion to reduce degree of external fragmentation of the heap. In case
160 of failure function returns NULL, otherwise function returns a pointer
161 to reallocated memory and memory referenced by ptr was released and
162 should not be accessed. If ptr is NULL, then memkind_defrag_reallo‐
163 cate() returns NULL. In cases where the kind is unknown in the context
164 of the call to memkind_defrag_reallocate() NULL can be given as the
165 kind specified to memkind_defrag_reallocate(), but this will require an
166 internal look up for correct kind. Note: The look up for kind could
167 result in serious performance penalty, which can be avoided by specify‐
168 ing a correct kind.
169 memkind_detect_kind() returns the kind associated with allocated memory
171 referenced by ptr. This pointer must have been returned by a previous
172 call to memkind_malloc(), memkind_calloc(), memkind_realloc(),
173 memkind_defrag_reallocate() or memkind_posix_memalign(). If ptr is
174 NULL, then memkind_detect_kind() returns NULL. Note: This function has
175 non-trivial performance overhead.
176 memkind_free() causes the allocated memory referenced by ptr to be made
178 available for future allocations. This pointer must have been returned
179 by a previous call to memkind_malloc(), memkind_calloc(), memkind_real‐
180 loc(), memkind_defrag_reallocate() or memkind_posix_memalign(). Other‐
181 wise, if memkind_free(kind, ptr) was already called before, undefined
182 behavior occurs. If ptr is NULL, no operation is performed. In cases
183 where the kind is unknown in the context of the call to memkind_free()
184 NULL can be given as the kind specified to memkind_free(), but this
185 will require an internal look up for correct kind. Note: The look up
186 for kind could result in serious performance penalty, which can be
187 avoided by specifying a correct kind.
188 KIND CONFIGURATION MANAGEMENT:
190 The functions described in this section define a way to create, delete
191 and update kind specific configuration. Except of memkind_con‐
192 fig_new(), user must specify the memkind configuration with the first
193 argument to each function. API described here is most useful with
194 file-backed kind of memory, e.g. memkind_create_pmem_with_config()
195 method.
196 memkind_config_new() creates the memkind configuration.
198 memkind_config_delete() deletes previously created memkind configura‐
200 tion, which must have been returned by a previous call to memkind_con‐
201 fig_new().
202 memkind_config_set_path() updates the memkind pmem_dir configuration
204 parameter, which specifies directory path, where file-backed kind of
205 memory will be created. Note: This function does not validate that
206 pmem_dir specifies a valid path.
207 memkind_config_set_size() updates the memkind pmem_size configuration
209 parameter, which allows to limit the file-backed kind memory partition.
210 Note: This function does not validate that pmem_size is in valid range.
211 memkind_config_set_memory_usage_policy() updates the memkind policy
213 configuration parameter, which allows to tune up memory utilization.
214 The user should set the value based on the characteristics of applica‐
215 tion that is using the library (e.g. prioritize memory usage, CPU uti‐
216 lization), for more details about policy see the MEMORY USAGE POLICY
217 section below. Note: This function does not validate that policy is in
218 valid range.
219 KIND MANAGEMENT:
221 There are built-in kinds that are always available and these are enu‐
222 merated in the KINDS section. The user can also create their own kinds
223 of memory. This section describes the API's that enable the tracking of
224 the different kinds of memory and determining their properties.
225 memkind_create_fixed() is a function used to create a kind under user-
227 specified area of memory. The memory can be allocated in any possible
228 way, e.g. it might be a static array or an mmapped area. User can spec‐
229 ify any properties using functions such as mbind. User is also respon‐
230 sible for de-allocation of memory after the kind destruction. The mem‐
231 ory area must remain valid until fixed_kind is destroyed. The area
232 starts at address addr and has size size. When heap manager runs out
233 of memory (located under user-specified area), a call to memkind_mal‐
234 loc() returns NULL and errno is set to ENOMEM.
235 memkind_create_pmem() is a convenient function used to create a file-
237 backed kind of memory. It allocates a temporary file in the given di‐
238 rectory dir. The file is created in a fashion similar to tmpfile(3),
239 so that the file name does not appear when the directory is listed and
240 the space is automatically freed when the program terminates. The file
241 is truncated to a size of max_size bytes and the resulting space is
242 memory-mapped.
243 Note that the actual file system space is not allocated immediately,
244 but only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This
245 allows to create a pmem memkind of a pretty large size without the need
246 to reserve in advance the corresponding file system space for the en‐
247 tire heap. If the value of max_size equals 0, pmem memkind is only lim‐
248 ited by the capacity of the file system mounted under dir argument.
249 The minimum max_size value which allows to limit the size of kind by
250 the library is defined as MEMKIND_PMEM_MIN_SIZE. Calling memkind_cre‐
251 ate_pmem() with a size smaller than that and different than 0 will re‐
252 turn an error. The maximum allowed size is not limited by memkind, but
253 by the file system specified by the dir argument. The max_size passed
254 in is the raw size of the memory pool and jemalloc will use some of
255 that space for its own metadata. Returns zero if the pmem memkind is
256 created successfully or an error code from the ERRORS section if not.
257 memkind_create_pmem_with_config() is a second function used to create a
259 file-backed kind of memory. Function behaves simillar to memkind_cre‐
260 ate_pmem() but instead of passing dir and max_size arguments, it uses
261 config param to specify characteristics of created file-backed kind of
262 memory (see KIND CONFIGURATION MANAGEMENT section).
263 memkind_create_kind() creates kind that allocates memory with specific
265 memory type, memory binding policy and flags (see MEMORY FLAGS sec‐
266 tion). The memtype_flags (see MEMORY TYPES section) determine memory
267 types to allocate, policy argument is policy for specifying page bind‐
268 ing to memory types selected by memtype_flags. Returns zero if the
269 specified kind is created successfully or an error code from the ERRORS
270 section if not.
271 memkind_destroy_kind() destroys previously created kind object, which
273 must have been returned by a previous call to memkind_create_pmem(),
274 memkind_create_pmem_with_config() or memkind_create_kind(). Otherwise,
275 or if memkind_destroy_kind(kind) was already called before, undefined
276 behavior occurs. Note that, when the kind was returned by memkind_cre‐
277 ate_kind() all allocated memory must be freed before kind is destroyed,
278 otherwise this will cause memory leak. When the kind was returned by
279 memkind_create_pmem() or memkind_create_pmem_with_config() all allo‐
280 cated memory will be freed after kind will be destroyed.
281 memkind_check_available() returns zero if the specified kind is avail‐
283 able or an error code from the ERRORS section if it is not.
284 memkind_get_capacity() returns memory capacity of nodes available to a
286 given kind (file size or filesystem capacity in case of a file-backed
287 PMEM kind; total area size in the case of fixed-kind) or -1 in case of
288 an error. Supported kinds are: MEMKIND_DEFAULT, MEMKIND_HIGHEST_CAPAC‐
289 ITY, MEMKIND_HIGHEST_CAPACITY_LOCAL, MEMKIND_LOWEST_LATENCY_LOCAL,
290 MEMKIND_HIGHEST_BANDWIDTH_LOCAL, MEMKIND_HUGETLB, MEMKIND_INTERLEAVE,
291 MEMKIND_HBW, MEMKIND_HBW_ALL, MEMKIND_HBW_INTERLEAVE, MEMKIND_DAX_KMEM,
292 MEMKIND_DAX_KMEM_ALL, MEMKIND_DAX_KMEM_INTERLEAVE, MEMKIND_REGULAR ,
293 file-backed PMEM and fixed-kind. kind. For MEMKIND_HUGETLB only pages
294 with a default size of 2MB are supported.
295 memkind_check_dax_path() returns zero if file-backed kind memory is in
297 the specified directory path pmem_dir can be created with the DAX at‐
298 tribute or an error code from the ERRORS section if it is not.
299 MEMKIND_PMEM_MIN_SIZE The minimum size which allows to limit the file-
301 backed memory partition.
302 STATISTICS:
304 The functions described in this section define a way to get specific
305 memory allocation statistics.
306 memkind_update_cached_stats() is used to force an update of cached dy‐
308 namic allocator statistics. Statistics are not updated real-time by
309 memkind library and this method allows to force its update.
310 memkind_get_stat() retrieves statistic of the specified type and re‐
312 turns it in value. For more details about stat see the MEMORY STATIS‐
313 TICS TYPE section below. Measured statistic applies to specific kind,
314 when NULL is given as kind then statistic applies to memory used by the
315 whole memkind library. Note: You need to call memkind_up‐
316 date_cached_stats() before calling memkind_get_stat() because statis‐
317 tics are cached by memkind library.
318 memkind_stats_print() prints summary statistics. This function wraps
320 jemalloc's function je_malloc_stats_print(). Uses write_cb function to
321 print the output. While providing custom writer function, use
322 syscall(2) rather than write(2). Pass NULL in order to use the default
323 write_cb function which prints the output to the stderr. Use cbopaque
324 parameter in order to pass some data to your write_cb function. Pass
325 additional options using opts. For more details on opts see the MEMORY
326 STATISTICS PRINT OPTIONS section below. Returns MEMKIND_ERROR_INVALID
327 when failed to parse options string, MEMKIND_SUCCESS on success.
328 DECORATORS:
330 The memkind library enables the user to define decorator functions that
331 can be called before and after each memkind heap management API. The
332 decorators that are called at the beginning of the function end are
333 named after that function with _pre appended to the name and those that
334 are called at the end of the function are named after that function
335 with _post appended to the name. These are weak symbols and if they are
336 not present at link time they are not called. The memkind library does
337 not define these symbols which are reserved for user definition. These
338 decorators can be used to track calls to the heap management interface
339 or to modify parameters. The decorators that are called at the begin‐
340 ning of the allocator pass all inputs by reference and the decorators
341 that are called at the end of the allocator pass the output by refer‐
342 ence. This enables the modification of the input and output of each
343 heap management function by the decorators.
344 ENHANCEMENTS:
346 memkind_set_bg_threads() enables/disables internal background worker
347 threads in jemalloc.
348 LIBRARY VERSION:
350 The memkind library version scheme consist major, minor and patch num‐
351 bers separated by dot. Combining those numbers, we got the following
352 representation:
353 major.minor.patch, where:
354 -major number is incremented whenever API is changed (loss of
355 backward compatibility),
356 -minor number is incremented whenever additional extensions are
357 introduced or behavior has been changed,
358 -patch number is incremented whenever small bug fixes are added.
359 memkind library provide numeric representation of the version by expos‐
361 ing the following API:
362 memkind_get_version() returns version number represented by a single
364 integer number, obtained from the formula:
365 major * 1000000 + minor * 1000 + patch
366 Note: major < 1 means unstable API.
368 API standards:
370 -STANDARD API, API is considered as stable
371 -NON-STANDARD API, API is considered as stable, however this is not a
372 standard way to use memkind
373 -EXPERIMENTAL API, API is considered as unstable and the subject to
374 change
375
378 memkind_calloc(), memkind_malloc(), memkind_realloc() and memkind_de‐
379 frag_reallocate() returns the pointer to the allocated memory or NULL
380 if the request fails. memkind_malloc_usable_size() returns the number
381 of usable bytes in the block of allocated memory pointed to by ptr, a
382 pointer to a block of memory allocated by memkind_malloc() or a related
383 function. If ptr is NULL, 0 is returned. memkind_free() and
384 memkind_error_message() do not have return values. All other memkind
385 API's return 0 upon success and an error code defined in the ERRORS
386 section upon failure. The memkind library avoids setting errno di‐
387 rectly, but calls to underlying libraries and system calls may set er‐
388 rno (e.g. memkind_create_pmem()).
389
391 The available kinds of memory:
392 MEMKIND_DEFAULT
394 Default allocation using standard memory and default page size.
395 MEMKIND_HIGHEST_CAPACITY
397 Allocate from a NUMA node(s) that has the highest capacity among
398 all nodes in the system.
399 MEMKIND_HIGHEST_CAPACITY_PREFERRED
401 Same as MEMKIND_HIGHEST_CAPACITY except that if there is not
402 enough memory in the NUMA node that has the highest capacity in
403 the local domain to satisfy the request, the allocation will
404 fall back on other memory NUMA nodes. Note: For this kind, the
405 allocation will not succeed if there are two or more NUMA nodes
406 that have the highest capacity.
407 MEMKIND_HIGHEST_CAPACITY_LOCAL
409 Allocate from a NUMA node that has the highest capacity among
410 all NUMA Nodes from the local domain. NUMA Nodes have the same
411 local domain for a set of CPUs associated with them, e.g. socket
412 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
413 the same local domain that have the highest capacity - alloca‐
414 tion will be done from NUMA node with worse latency attribute.
415 This kind requires locality information described in SYSTEM CON‐
416 FIGURATION section.
417 MEMKIND_HIGHEST_CAPACITY_LOCAL_PREFERRED
419 Same as MEMKIND_HIGHEST_CAPACITY_LOCAL except that if there is
420 not enough memory in the NUMA node that has the highest capacity
421 to satisfy the request, the allocation will fall back on other
422 memory NUMA nodes.
423 MEMKIND_LOWEST_LATENCY_LOCAL
425 Allocate from a NUMA node that has the lowest latency among all
426 NUMA Nodes from the local domain. NUMA Nodes have the same lo‐
427 cal domain for a set of CPUs associated with them, e.g. socket
428 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
429 the same local domain that have the lowest latency - allocation
430 will be done from NUMA node with smaller memory capacity. This
431 kind requires locality and memory performance characteristics
432 information described in SYSTEM CONFIGURATION section.
433 MEMKIND_LOWEST_LATENCY_LOCAL_PREFERRED
435 Same as MEMKIND_LOWEST_LATENCY_LOCAL except that if there is not
436 enough memory in the NUMA node that has the lowest latency to
437 satisfy the request, the allocation will fall back on other mem‐
438 ory NUMA nodes.
439 MEMKIND_HIGHEST_BANDWIDTH_LOCAL
441 Allocate from a NUMA node that has the highest bandwidth among
442 all NUMA Nodes from the local domain. NUMA Nodes have the same
443 local domain for a set of CPUs associated with them, e.g. socket
444 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
445 the same local domain that have the highest bandwidth - alloca‐
446 tion will be done from NUMA node with smaller memory capacity.
447 This kind requires locality and memory performance characteris‐
448 tics information described in SYSTEM CONFIGURATION section.
449 MEMKIND_HIGHEST_BANDWIDTH_LOCAL_PREFERRED
451 Same as MEMKIND_HIGHEST_BANDWIDTH_LOCAL except that if there is
452 not enough memory in the NUMA node that has the highest band‐
453 width to satisfy the request, the allocation will fall back on
454 other memory NUMA nodes.
455 MEMKIND_HUGETLB
457 Allocate from standard memory using huge pages. Note: This kind
458 requires huge pages configuration described in SYSTEM CONFIGURA‐
459 TION section.
460 MEMKIND_GBTLB (DEPRECATED)
462 Allocate from standard memory using 1GB chunks backed by huge
463 pages. Note: This kind requires huge pages configuration de‐
464 scribed in SYSTEM CONFIGURATION section.
465 MEMKIND_INTERLEAVE
467 Allocate pages interleaved across all NUMA nodes with transpar‐
468 ent huge pages disabled.
469 MEMKIND_HBW
471 Allocate from the closest high bandwidth memory NUMA node(s) at
472 the time of allocation. If there is not enough high bandwidth
473 memory to satisfy the request errno is set to ENOMEM and the al‐
474 located pointer is set to NULL. Note: This kind requires memory
475 performance characteristics information described in SYSTEM CON‐
476 FIGURATION section.
477 MEMKIND_HBW_ALL
479 Same as MEMKIND_HBW except decision regarding closest NUMA
480 node(s) is postponed until the time of first write.
481 MEMKIND_HBW_HUGETLB
483 Same as MEMKIND_HBW except the allocation is backed by huge
484 pages. Note: This kind requires huge pages configuration de‐
485 scribed in SYSTEM CONFIGURATION section.
486 MEMKIND_HBW_ALL_HUGETLB
488 Combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB proper‐
489 ties. Note: This kind requires huge pages configuration de‐
490 scribed in SYSTEM CONFIGURATION section.
491 MEMKIND_HBW_PREFERRED
493 Same as MEMKIND_HBW except that if there is not enough high
494 bandwidth memory to satisfy the request, the allocation will
495 fall back on standard memory. Note: For this kind, the alloca‐
496 tion will not succeed if two or more high bandwidth memory NUMA
497 nodes are in the same shortest distance to the same CPU on which
498 process is eligible to run. Check on that eligibility is done
499 upon starting the application.
500 MEMKIND_HBW_PREFERRED_HUGETLB
502 Same as MEMKIND_HBW_PREFERRED except the allocation is backed by
503 huge pages. Note: This kind requires huge pages configuration
504 described in SYSTEM CONFIGURATION section.
505 MEMKIND_HBW_GBTLB (DEPRECATED)
507 Same as MEMKIND_HBW except the allocation is backed by 1GB
508 chunks of huge pages. Note that size can take on any value, but
509 full gigabyte pages will allocated for each request, so remain‐
510 der of the last page will be wasted. This kind requires huge
511 pages configuration described in SYSTEM CONFIGURATION section.
512 MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED)
514 Same as MEMKIND_HBW_GBTLB except that if there is not enough
515 high bandwidth memory to satisfy the request, the allocation
516 will fall back on standard memory. Note: This kind requires
517 huge pages configuration described in SYSTEM CONFIGURATION sec‐
518 tion. For this kind, the allocation will not succeed if two or
519 more high bandwidth memory NUMA nodes are in the same shortest
520 distance to the same CPU on which process is eligible to run.
521 Check on that eligibility is done upon starting the application.
522 MEMKIND_HBW_INTERLEAVE
524 Same as MEMKIND_HBW except that the pages that support the allo‐
525 cation are interleaved across all high bandwidth nodes and
526 transparent huge pages are disabled.
527 MEMKIND_DAX_KMEM
529 Allocate from the closest persistent memory NUMA node at the
530 time of allocation. If there is not enough memory in the closest
531 persistent memory NUMA node to satisfy the request errno is set
532 to ENOMEM and the allocated pointer is set to NULL.
533 MEMKIND_DAX_KMEM_ALL
535 Allocate from the closest persistent memory NUMA node available
536 at the time of allocation. If there is not enough memory on any
537 of persistent memory NUMA nodes to satisfy the request errno is
538 set to ENOMEM and the allocated pointer is set to NULL.
539 MEMKIND_DAX_KMEM_PREFERRED
541 Same as MEMKIND_DAX_KMEM except that if there is not enough mem‐
542 ory in the closest persistent memory NUMA node to satisfy the
543 request, the allocation will fall back on other memory NUMA
544 nodes. Note: For this kind, the allocation will not succeed if
545 two or more persistent memory NUMA nodes are in the same short‐
546 est distance to the same CPU on which process is eligible to
547 run. Check on that eligibility is done upon starting the appli‐
548 cation.
549 MEMKIND_DAX_KMEM_INTERLEAVE
551 Same as MEMKIND_DAX_KMEM except that the pages that support the
552 allocation are interleaved across all persistent memory NUMA
553 nodes.
554 MEMKIND_REGULAR
556 Allocate from regular memory using the default page size. Regu‐
557 lar means general purpose memory from the NUMA nodes containing
558 CPUs.
559
561 The available types of memory:
562 MEMKIND_MEMTYPE_DEFAULT
564 Standard memory, the same as process uses.
565 MEMKIND_MEMTYPE_HIGH_BANDWIDTH
567 High bandwidth memory (HBM). There must be at least two memory
568 types with different bandwidth to determine which is the HBM.
569
571 The available types of memory binding policy:
572 MEMKIND_POLICY_BIND_LOCAL
574 Allocate local memory. If there is not enough memory to satisfy
575 the request errno is set to ENOMEM and the allocated pointer is
576 set to NULL.
577 MEMKIND_POLICY_BIND_ALL
579 Memory locality is ignored. If there is not enough memory to
580 satisfy the request errno is set to ENOMEM and the allocated
581 pointer is set to NULL.
582 MEMKIND_POLICY_PREFERRED_LOCAL
584 Allocate preferred memory that is local. If there is not enough
585 preferred memory to satisfy the request or preferred memory is
586 not available, the allocation will fall back on any other mem‐
587 ory.
588 MEMKIND_POLICY_INTERLEAVE_LOCAL
590 Interleave allocation across local memory. For n memory types
591 the allocation will be interleaved across all of them.
592 MEMKIND_POLICY_INTERLEAVE_ALL
594 Interleave allocation. Locality is ignored. For n memory types
595 the allocation will be interleaved across all of them.
596 MEMKIND_POLICY_MAX_VALUE
598 Max policy value.
599
601 The available types of memory flags:
602 MEMKIND_MASK_PAGE_SIZE_2MB
604 Allocation backed by 2MB page size.
605
607 The available types of memory usage policy:
608 MEMKIND_MEM_USAGE_POLICY_DEFAULT
610 Default memory usage policy.
611 MEMKIND_MEM_USAGE_POLICY_CONSERVATIVE
613 Conservative memory usage policy - prioritize memory usage at
614 cost of performance. Note: Memory usage policies have no effect
615 for TBB heap manager described in ENVIRONMENT section.
616
618 The available types of memory statistics:
619 MEMKIND_STAT_TYPE_RESIDENT
621 Maximum number of bytes in physically resident data pages
622 mapped.
623 MEMKIND_STAT_TYPE_ACTIVE
625 Total number of bytes in active pages.
626 MEMKIND_STAT_TYPE_ALLOCATED
628 Total number of allocated bytes.
629
631 The available options for printing statistics:
632 MEMKIND_STAT_PRINT_ALL
634 Print all statistics.
635 MEMKIND_STAT_PRINT_JSON_FORMAT
637 Print statistics in JSON format.
638 MEMKIND_STAT_PRINT_OMIT_GENERAL
640 Omit general information that never changes during execution.
641 MEMKIND_STAT_PRINT_OMIT_MERGED_ARENA
643 Omit merged arena statistics.
644 MEMKIND_STAT_PRINT_OMIT_DESTROYED_MERGED_ARENA
646 Omit destroyed merged arena statistics.
647 MEMKIND_STAT_PRINT_OMIT_PER_ARENA
649 Omit per arena statistics.
650 MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_BINS
652 Omit per size class statistics for bins.
653 MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_LARGE
655 Omit per size class statistics for large objects.
656 MEMKIND_STAT_PRINT_OMIT_MUTEX
658 Omit all mutex statistics.
659 MEMKIND_STAT_PRINT_OMIT_EXTENT
661 Omit extent statistics.
662
664 memkind_posix_memalign()
665 returns the one of the POSIX standard error codes EINVAL or
666 ENOMEM as defined in <errno.h> if an error occurs (these have
667 positive values). If the alignment parameter is not a power of
668 two or is not a multiple of sizeof(void*), then EINVAL is re‐
669 turned. If there is insufficient memory to satisfy the request
670 then ENOMEM is returned.
671 All functions other than memkind_posix_memalign() which have an integer
673 return type return one of the negative error codes as defined in
674 <memkind.h> and described below.
675 MEMKIND_ERROR_UNAVAILABLE
677 Requested memory kind is not available
678 MEMKIND_ERROR_MBIND
680 Call to mbind(2) failed
681 MEMKIND_ERROR_MMAP
683 Call to mmap(2) failed
684 MEMKIND_ERROR_MALLOC
686 Call to jemalloc's malloc() failed
687 MEMKIND_ERROR_ENVIRON
689 Error parsing environment variable MEMKIND_*
690 MEMKIND_ERROR_INVALID
692 Invalid input arguments to memkind routine
693 MEMKIND_ERROR_TOOMANY
695 Error trying to initialize more than maximum MEMKIND_MAX_KIND
696 number of kinds
697 MEMKIND_ERROR_BADOPS
699 Error memkind operation structure is missing or invalid
700 MEMKIND_ERROR_HUGETLB
702 Unable to allocate huge pages
703 MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE
705 Error requested memory type is not available
706 MEMKIND_ERROR_OPERATION_FAILED
708 Error memkind operation failed
709 MEMKIND_ERROR_ARENAS_CREATE
711 Call to jemalloc's arenas.create() failed
712 MEMKIND_ERROR_RUNTIME
714 Unspecified run-time error
715
717 /usr/bin/memkind-hbw-nodes
718 Prints a comma-separated list of high bandwidth nodes.
719 /usr/bin/memkind-auto-dax-kmem-nodes
721 Prints a comma-separated list of persistent memory NUMA nodes,
722 which are automatically detected.
723
725 MEMKIND_HBW_NODES
726 This environment variable is a comma-separated list of NUMA
727 nodes that are treated as high bandwidth. Uses the libnuma rou‐
728 tine numa_parse_nodestring() for parsing, so the syntax de‐
729 scribed in the numa(3) man page for this routine applies: e.g.
730 1-3,5 is a valid setting.
731 MEMKIND_HBW_THRESHOLD
733 This environment variable is bandwidth in MB/s that is the
734 threshold for identifying high bandwidth memory. The default
735 threshold is 204800 (200 GB/s), which is used if this variable
736 is not set. When set, it must be greater than or equal to 0.
737 MEMKIND_DAX_KMEM_NODES
739 This environment variable is a comma-separated list of NUMA
740 nodes that are treated as PMEM memory. Uses the libnuma routine
741 numa_parse_nodestring() for parsing, so the syntax described in
742 the numa(3) man page for this routine applies: e.g. 1-3,5 is a
743 valid setting.
744 MEMKIND_ARENA_NUM_PER_KIND
746 This environment variable allows leveraging internal mechanism
747 of the library for setting number of arenas per kind. Value
748 should be a positive integer (not greater than INT_MAX defined
749 in <limits.h>). The user should set the value based on the
750 characteristics of application that is using the library. Higher
751 value can provide better performance in extremely multithreaded
752 applications at the cost of memory overhead. See section IMPLE‐
753 MENTATION NOTES of jemalloc(3) for more details about arenas.
754 MEMKIND_HOG_MEMORY
756 Controls behavior of memkind with regards to returning memory to
757 underlying OS. Setting MEMKIND_HOG_MEMORY to 1 causes memkind to
758 not release memory to OS in anticipation of memory reuse soon.
759 This will improve latency of 'free' operations but increase mem‐
760 ory usage. Note: For file-backed kind memory will be released
761 to OS only after calling memkind_destroy_kind(), not after
762 'free' operations. In context of MEMKIND_MEM_USAGE_POLICY_CON‐
763 SERVATIVE memory usage policy - it will also impact memory coa‐
764 lescing and results that blocks pages will be often reused (bet‐
765 ter memory usage at cost of performance).
766 MEMKIND_DEBUG
768 Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to
769 1 enables printing messages like errors and general information
770 about environment to stderr.
771 MEMKIND_BACKGROUND_THREAD_LIMIT
773 Enable background worker threads. Value should be from range 0
774 to maximum number of cpus. Setting MEMKIND_BACK‐
775 GROUND_THREAD_LIMIT to specific value will limit maximum number
776 of background worker threads to this value. 0 means maximum
777 number of background worker threads will be limited to maximum
778 number of cpus.
779 MEMKIND_HEAP_MANAGER
781 Controls heap management behavior in memkind library by switch‐
782 ing to one of the available heap managers.
783 Values:
784 JEMALLOC - sets the jemalloc heap manager
785 TBB - sets the Intel Threading Building Blocks heap manager.
786 This option requires installed
787 Intel Threading Building Blocks library.
788 If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager
790 will be used by default.
791
793 Interfaces for obtaining 2MB (HUGETLB) memory need allocated huge pages
794 in the kernel's huge page pool.
795 HUGETLB (huge pages)
797 Current number of "persistent" huge pages can be read from
798 /proc/sys/vm/nr_hugepages file. Proposed way of setting
799 hugepages is: sudo sysctl vm.nr_hugepages=<number_of_hugepages>.
800 More information can be found here:
801 ⟨https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt⟩
802 Interfaces for obtaining locality information are provided by libhwloc
804 dependency. Functionality based on locality requires that memkind li‐
805 brary is configured and built with the support of libhwloc (./configure
806 --enable-hwloc).
807 Interfaces for obtaining memory performance characteristics information
809 are based on HMAT (Heterogeneous Memory Attribute Table)
810 ⟨https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf⟩
811 Functionality based on memory performance characteristics requires that
812 platform configuration fully supports HMAT and memkind library is con‐
813 figured and built with the support of libhwloc (./configure --enable-
814 hwloc).
815 Note: For a given target NUMA Node, the OS exposes only the performance
817 characteristics of the best performing NUMA node.
818 libhwloc can be reached on: ⟨https://www.open-mpi.org/projects/hwloc⟩
820
822 When linking statically against memkind, libmemkind.a should be used
823 together with its dependencies libnuma and pthread. Pthread can be
824 linked by adding /usr/lib64/libpthread.a as a dependency (exact path
825 may vary). Typically libnuma will need to be compiled from sources to
826 use it as a static dependency. libnuma can be reached on GitHub:
827 ⟨https://github.com/numactl/numactl⟩
828
830 HUGETLB (huge pages)
831 There might be some overhead in huge pages consumption caused by
832 heap management. If your allocation fails because of OOM,
833 please try to allocate extra huge pages (e.g. 8 huge pages).
834
836 Copyright (C) 2014 - 2021 Intel Corporation. All rights reserved.
837
839 malloc(3), malloc_usable_size(3), numa(3), hwloc(3), numactl(8),
840 mbind(2), mmap(2), move_pages(2), jemalloc(3), memkind_dax_kmem(3),
841 memkind_default(3), memkind_arena(3), memkind_fixed(3), memkind_hbw(3),
842 memkind_hugetlb(3), memkind_pmem(3), syscall(2), write(2)
843Intel Corporation 2015-03-31 MEMKIND(3)