1MEMKIND(3) MEMKIND MEMKIND(3)
2
6 memkind - Heap manager that enables allocations to memory with differ‐
7 ent properties.
8 This header expose STANDARD and EXPERIMENTAL API. API Standards are de‐
9 scribed below in this man page.
10
12 #include <memkind.h>
13 Link with -lmemkind
15 EXPERIMENTAL API:
17 HEAP MANAGEMENT:
18 int memkind_posix_memalign(memkind_t kind, void **memptr, size_t align‐
19 ment, size_t size);
20 KIND MANAGEMENT:
22 int memkind_create_kind(memkind_memtype_t memtype_flags, memkind_pol‐
23 icy_t policy, memkind_bits_t flags, memkind_t *kind);
24 STANDARD API:
27 ERROR HANDLING:
28 void memkind_error_message(int err, char *msg, size_t size);
29 LIBRARY VERSION:
31 int memkind_get_version();
32 HEAP MANAGEMENT:
34 void *memkind_malloc(memkind_t kind, size_t size);
35 void *memkind_calloc(memkind_t kind, size_t num, size_t size);
36 void *memkind_realloc(memkind_t kind, void *ptr, size_t size);
37 void memkind_free(memkind_t kind, void *ptr);
38 size_t memkind_malloc_usable_size(memkind_t kind, void *ptr);
39 void *memkind_defrag_reallocate(memkind_t kind, void *ptr);
40 memkind_t memkind_detect_kind(void *ptr);
41 KIND CONFIGURATION MANAGEMENT:
43 struct memkind_config *memkind_config_new();
44 void memkind_config_delete(struct memkind_config *cfg);
45 void memkind_config_set_path(struct memkind_config *cfg, const char
46 *pmem_dir);
47 void memkind_config_set_size(struct memkind_config *cfg, size_t
48 pmem_size);
49 void memkind_config_set_memory_usage_policy(struct memkind_config *cfg,
50 memkind_mem_usage_policy policy);
51 KIND MANAGEMENT:
53 int memkind_create_pmem(const char *dir, size_t max_size, memkind_t
54 *kind);
55 int memkind_create_pmem_with_config(struct memkind_config *cfg,
56 memkind_t *kind);
57 int memkind_destroy_kind(memkind_t kind);
58 int memkind_check_available(memkind_t kind);
59 int memkind_check_dax_path(const char *pmem_dir);
60 STATISTICS:
62 int memkind_update_cached_stats(void);
63 int memkind_get_stat(memkind_t kind, memkind_stat stat, size_t *value);
64 int memkind_stats_print(void (*write_cb) (void *, const char *), void
65 *cbopaque, memkind_stat_print_opt opts);
66 DECORATORS:
68 void memkind_malloc_pre(memkind_t *kind, size_t *size);
69 void memkind_malloc_post(memkind_t kind, size_t size, void **result);
70 void memkind_calloc_pre(memkind_t *kind, size_t *nmemb, size_t *size);
71 void memkind_calloc_post(memkind_t kind, size_t nmemb, size_t size,
72 void **result);
73 void memkind_posix_memalign_pre(memkind_t *kind, void **memptr, size_t
74 *alignment, size_t *size);
75 void memkind_posix_memalign_post(memkind_t kind, void **memptr, size_t
76 alignment, size_t size, int *err);
77 void memkind_realloc_pre(memkind_t *kind, void **ptr, size_t *size);
78 void memkind_realloc_post(memkind_t *kind, void *ptr, size_t size, void
79 **result);
80 void memkind_free_pre(memkind_t *kind, void **ptr);
81 void memkind_free_post(memkind_t kind, void *ptr);
82 ENHANCEMENTS:
84 int memkind_set_bg_threads(bool state);
85
89 memkind_error_message() converts an error number err returned by a mem‐
90 ber of the memkind interface to an error message msg where the maximum
91 size of the message is passed by the size parameter.
92 HEAP MANAGEMENT:
94 The functions described in this section define a heap manager with an
95 interface modeled on the ISO C standard API's, except that the user
96 must specify the kind of memory with the first argument to each func‐
97 tion. See the KINDS section below for a full description of the imple‐
98 mented kinds. For file-backed kind of memory see memkind_create_pmem()
99 or memkind_create_pmem_with_config().
100 memkind_malloc() allocates size bytes of uninitialized memory of the
102 specified kind. The allocated space is suitably aligned (after possi‐
103 ble pointer coercion) for storage of any type of object. If size is 0,
104 then memkind_malloc() returns NULL.
105 memkind_calloc() allocates space for num objects each size bytes in
107 length in memory of the specified kind. The result is identical to
108 calling memkind_malloc() with an argument of num * size, with the ex‐
109 ception that the allocated memory is explicitly initialized to zero
110 bytes. If num or size is 0, then memkind_calloc() returns NULL.
111 memkind_realloc() changes the size of the previously allocated memory
113 referenced by ptr to size bytes of the specified kind. The contents of
114 the memory remain unchanged up to the lesser of the new and old sizes.
115 If the new size is larger, the contents of the newly allocated portion
116 of the memory are undefined. Upon success, the memory referenced by ptr
117 is freed and a pointer to the newly allocated memory is returned.
118 Note: memkind_realloc() may move the memory allocation, resulting in a
120 different return value than ptr.
121 If ptr is NULL, the memkind_realloc() function behaves identically to
123 memkind_malloc() for the specified size. If size is equal to zero, and
124 ptr is not NULL, then the call is equivalent to memkind_free(kind, ptr)
125 and NULL is returned. The address ptr, if not NULL, must have been re‐
126 turned by a previous call to memkind_malloc(), memkind_calloc(),
127 memkind_realloc(), memkind_defrag_reallocate() or memkind_posix_mema‐
128 lign() with the same kind as specified to the call to memkind_real‐
129 loc(). Otherwise, if memkind_free(kind, ptr) was called before, unde‐
130 fined behavior occurs. In cases where the kind is unknown in the con‐
131 text of the call to memkind_realloc() NULL can be given as the kind
132 specified to memkind_realloc(), but this will require an internal look
133 up for correct kind. Note: The look up for kind could result in seri‐
134 ous performance penalty, which can be avoided by specifying a correct
135 kind. If kind is NULL and ptr is NULL, then memkind_realloc() returns
136 NULL and sets errno to EINVAL.
137 memkind_posix_memalign() allocates size bytes of memory of a specified
139 kind such that the allocation's base address is an even multiple of
140 alignment, and returns the allocation in the value pointed to by
141 memptr. The requested alignment must be a power of 2 at least as large
142 as sizeof(void*). If size is 0, then memkind_posix_memalign() returns
143 0, with a NULL returned in memptr.
144 memkind_malloc_usable_size() function provides the same semantics as
146 malloc_usable_size(3), but operates on specified kind.
147 Note: In cases where the kind is unknown in the context of the call to
148 memkind_malloc_usable_size() NULL can be given as the kind specified to
149 memkind_malloc_usable_size(), but this could require a internal look up
150 for correct kind. memkind_malloc_usable_size() is supported by TBB
151 heap manager described in ENVIRONMENT section since Intel TBB 2019 Up‐
152 date 4.
153 memkind_defrag_reallocate() reallocates the object conditionally inside
155 specific kind. Function determines if it's worthwhile to move alloca‐
156 tion to reduce degree of external fragmentation of the heap. In case
157 of failure function returns NULL, otherwise function returns a pointer
158 to reallocated memory and memory referenced by ptr was released and
159 should not be accessed. If ptr is NULL, then memkind_defrag_reallo‐
160 cate() returns NULL. In cases where the kind is unknown in the context
161 of the call to memkind_defrag_reallocate() NULL can be given as the
162 kind specified to memkind_defrag_reallocate(), but this will require an
163 internal look up for correct kind. Note: The look up for kind could
164 result in serious performance penalty, which can be avoided by specify‐
165 ing a correct kind.
166 memkind_detect_kind() returns the kind associated with allocated memory
168 referenced by ptr. This pointer must have been returned by a previous
169 call to memkind_malloc(), memkind_calloc(), memkind_realloc(),
170 memkind_defrag_reallocate() or memkind_posix_memalign(). If ptr is
171 NULL, then memkind_detect_kind() returns NULL. Note: This function has
172 non-trivial performance overhead.
173 memkind_free() causes the allocated memory referenced by ptr to be made
175 available for future allocations. This pointer must have been returned
176 by a previous call to memkind_malloc(), memkind_calloc(), memkind_real‐
177 loc(), memkind_defrag_reallocate() or memkind_posix_memalign(). Other‐
178 wise, if memkind_free(kind, ptr) was already called before, undefined
179 behavior occurs. If ptr is NULL, no operation is performed. In cases
180 where the kind is unknown in the context of the call to memkind_free()
181 NULL can be given as the kind specified to memkind_free(), but this
182 will require an internal look up for correct kind. Note: The look up
183 for kind could result in serious performance penalty, which can be
184 avoided by specifying a correct kind.
185 KIND CONFIGURATION MANAGEMENT:
187 The functions described in this section define a way to create, delete
188 and update kind specific configuration. Except of memkind_con‐
189 fig_new(), user must specify the memkind configuration with the first
190 argument to each function. API described here is most useful with
191 file-backed kind of memory, e.g. memkind_create_pmem_with_config()
192 method.
193 memkind_config_new() creates the memkind configuration.
195 memkind_config_delete() deletes previously created memkind configura‐
197 tion, which must have been returned by a previous call to memkind_con‐
198 fig_new().
199 memkind_config_set_path() updates the memkind pmem_dir configuration
201 parameter, which specifies directory path, where file-backed kind of
202 memory will be created. Note: This function does not validate that
203 pmem_dir specifies a valid path.
204 memkind_config_set_size() updates the memkind pmem_size configuration
206 parameter, which allows to limit the file-backed kind memory partition.
207 Note: This function does not validate that pmem_size is in valid range.
208 memkind_config_set_memory_usage_policy() updates the memkind policy
210 configuration parameter, which allows to tune up memory utilization.
211 The user should set the value based on the characteristics of applica‐
212 tion that is using the library (e.g. prioritize memory usage, CPU uti‐
213 lization), for more details about policy see the MEMORY USAGE POLICY
214 section below. Note: This function does not validate that policy is in
215 valid range.
216 KIND MANAGEMENT:
218 There are built-in kinds that are always available and these are enu‐
219 merated in the KINDS section. The user can also create their own kinds
220 of memory. This section describes the API's that enable the tracking of
221 the different kinds of memory and determining their properties.
222 memkind_create_pmem() is a convenient function used to create a file-
224 backed kind of memory. It allocates a temporary file in the given di‐
225 rectory dir. The file is created in a fashion similar to tmpfile(3),
226 so that the file name does not appear when the directory is listed and
227 the space is automatically freed when the program terminates. The file
228 is truncated to a size of max_size bytes and the resulting space is
229 memory-mapped.
230 Note that the actual file system space is not allocated immediately,
231 but only on a call to memkind_pmem_mmap() (see memkind_pmem(3)). This
232 allows to create a pmem memkind of a pretty large size without the need
233 to reserve in advance the corresponding file system space for the en‐
234 tire heap. If the value of max_size equals 0, pmem memkind is only lim‐
235 ited by the capacity of the file system mounted under dir argument.
236 The minimum max_size value which allows to limit the size of kind by
237 the library is defined as MEMKIND_PMEM_MIN_SIZE. Calling memkind_cre‐
238 ate_pmem() with a size smaller than that and different than 0 will re‐
239 turn an error. The maximum allowed size is not limited by memkind, but
240 by the file system specified by the dir argument. The max_size passed
241 in is the raw size of the memory pool and jemalloc will use some of
242 that space for its own metadata. Returns zero if the pmem memkind is
243 created successfully or an error code from the ERRORS section if not.
244 memkind_create_pmem_with_config() is a second function used to create a
246 file-backed kind of memory. Function behaves simillar to memkind_cre‐
247 ate_pmem() but instead of passing dir and max_size arguments, it uses
248 config param to specify characteristics of created file-backed kind of
249 memory (see KIND CONFIGURATION MANAGEMENT section).
250 memkind_create_kind() creates kind that allocates memory with specific
252 memory type, memory binding policy and flags (see MEMORY FLAGS sec‐
253 tion). The memtype_flags (see MEMORY TYPES section) determine memory
254 types to allocate, policy argument is policy for specifying page bind‐
255 ing to memory types selected by memtype_flags. Returns zero if the
256 specified kind is created successfully or an error code from the ERRORS
257 section if not.
258 memkind_destroy_kind() destroys previously created kind object, which
260 must have been returned by a previous call to memkind_create_pmem(),
261 memkind_create_pmem_with_config() or memkind_create_kind(). Otherwise,
262 or if memkind_destroy_kind(kind) was already called before, undefined
263 behavior occurs. Note that, when the kind was returned by memkind_cre‐
264 ate_kind() all allocated memory must be freed before kind is destroyed,
265 otherwise this will cause memory leak. When the kind was returned by
266 memkind_create_pmem() or memkind_create_pmem_with_config() all allo‐
267 cated memory will be freed after kind will be destroyed.
268 memkind_check_available() returns zero if the specified kind is avail‐
270 able or an error code from the ERRORS section if it is not.
271 memkind_check_dax_path() returns zero if file-backed kind memory is in
273 the specified directory path pmem_dir can be created with the DAX at‐
274 tribute or an error code from the ERRORS section if it is not.
275 MEMKIND_PMEM_MIN_SIZE The minimum size which allows to limit the file-
277 backed memory partition.
278 STATISTICS:
280 The functions described in this section define a way to get specific
281 memory allocation statistics.
282 memkind_update_cached_stats() is used to force an update of cached dy‐
284 namic allocator statistics. Statistics are not updated real-time by
285 memkind library and this method allows to force its update.
286 memkind_get_stat() retrieves statistic of the specified type and re‐
288 turns it in value. For more details about stat see the MEMORY STATIS‐
289 TICS TYPE section below. Measured statistic applies to specific kind,
290 when NULL is given as kind then statistic applies to memory used by the
291 whole memkind library. Note: You need to call memkind_up‐
292 date_cached_stats() before calling memkind_get_stat() because statis‐
293 tics are cached by memkind library.
294 memkind_stats_print() prints summary statistics. This function wraps
296 jemalloc's function je_malloc_stats_print(). Uses write_cb function to
297 print the output. While providing custom writer function, use
298 syscall(2) rather than write(2). Pass NULL in order to use the default
299 write_cb function which prints the output to the stderr. Use cbopaque
300 parameter in order to pass some data to your write_cb function. Pass
301 additional options using opts. For more details on opts see the MEMORY
302 STATISTICS PRINT OPTIONS section below. Returns MEMKIND_ERROR_INVALID
303 when failed to parse options string, MEMKIND_SUCCESS on success.
304 DECORATORS:
306 The memkind library enables the user to define decorator functions that
307 can be called before and after each memkind heap management API. The
308 decorators that are called at the beginning of the function end are
309 named after that function with _pre appended to the name and those that
310 are called at the end of the function are named after that function
311 with _post appended to the name. These are weak symbols and if they are
312 not present at link time they are not called. The memkind library does
313 not define these symbols which are reserved for user definition. These
314 decorators can be used to track calls to the heap management interface
315 or to modify parameters. The decorators that are called at the begin‐
316 ning of the allocator pass all inputs by reference and the decorators
317 that are called at the end of the allocator pass the output by refer‐
318 ence. This enables the modification of the input and output of each
319 heap management function by the decorators.
320 ENHANCEMENTS:
322 memkind_set_bg_threads() enables/disables internal background worker
323 threads in jemalloc.
324 LIBRARY VERSION:
326 The memkind library version scheme consist major, minor and patch num‐
327 bers separated by dot. Combining those numbers, we got the following
328 representation:
329 major.minor.patch, where:
330 -major number is incremented whenever API is changed (loss of
331 backward compatibility),
332 -minor number is incremented whenever additional extensions are
333 introduced or behavior has been changed,
334 -patch number is incremented whenever small bug fixes are added.
335 memkind library provide numeric representation of the version by expos‐
337 ing the following API:
338 memkind_get_version() returns version number represented by a single
340 integer number, obtained from the formula:
341 major * 1000000 + minor * 1000 + patch
342 Note: major < 1 means unstable API.
344 API standards:
346 -STANDARD API, API is considered as stable
347 -NON-STANDARD API, API is considered as stable, however this is not a
348 standard way to use memkind
349 -EXPERIMENTAL API, API is considered as unstable and the subject to
350 change
351
354 memkind_calloc(), memkind_malloc(), memkind_realloc() and memkind_de‐
355 frag_reallocate() returns the pointer to the allocated memory or NULL
356 if the request fails. memkind_malloc_usable_size() returns the number
357 of usable bytes in the block of allocated memory pointed to by ptr, a
358 pointer to a block of memory allocated by memkind_malloc() or a related
359 function. If ptr is NULL, 0 is returned. memkind_free() and
360 memkind_error_message() do not have return values. All other memkind
361 API's return 0 upon success and an error code defined in the ERRORS
362 section upon failure. The memkind library avoids setting errno di‐
363 rectly, but calls to underlying libraries and system calls may set er‐
364 rno (e.g. memkind_create_pmem()).
365
367 The available kinds of memory:
368 MEMKIND_DEFAULT
370 Default allocation using standard memory and default page size.
371 MEMKIND_HIGHEST_CAPACITY
373 Allocate from a NUMA node(s) that has the highest capacity among
374 all nodes in the system.
375 MEMKIND_HIGHEST_CAPACITY_PREFERRED
377 Same as MEMKIND_HIGHEST_CAPACITY except that if there is not
378 enough memory in the NUMA node that has the highest capacity in
379 the local domain to satisfy the request, the allocation will
380 fall back on other memory NUMA nodes. Note: For this kind, the
381 allocation will not succeed if there are two or more NUMA nodes
382 that have the highest capacity.
383 MEMKIND_HIGHEST_CAPACITY_LOCAL
385 Allocate from a NUMA node that has the highest capacity among
386 all NUMA Nodes from the local domain. NUMA Nodes have the same
387 local domain for a set of CPUs associated with them, e.g. socket
388 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
389 the same local domain that have the highest capacity - alloca‐
390 tion will be done from NUMA node with worse latency attribute.
391 This kind requires locality information described in SYSTEM CON‐
392 FIGURATION section.
393 MEMKIND_HIGHEST_CAPACITY_LOCAL_PREFERRED
395 Same as MEMKIND_HIGHEST_CAPACITY_LOCAL except that if there is
396 not enough memory in the NUMA node that has the highest capacity
397 to satisfy the request, the allocation will fall back on other
398 memory NUMA nodes.
399 MEMKIND_LOWEST_LATENCY_LOCAL
401 Allocate from a NUMA node that has the lowest latency among all
402 NUMA Nodes from the local domain. NUMA Nodes have the same lo‐
403 cal domain for a set of CPUs associated with them, e.g. socket
404 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
405 the same local domain that have the lowest latency - allocation
406 will be done from NUMA node with smaller memory capacity. This
407 kind requires locality and memory performance characteristics
408 information described in SYSTEM CONFIGURATION section.
409 MEMKIND_LOWEST_LATENCY_LOCAL_PREFERRED
411 Same as MEMKIND_LOWEST_LATENCY_LOCAL except that if there is not
412 enough memory in the NUMA node that has the lowest latency to
413 satisfy the request, the allocation will fall back on other mem‐
414 ory NUMA nodes.
415 MEMKIND_HIGHEST_BANDWIDTH_LOCAL
417 Allocate from a NUMA node that has the highest bandwidth among
418 all NUMA Nodes from the local domain. NUMA Nodes have the same
419 local domain for a set of CPUs associated with them, e.g. socket
420 or sub-NUMA cluster. Note: If there are multiple NUMA nodes in
421 the same local domain that have the highest bandwidth - alloca‐
422 tion will be done from NUMA node with smaller memory capacity.
423 This kind requires locality and memory performance characteris‐
424 tics information described in SYSTEM CONFIGURATION section.
425 MEMKIND_HIGHEST_BANDWIDTH_LOCAL_PREFERRED
427 Same as MEMKIND_HIGHEST_BANDWIDTH_LOCAL except that if there is
428 not enough memory in the NUMA node that has the highest band‐
429 width to satisfy the request, the allocation will fall back on
430 other memory NUMA nodes.
431 MEMKIND_HUGETLB
433 Allocate from standard memory using huge pages. Note: This kind
434 requires huge pages configuration described in SYSTEM CONFIGURA‐
435 TION section.
436 MEMKIND_GBTLB (DEPRECATED)
438 Allocate from standard memory using 1GB chunks backed by huge
439 pages. Note: This kind requires huge pages configuration de‐
440 scribed in SYSTEM CONFIGURATION section.
441 MEMKIND_INTERLEAVE
443 Allocate pages interleaved across all NUMA nodes with transpar‐
444 ent huge pages disabled.
445 MEMKIND_HBW
447 Allocate from the closest high bandwidth memory NUMA node(s) at
448 the time of allocation. If there is not enough high bandwidth
449 memory to satisfy the request errno is set to ENOMEM and the al‐
450 located pointer is set to NULL. Note: This kind requires memory
451 performance characteristics information described in SYSTEM CON‐
452 FIGURATION section.
453 MEMKIND_HBW_ALL
455 Same as MEMKIND_HBW except decision regarding closest NUMA
456 node(s) is postponed until the time of first write.
457 MEMKIND_HBW_HUGETLB
459 Same as MEMKIND_HBW except the allocation is backed by huge
460 pages. Note: This kind requires huge pages configuration de‐
461 scribed in SYSTEM CONFIGURATION section.
462 MEMKIND_HBW_ALL_HUGETLB
464 Combination of MEMKIND_HBW_ALL and MEMKIND_HBW_HUGETLB proper‐
465 ties. Note: This kind requires huge pages configuration de‐
466 scribed in SYSTEM CONFIGURATION section.
467 MEMKIND_HBW_PREFERRED
469 Same as MEMKIND_HBW except that if there is not enough high
470 bandwidth memory to satisfy the request, the allocation will
471 fall back on standard memory. Note: For this kind, the alloca‐
472 tion will not succeed if two or more high bandwidth memory NUMA
473 nodes are in the same shortest distance to the same CPU on which
474 process is eligible to run. Check on that eligibility is done
475 upon starting the application.
476 MEMKIND_HBW_PREFERRED_HUGETLB
478 Same as MEMKIND_HBW_PREFERRED except the allocation is backed by
479 huge pages. Note: This kind requires huge pages configuration
480 described in SYSTEM CONFIGURATION section.
481 MEMKIND_HBW_GBTLB (DEPRECATED)
483 Same as MEMKIND_HBW except the allocation is backed by 1GB
484 chunks of huge pages. Note that size can take on any value, but
485 full gigabyte pages will allocated for each request, so remain‐
486 der of the last page will be wasted. This kind requires huge
487 pages configuration described in SYSTEM CONFIGURATION section.
488 MEMKIND_HBW_PREFERRED_GBTLB (DEPRECATED)
490 Same as MEMKIND_HBW_GBTLB except that if there is not enough
491 high bandwidth memory to satisfy the request, the allocation
492 will fall back on standard memory. Note: This kind requires
493 huge pages configuration described in SYSTEM CONFIGURATION sec‐
494 tion. For this kind, the allocation will not succeed if two or
495 more high bandwidth memory NUMA nodes are in the same shortest
496 distance to the same CPU on which process is eligible to run.
497 Check on that eligibility is done upon starting the application.
498 MEMKIND_HBW_INTERLEAVE
500 Same as MEMKIND_HBW except that the pages that support the allo‐
501 cation are interleaved across all high bandwidth nodes and
502 transparent huge pages are disabled.
503 MEMKIND_DAX_KMEM
505 Allocate from the closest persistent memory NUMA node at the
506 time of allocation. If there is not enough memory in the closest
507 persistent memory NUMA node to satisfy the request errno is set
508 to ENOMEM and the allocated pointer is set to NULL.
509 MEMKIND_DAX_KMEM_ALL
511 Allocate from the closest persistent memory NUMA node available
512 at the time of allocation. If there is not enough memory on any
513 of persistent memory NUMA nodes to satisfy the request errno is
514 set to ENOMEM and the allocated pointer is set to NULL.
515 MEMKIND_DAX_KMEM_PREFERRED
517 Same as MEMKIND_DAX_KMEM except that if there is not enough mem‐
518 ory in the closest persistent memory NUMA node to satisfy the
519 request, the allocation will fall back on other memory NUMA
520 nodes. Note: For this kind, the allocation will not succeed if
521 two or more persistent memory NUMA nodes are in the same short‐
522 est distance to the same CPU on which process is eligible to
523 run. Check on that eligibility is done upon starting the appli‐
524 cation.
525 MEMKIND_DAX_KMEM_INTERLEAVE
527 Same as MEMKIND_DAX_KMEM except that the pages that support the
528 allocation are interleaved across all persistent memory NUMA
529 nodes.
530 MEMKIND_REGULAR
532 Allocate from regular memory using the default page size. Regu‐
533 lar means general purpose memory from the NUMA nodes containing
534 CPUs.
535
537 The available types of memory:
538 MEMKIND_MEMTYPE_DEFAULT
540 Standard memory, the same as process uses.
541 MEMKIND_MEMTYPE_HIGH_BANDWIDTH
543 High bandwidth memory (HBM). There must be at least two memory
544 types with different bandwidth to determine which is the HBM.
545
547 The available types of memory binding policy:
548 MEMKIND_POLICY_BIND_LOCAL
550 Allocate local memory. If there is not enough memory to satisfy
551 the request errno is set to ENOMEM and the allocated pointer is
552 set to NULL.
553 MEMKIND_POLICY_BIND_ALL
555 Memory locality is ignored. If there is not enough memory to
556 satisfy the request errno is set to ENOMEM and the allocated
557 pointer is set to NULL.
558 MEMKIND_POLICY_PREFERRED_LOCAL
560 Allocate preferred memory that is local. If there is not enough
561 preferred memory to satisfy the request or preferred memory is
562 not available, the allocation will fall back on any other mem‐
563 ory.
564 MEMKIND_POLICY_INTERLEAVE_LOCAL
566 Interleave allocation across local memory. For n memory types
567 the allocation will be interleaved across all of them.
568 MEMKIND_POLICY_INTERLEAVE_ALL
570 Interleave allocation. Locality is ignored. For n memory types
571 the allocation will be interleaved across all of them.
572 MEMKIND_POLICY_MAX_VALUE
574 Max policy value.
575
577 The available types of memory flags:
578 MEMKIND_MASK_PAGE_SIZE_2MB
580 Allocation backed by 2MB page size.
581
583 The available types of memory usage policy:
584 MEMKIND_MEM_USAGE_POLICY_DEFAULT
586 Default memory usage policy.
587 MEMKIND_MEM_USAGE_POLICY_CONSERVATIVE
589 Conservative memory usage policy - prioritize memory usage at
590 cost of performance. Note: Memory usage policies have no effect
591 for TBB heap manager described in ENVIRONMENT section.
592
594 The available types of memory statistics:
595 MEMKIND_STAT_TYPE_RESIDENT
597 Maximum number of bytes in physically resident data pages
598 mapped.
599 MEMKIND_STAT_TYPE_ACTIVE
601 Total number of bytes in active pages.
602 MEMKIND_STAT_TYPE_ALLOCATED
604 Total number of allocated bytes.
605
607 The available options for printing statistics:
608 MEMKIND_STAT_PRINT_ALL
610 Print all statistics.
611 MEMKIND_STAT_PRINT_JSON_FORMAT
613 Print statistics in JSON format.
614 MEMKIND_STAT_PRINT_OMIT_GENERAL
616 Omit general information that never changes during execution.
617 MEMKIND_STAT_PRINT_OMIT_MERGED_ARENA
619 Omit merged arena statistics.
620 MEMKIND_STAT_PRINT_OMIT_DESTROYED_MERGED_ARENA
622 Omit destroyed merged arena statistics.
623 MEMKIND_STAT_PRINT_OMIT_PER_ARENA
625 Omit per arena statistics.
626 MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_BINS
628 Omit per size class statistics for bins.
629 MEMKIND_STAT_PRINT_OMIT_PER_SIZE_CLASS_LARGE
631 Omit per size class statistics for large objects.
632 MEMKIND_STAT_PRINT_OMIT_MUTEX
634 Omit all mutex statistics.
635 MEMKIND_STAT_PRINT_OMIT_EXTENT
637 Omit extent statistics.
638
640 memkind_posix_memalign()
641 returns the one of the POSIX standard error codes EINVAL or
642 ENOMEM as defined in <errno.h> if an error occurs (these have
643 positive values). If the alignment parameter is not a power of
644 two or is not a multiple of sizeof(void*), then EINVAL is re‐
645 turned. If there is insufficient memory to satisfy the request
646 then ENOMEM is returned.
647 All functions other than memkind_posix_memalign() which have an integer
649 return type return one of the negative error codes as defined in
650 <memkind.h> and described below.
651 MEMKIND_ERROR_UNAVAILABLE
653 Requested memory kind is not available
654 MEMKIND_ERROR_MBIND
656 Call to mbind(2) failed
657 MEMKIND_ERROR_MMAP
659 Call to mmap(2) failed
660 MEMKIND_ERROR_MALLOC
662 Call to jemalloc's malloc() failed
663 MEMKIND_ERROR_ENVIRON
665 Error parsing environment variable MEMKIND_*
666 MEMKIND_ERROR_INVALID
668 Invalid input arguments to memkind routine
669 MEMKIND_ERROR_TOOMANY
671 Error trying to initialize more than maximum MEMKIND_MAX_KIND
672 number of kinds
673 MEMKIND_ERROR_BADOPS
675 Error memkind operation structure is missing or invalid
676 MEMKIND_ERROR_HUGETLB
678 Unable to allocate huge pages
679 MEMKIND_ERROR_MEMTYPE_NOT_AVAILABLE
681 Error requested memory type is not available
682 MEMKIND_ERROR_OPERATION_FAILED
684 Error memkind operation failed
685 MEMKIND_ERROR_ARENAS_CREATE
687 Call to jemalloc's arenas.create() failed
688 MEMKIND_ERROR_RUNTIME
690 Unspecified run-time error
691
693 /usr/bin/memkind-hbw-nodes
694 Prints a comma-separated list of high bandwidth nodes.
695 /usr/bin/memkind-auto-dax-kmem-nodes
697 Prints a comma-separated list of persistent memory NUMA nodes,
698 which are automatically detected.
699
701 MEMKIND_HBW_NODES
702 This environment variable is a comma-separated list of NUMA
703 nodes that are treated as high bandwidth. Uses the libnuma rou‐
704 tine numa_parse_nodestring() for parsing, so the syntax de‐
705 scribed in the numa(3) man page for this routine applies: e.g.
706 1-3,5 is a valid setting.
707 MEMKIND_HBW_THRESHOLD
709 This environment variable is bandwidth in MB/s that is the
710 threshold for identifying high bandwidth memory. The default
711 threshold is 204800 (200 GB/s), which is used if this variable
712 is not set. When set, it must be greater than or equal to 0.
713 MEMKIND_DAX_KMEM_NODES
715 This environment variable is a comma-separated list of NUMA
716 nodes that are treated as PMEM memory. Uses the libnuma routine
717 numa_parse_nodestring() for parsing, so the syntax described in
718 the numa(3) man page for this routine applies: e.g. 1-3,5 is a
719 valid setting.
720 MEMKIND_ARENA_NUM_PER_KIND
722 This environment variable allows leveraging internal mechanism
723 of the library for setting number of arenas per kind. Value
724 should be a positive integer (not greater than INT_MAX defined
725 in <limits.h>). The user should set the value based on the
726 characteristics of application that is using the library. Higher
727 value can provide better performance in extremely multithreaded
728 applications at the cost of memory overhead. See section IMPLE‐
729 MENTATION NOTES of jemalloc(3) for more details about arenas.
730 MEMKIND_HOG_MEMORY
732 Controls behavior of memkind with regards to returning memory to
733 underlying OS. Setting MEMKIND_HOG_MEMORY to 1 causes memkind to
734 not release memory to OS in anticipation of memory reuse soon.
735 This will improve latency of 'free' operations but increase mem‐
736 ory usage. Note: For file-backed kind memory will be released
737 to OS only after calling memkind_destroy_kind(), not after
738 'free' operations. In context of MEMKIND_MEM_USAGE_POLICY_CON‐
739 SERVATIVE memory usage policy - it will also impact memory coa‐
740 lescing and results that blocks pages will be often reused (bet‐
741 ter memory usage at cost of performance).
742 MEMKIND_DEBUG
744 Controls logging mechanism in memkind. Setting MEMKIND_DEBUG to
745 1 enables printing messages like errors and general information
746 about environment to stderr.
747 MEMKIND_BACKGROUND_THREAD_LIMIT
749 Enable background worker threads. Value should be from range 0
750 to maximum number of cpus. Setting MEMKIND_BACK‐
751 GROUND_THREAD_LIMIT to specific value will limit maximum number
752 of background worker threads to this value. 0 means maximum
753 number of background worker threads will be limited to maximum
754 number of cpus.
755 MEMKIND_HEAP_MANAGER
757 Controls heap management behavior in memkind library by switch‐
758 ing to one of the available heap managers.
759 Values:
760 JEMALLOC - sets the jemalloc heap manager
761 TBB - sets the Intel Threading Building Blocks heap manager.
762 This option requires installed
763 Intel Threading Building Blocks library.
764 If the MEMKIND_HEAP_MANAGER is not set then the jemalloc heap manager
766 will be used by default.
767
769 Interfaces for obtaining 2MB (HUGETLB) memory need allocated huge pages
770 in the kernel's huge page pool.
771 HUGETLB (huge pages)
773 Current number of "persistent" huge pages can be read from
774 /proc/sys/vm/nr_hugepages file. Proposed way of setting
775 hugepages is: sudo sysctl vm.nr_hugepages=<number_of_hugepages>.
776 More information can be found here:
777 ⟨https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt⟩
778 Interfaces for obtaining locality information are provided by libhwloc
780 dependency. Functionality based on locality requires that memkind li‐
781 brary is configured and built with the support of libhwloc (./configure
782 --enable-hwloc).
783 Interfaces for obtaining memory performance characteristics information
785 are based on HMAT (Heterogeneous Memory Attribute Table)
786 ⟨https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf⟩
787 Functionality based on memory performance characteristics requires that
788 platform configuration fully supports HMAT and memkind library is con‐
789 figured and built with the support of libhwloc (./configure --enable-
790 hwloc).
791 Note: For a given target NUMA Node, the OS exposes only the performance
793 characteristics of the best performing NUMA node.
794 libhwloc can be reached on: ⟨https://www.open-mpi.org/projects/hwloc⟩
796
798 When linking statically against memkind, libmemkind.a should be used
799 together with its dependencies libnuma and pthread. Pthread can be
800 linked by adding /usr/lib64/libpthread.a as a dependency (exact path
801 may vary). Typically libnuma will need to be compiled from sources to
802 use it as a static dependency. libnuma can be reached on GitHub:
803 ⟨https://github.com/numactl/numactl⟩
804
806 HUGETLB (huge pages)
807 There might be some overhead in huge pages consumption caused by
808 heap management. If your allocation fails because of OOM,
809 please try to allocate extra huge pages (e.g. 8 huge pages).
810
812 Copyright (C) 2014 - 2021 Intel Corporation. All rights reserved.
813
815 malloc(3), malloc_usable_size(3), numa(3), hwloc(3), numactl(8),
816 mbind(2), mmap(2), move_pages(2), jemalloc(3), memkind_dax_kmem(3),
817 memkind_default(3), memkind_arena(3), memkind_hbw(3),
818 memkind_hugetlb(3), memkind_pmem(3), syscall(2), write(2)
819Intel Corporation 2015-03-31 MEMKIND(3)