LIBVMMALLOC(7) PMDK Programmer's Manual LIBVMMALLOC(7)
libvmmalloc - general purpose volatile memory allocation library
$ LD_PRELOAD=libvmmalloc.so.1 command [ args... ]
cc [ flag... ] file... -lvmmalloc [ library... ]
void *malloc(size_t size);
void free(void *ptr);
void *calloc(size_t number, size_t size);
void *realloc(void *ptr, size_t size);
int posix_memalign(void **memptr, size_t alignment, size_t size);
void *aligned_alloc(size_t alignment, size_t size);
void *memalign(size_t alignment, size_t size);
void *valloc(size_t size);
void *pvalloc(size_t size);
size_t malloc_usable_size(const void *ptr);
void cfree(void *ptr);
libvmmalloc transparently converts all dynamic memory allocations into
Persistent Memory allocations.
The typical usage of libvmmalloc does not require any modification of
the target program. It is enough to load libvmmalloc before all other
libraries by setting the environment variable LD_PRELOAD. When used in
that way, libvmmalloc interposes the standard system memory allocation
routines, as defined in malloc(3), posix_memalign(3) and malloc_us‐
able_size(3), and provides that all dynamic memory allocations are made
from a memory pool built on a memory-mapped file, instead of the system
heap. The memory managed by libvmmalloc may have different attributes,
depending on the file system containing the memory-mapped file. In
particular, libvmmalloc is part of the Persistent Memory Development
Kit because it is sometimes useful to use non-volatile memory as a
volatile memory pool, leveraging its capacity, cost, or performance
libvmmalloc may be also linked to the program, by providing the **-lvm‐
malloc* argument to the linker. Then it becomes the default memory al‐
locator for the program.
NOTE: Due to the fact the library operates on a memory-mapped
file, it may not work properly with programs that perform
fork(2) not followed by exec(3). There are two variants of ex‐
perimental fork(2) support available in libvmmalloc. The de‐
sired library behavior may be selected by setting the VMMAL‐
LOC_FORK environment variable. By default variant #1 is en‐
abled. See ENVIRONMENT for more details.
libvmmalloc uses the mmap(2) system call to create a pool of volatile
memory. The library is most useful when used with Direct Access stor‐
age (DAX), which is memory-addressable persistent storage that supports
load/store access without being paged via the system page cache. A
Persistent Memory-aware file system is typically used to provide this
type of access. Memory-mapping a file from a Persistent Memory-aware
file system provides the raw memory pools, and this library supplies
the traditional malloc interfaces on top of those pools.
The memory pool acting as a system heap replacement is created automat‐
ically at library initialization time. The user may control its loca‐
tion and size by setting the environment variables described in ENVI‐
RONMENT, below. The allocated file space is reclaimed when the process
terminates or in case of system crash.
Under normal usage, libvmmalloc will never print messages or intention‐
ally cause the process to exit. The library uses pthreads(7) to be
fully MT-safe, but never creates or destroys threads itself. The li‐
brary does not make use of any signals, networking, and never calls se‐
lect(2) or poll(2).
The VMMALLOC_POOL_DIR and VMMALLOC_POOL_SIZE environment variables must
be set for libvmmalloc to work properly. If either of them is not
specified, or if their values are not valid, the library prints an ap‐
propriate error message and terminates the process. Any other environ‐
ment variables are optional.
Specifies a path to the directory where the memory pool file should be
created. The directory must exist and be writable.
Defines the desired size (in bytes) of the memory pool file. It must
be not less than the minimum allowed size VMMALLOC_MIN_POOL as defined
NOTE: Due to the fact the library adds some metadata to the mem‐
ory pool, the amount of actual usable space is typically less
than the size of the memory pool file.
· VMMALLOC_FORK=val (EXPERIMENTAL)
VMMALLOC_FORK controls the behavior of libvmmalloc in case of fork(3),
and can be set to the following values:
· 0 - fork(2) support is disabled. The behavior of fork(2) is unde‐
fined in this case, but most likely results in memory pool corruption
and a program crash due to segmentation fault.
· 1 - The memory pool file is remapped with the MAP_PRIVATE flag before
the fork completes. From this moment, any access to memory that mod‐
ifies the heap pages, both in the parent and in the child process,
will trigger creation of a copy of those pages in RAM
(copy-on-write). The benefit of this approach is that it does not
significantly increase the time of the initial fork operation, and
does not require additional space on the file system. However, all
subsequent memory allocations, and modifications of any memory allo‐
cated before fork, will consume system memory resources instead of
the memory pool.
This is the default option if VMMALLOC_FORK is not set.
· 2 - A copy of the entire memory pool file is created for the use of
the child process. This requires additional space on the file sys‐
tem, but both the parent and the child process may still operate on
their memory pools, not consuming system memory resources.
NOTE: In case of large memory pools, creating a copy of the pool file
may stall the fork operation for a quite long time.
· 3 - The library first attempts to create a copy of the memory pool
(as for option #2), but if it fails (i.e. because of insufficient
free space on the file system), it will fall back to option #1.
NOTE: Options 2 and 3 are not currently supported on FreeBSD.
Environment variables used for debugging are described in DEBUGGING,
libvmmalloc relies on the library destructor being called from the main
thread. For this reason, all functions that might trigger destruction
(e.g. dlclose(3)) should be called in the main thread. Otherwise some
of the resources associated with that thread might not be cleaned up
Two versions of libvmmalloc are typically available on a development
system. The normal version is optimized for performance. That version
skips checks that impact performance and never logs any trace informa‐
tion or performs any run-time assertions. A second version, accessed
when using libraries from /usr/lib/pmdk_debug, contains run-time asser‐
tions and trace points. The typical way to access the debug version is
to set the LD_LIBRARY_PATH environment variable to /usr/lib/pmdk_debug
or /usr/lib64/pmdk_debug, as appropriate. Debugging output is con‐
trolled using the following environment variables. These variables
have no effect on the non-debug version of the library.
The value of VMMALLOC_LOG_LEVEL enables trace points in the debug ver‐
sion of the library, as follows:
· 0 - Tracing is disabled. This is the default level when VMMAL‐
LOC_LOG_LEVEL is not set.
· 1 - Additional details on any errors detected are logged, in addition
to returning the errno-based errors as usual.
· 2 - A trace of basic operations is logged.
· 3 - Enables a very verbose amount of function call tracing in the li‐
· 4 - Enables voluminous tracing information about all memory alloca‐
tions and deallocations.
Unless VMMALLOC_LOG_FILE is set, debugging output is written to stderr.
Specifies the name of a file where all logging information should be
written. If the last character in the name is “-”, the PID of the cur‐
rent process will be appended to the file name when the log file is
created. If VMMALLOC_LOG_FILE is not set, output is written to stderr.
Setting VMMALLOC_LOG_STATS to 1 enables logging human-readable summary
statistics at program termination.
Unlike the normal malloc(3), which asks the system for additional memo‐
ry when it runs out, libvmmalloc allocates the size it is told to and
never attempts to grow or shrink that memory pool.
libvmmalloc may not work properly with programs that perform fork(2)
and do not call exec(3) immediately afterwards. See ENVIRONMENT for
more details about experimental fork(2) support.
If logging is enabled in the debug version of the library and the
process performs fork(2), no new log file is created for the child
process, even if the configured log file name ends with “-”. All log‐
ging information from the child process will be written to the log file
owned by the parent process, which may lead to corruption or partial
loss of log data.
Malloc hooks (see malloc_hook(3)), are not supported when using libvm‐
libvmmalloc depends on jemalloc, written by Jason Evans, to do the
heavy lifting of managing dynamic memory allocation. See:
fork(2), dlclose(3), exec(3), malloc(3), malloc_usable_size(3),
posix_memalign(3), libpmem(7), libvmem(7) and <http://pmem.io>
jemalloc(3), malloc_hook(3), pthreads(7), ld.so(8)
PMDK - vmmalloc API version 1.1 2018-07-18 LIBVMMALLOC(7)