1() PMDK Programmer's Manual ()
2
6 libpmem2 - persistent memory support library
7 NOTE: Support for Windows and FreeBSD deprecated since PMDK
9 1.13.0 release and will be removed in the PMDK 1.14.0 release.
10
12 #include <libpmem2.h>
13 cc ... -lpmem2
14
16 libpmem2 provides low-level persistent memory (pmem) support for appli‐
17 cations using direct access storage (DAX), which is storage that sup‐
18 ports load/store access without paging blocks from a block storage de‐
19 vice. Some types of non-volatile memory DIMMs (NVDIMMs) provide this
20 type of byte addressable access to storage. A persistent memory aware
21 file system is typically used to expose the direct access to applica‐
22 tions. Memory mapping a file from this type of file system results in
23 the load/store, non-paged access to pmem.
24 This library is for applications that use persistent memory directly,
26 without the help of any library-supplied transactions or memory alloca‐
27 tion. Higher-level libraries that currently build on libpmem (previous
28 variation of libpmem2) are available and are recommended for most ap‐
29 plications, see:
30 • libpmemobj(7), a general use persistent memory API, providing memory
32 allocation and transactional operations on variable-sized objects.
33 • libpmemblk(7), providing pmem-resident arrays of fixed-sized blocks
35 with atomic updates.
36 • libpmemlog(7), providing a pmem-resident log file.
38 The libpmem2 library provides a comprehensive set of functions for ro‐
40 bust use of Persistent Memory. It relies on three core concepts:
41 struct pmem2_src source, struct pmem2_config config and struct
42 pmem2_map map:
43 • source - an object describing the data source for mapping. The data
45 source can be a file descriptor, a file handle, or an anonymous map‐
46 ping. APIs dedicated for creating source are:
47 pmem2_source_from_fd(3), pmem2_source_from_handle(3),
48 pmem2_source_from_anon(3).
49 • config - an object containing parameters that are used to create a
51 mapping from a source. The configuration structure must always be
52 provided to create a mapping, but the only required parameter to set
53 in the config is granularity. The granularity should by set using
54 dedicated libpmem2 function pmem2_config_set_required_store_granular‐
55 ity(3) which defines a maximum permitted granularity requested by the
56 user. For more information about the granularity concept read GRANU‐
57 LARITY section below.
58 In addition to the granularity setting, libpmem2 provides multiple op‐
60 tional functions to configure target mapping, e.g., pmem2_con‐
61 fig_set_length(3) to set length which will be used for mapping, or
62 pmem2_config_set_offset(3) which will be used to map the contents from
63 the specified location of the source, pmem2_config_set_sharing(3) which
64 defines the behavior and visibility of writes to the mapping’s pages.
65 • map - an object created by pmem2_map_new(3) using source and config
67 as an input parameters. The map structure can be then used to di‐
68 rectly operate on the created mapping through the use of its associ‐
69 ated set of functions: pmem2_map_get_address(3),
70 pmem2_map_get_size(3), pmem2_map_get_store_granularity(3) - for get‐
71 ting address, size and effective mapping granularity.
72 In addition to the basic functionality of managing the virtual address
74 mapping, libpmem2 also provides optimized functions for modifying the
75 mapped data. This includes data flushing as well as memory copying.
76 To get proper function for data flushing use: pmem2_get_flush_fn(3),
78 pmem2_get_persist_fn(3) or pmem2_get_drain_fn(3). To get proper func‐
79 tion for copying to persistent memory, use map getters: pmem2_get_mem‐
80 cpy_fn(3), pmem2_get_memset_fn(3), pmem2_get_memmove_fn(3).
81 The libpmem2 API also provides support for the badblock and unsafe
83 shutdown state handling.
84 To read or clear badblocks, the following functions are provided:
86 pmem2_badblock_context_new(3), pmem2_badblock_context_delete(3),
87 pmem2_badblock_next(3) and pmem2_badblock_clear(3).
88 To handle unsafe shutdown in the application, the following functions
90 are provided: pmem2_source_device_id(3), pmem2_source_device_usc(3).
91 More detailed information about unsafe shutdown detection and unsafe
92 shutdown count can be found in the libpmem2_unsafe_shutdown(7) man
93 page.
94
96 The libpmem2 library introduces the concept of granularity through
97 which you may easily distinguish between different levels of storage
98 performance capabilities available to the application as related to
99 power-fail protected domain. The way data reaches this protected do‐
100 main differs based on the platform and storage device capabilities.
101 Traditional block storage devices (SSD, HDD) must use system API calls
103 such as msync(), fsync() on Linux, or FlushFileBuffers(),FlushViewOf‐
104 File() on Windows to write data reliably. Invoking these functions
105 flushes the data to the medium with page granularity. In the libpmem2
106 library, this type of flushing behavior is called PMEM2_GRANULARI‐
107 TY_PAGE.
108 In systems with persistent memory support, a power-fail protected do‐
110 main may cover different sets of resources: either the memory con‐
111 troller or the memory controller and CPU caches. For this reason,
112 libpmem2 distinguishes two types of granularity for persistent memory:
113 PMEM2_GRANULARITY_CACHE_LINE and PMEM2_GRANULARITY_BYTE.
114 If the power-fail protected domain covers only the memory controller,
116 the CPU appropriate cache lines must be flushed for the data to be con‐
117 sidered persistent. This granularity type is called PMEM2_GRANULARI‐
118 TY_CACHE_LINE. Depending on the architecture, there are different
119 types of machine instructions for flushing cache lines (e.g., CLWB,
120 CLFLUSHOPT, CLFLUSH for Intel x86_64 architecture). Usually, to ensure
121 the ordering of stores, such instructions must be followed by a barrier
122 (e.g., SFENCE).
123 The third type of granularity PMEM2_GRANULARITY_BYTE applies to plat‐
125 forms where power-fail protected domain covers both the memory con‐
126 troller and CPU caches. In such cases, cache flush instructions are no
127 longer needed, and the platform itself guarantees the persistence of
128 data. But barriers might still be required for ordering.
129 The library declares these granularity level in pmem2_granularity enum,
131 which the application must set in pmem2_config to the appropriate level
132 for a mapping to succeed. The software should set this config parame‐
133 ter to a value that most accurately represents the target hardware
134 characteristics and the storage patterns of the application. For exam‐
135 ple, a database storage engine that operates on large logical pages
136 that reside either on SSDs or PMEM should set this value to PMEM2_GRAN‐
137 ULARITY_PAGE. The library will create mappings where the new map gran‐
138 ularity is lower or equal to the requested one. For example, a mapping
139 with PMEM2_GRANULARITY_CACHE_LINE can be created for the required gran‐
140 ularity PMEM2_GRANULARITY_PAGE, but not vice versa.
141
143 libpmem2 relies on the library destructor being called from the main
144 thread. For this reason, all functions that might trigger destruction
145 (e.g. dlclose(3)) should be called in the main thread. Otherwise some
146 of the resources associated with that thread might not be cleaned up
147 properly.
148
150 libpmem2 can change its default behavior based on the following envi‐
151 ronment variables. These are primarily intended for testing and are
152 generally not required.
153 • PMEM2_FORCE_GRANULARITY=val
155 Setting this environment variable to val forces libpmem2 to use persist
157 method specific for forced granularity and skip granularity autodetect‐
158 ing mechanism. The concept of the granularity is described in GRANU‐
159 LARITY section above. This variable is intended for use during library
160 testing.
161 The val argument accepts following text values:
163 • BYTE - force byte granularity.
165 • CACHE_LINE - force cache line granularity.
167 • PAGE - force page granularity.
169 Granularity values listed above are case-insensitive.
171 NOTE: The value of PMEM2_FORCE_GRANULARITY is not queried (and
173 cached) at library initialization time, but read during each
174 pmem2_map_new(3) call.
175 This means that PMEM2_FORCE_GRANULARITY may still be set or modified by
177 the program until the first attempt to map a file.
178 • PMEM_NO_CLWB=1
180 Setting this environment variable to 1 forces libpmem2 to never issue
182 the CLWB instruction on Intel hardware, falling back to other cache
183 flush instructions on that hardware instead (CLFLUSHOPT or CLFLUSH).
184 Without this setting, libpmem2 will always use the CLWB instruction for
185 flushing processor caches on platforms that support this instruction.
186 This variable is intended for use during library testing, but may be
187 required for some rare cases when using CLWB has a negative impact on
188 performance.
189 • PMEM_NO_CLFLUSHOPT=1
191 Setting this environment variable to 1 forces libpmem2 to never issue
193 the CLFLUSHOPT instruction on Intel hardware, falling back to the
194 CLFLUSH instructions instead. Without this environment variable, libp‐
195 mem2 will always use the CLFLUSHOPT instruction for flushing processor
196 caches on platforms that support the instruction, but where CLWB is not
197 available. This variable is intended for use during library testing.
198 • PMEM_NO_MOVNT=1
200 Setting this environment variable to 1 forces libpmem2 to never use the
202 non-temporal move instructions on Intel hardware. Without this envi‐
203 ronment variable, libpmem2 will use the non-temporal instructions for
204 copying larger ranges to persistent memory on platforms that support
205 these instructions. This variable is intended for use during library
206 testing.
207 • PMEM_MOVNT_THRESHOLD=val
209 This environment variable allows overriding the minimum length of the
211 pmem2_memmove_fn operations, for which libpmem2 uses non-temporal move
212 instructions. Setting this environment variable to 0 forces libpmem2
213 to always use the non-temporal move instructions if available. It has
214 no effect if PMEM_NO_MOVNT is set to 1. This variable is intended for
215 use during library testing.
216
218 Two versions of libpmem2 are typically available on a development sys‐
219 tem. The normal version, accessed when a program is linked using the
220 -lpmem2 option, is optimized for performance. That version skips
221 checks that impact performance and never logs any trace information or
222 performs any run-time assertions.
223 A second version of libpmem2, accessed when a program uses the li‐
225 braries under /usr/lib/pmdk_debug, contains run-time assertions and
226 trace points. The typical way to access the debug version is to set
227 the environment variable LD_LIBRARY_PATH to /usr/lib/pmdk_debug or
228 /usr/lib64/pmdk_debug, as appropriate. Debugging output is controlled
229 using the following environment variables. These variables have no ef‐
230 fect on the non-debug version of the library.
231 • PMEM2_LOG_LEVEL
233 The value of PMEM2_LOG_LEVEL enables trace points in the debug version
235 of the library, as follows:
236 • 0 - This is the default level when PMEM2_LOG_LEVEL is not set. No
238 log messages are emitted at this level.
239 • 1 - Additional details on any errors detected are logged, in addition
241 to returning the errno-based errors as usual. The same information
242 may be retrieved using pmem2_errormsg().
243 • 2 - A trace of basic operations is logged.
245 • 3 - Enables a very verbose amount of function call tracing in the li‐
247 brary.
248 • 4 - Enables voluminous and fairly obscure tracing information that is
250 likely only useful to the libpmem2 developers.
251 Unless PMEM2_LOG_FILE is set, debugging output is written to stderr.
253 • PMEM2_LOG_FILE
255 Specifies the name of a file where all logging information should be
257 written. If the last character in the name is “-”, the PID of the cur‐
258 rent process will be appended to the file name when the log file is
259 created. If PMEM2_LOG_FILE is not set, output is written to stderr.
260
262 The following example uses libpmem2 to flush changes made to raw, memo‐
263 ry-mapped persistent memory.
264 WARNING: There is nothing transactional about the persist from
266 pmem2_get_persist_fn(3) call in this example. Interrupting the
267 program may result in a partial write to pmem. Use a transac‐
268 tional library such as libpmemobj(7) to avoid torn updates.
269 The basic example can be found in the repository under path src/exam‐
271 ples/libpmem2/basic/basic.c (https://github.com/pmem/pmdk/blob/mas‐
272 ter/src/examples/libpmem2/basic/basic.c). It is described in detail
273 here (https://pmem.io/pmdk/libpmem2/).
274
276 libpmem2 builds on the persistent memory programming model recommended
277 by the SNIA NVM Programming Technical Work Group:
278 <https://snia.org/nvmp>
279
281 FlushFileBuffers(), fsync(2), msync(2), pmem2_config_set_length(3),
282 pmem2_config_set_offset(3), pmem2_config_set_required_store_granulari‐
283 ty(3), pmem2_config_set_sharing(3),pmem2_get_drain_fn(3),
284 pmem2_get_flush_fn(3), pmem2_get_memcpy_fn(3), pmem2_get_memmove_fn(3),
285 pmem2_get_memset_fn(3), pmem2_get_per‐
286 sist_fn(3),pmem2_map_get_store_granularity(3), pmem2_map_new(3),
287 pmem2_source_from_anon(3), pmem2_source_from_fd(3),
288 pmem2_source_from_handle(3), libpmem2_unsafe_shutdown(7), libpmem‐
289 blk(7), libpmemlog(7), libpmemobj(7) and <https://pmem.io>
290PMDK - 2023-06-05 ()