1MEMTIER(7) MEMTIER MEMTIER(7)
2
6 libmemtier.so - interposer library which enables the memkind memory
7 tiering
8
11 In order to use the memkind memory tiering, pass the following environ‐
12 ment variables along with the command:
13 LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="..." command {argu‐
15 ments ...}
16
19 This library enables the memkind memory tiering mechanism. With this
20 funcionality, allocations will be split between different types of mem‐
21 ory automatically. The library allows making allocations with the usage
22 of multiple kinds keeping a specified ratio between them. This ratio
23 determines how much of the total allocated memory should be allocated
24 with each kind. Both LD_PRELOAD and MEMKIND_MEM_TIERS environment vari‐
25 ables are mandatory for enabling the memkind memory tiering.
26
29 MEMKIND_MEM_TIERS
30 A semicolon-separated list of tier configurations ended with a
31 policy parameter. Each of the tier configurations is a comma-
32 separated list of tier parameters in the format:
33 param_name:value,(...) For the list of available tier parame‐
34 ters, please see the TIER PARAMETERS section below.
35 MEMKIND_MEM_THRESHOLDS
37 Semicolon-separated list of threshold configurations. Each con‐
38 figuration consists of a comma-separated list of threshold pa‐
39 rameters in the same format as MEMKIND_MEM_TIERS , where the
40 param_name is one of the INIT_VAL, MIN_VAL and MAX_VAL. Each
41 parameter can be set only once. When any of the parameters are
42 missing, they will be set to a default value. If threshold con‐
43 figuration is missing, threshold with default configuration will
44 be used. If any of the parameters is set more than once, the ap‐
45 plication will be aborted. Order of provided configurations is
46 important - configuration of Nth threshold defines threshold be‐
47 tween Nth and (N+1)th tier. Note that for N+1 tiers there should
48 be at most N thresholds defined. Set this variable only when
49 DYNAMIC_THRESHOLD policy is used. For more details about
50 param_name see the THRESHOLD PARAMETERS section below. See also
51 EXAMPLES section for the usage of MEMKIND_MEM_THRESHOLDS envi‐
52 ronment variable.
53
56 KIND (required) - kind of memory used in memory tier. Allowed kind
57 names are 'DRAM', 'KMEM_DAX' and 'FS_DAX'. There can be multi‐
58 ple different kind names provided in the MEMKIND_MEM_TIERS envi‐
59 ronment variable - one in each tier configuration. Additionally,
60 in this variable more than one 'FS_DAX' kind name is allowed.
61 More information about available kinds can be found in the
62 memkind(3) manual, where the 'DRAM' kind name allocates memory
63 with the usage of the MEMKIND_DEFAULT kind, the 'KMEM_DAX' with
64 the MEMKIND_DAX_KMEM kind and the 'FS_DAX' with a file-backed
65 kind of memory created with the memkind_create_pmem() function.
66 PATH (required only for the FS_DAX kind) - the path to the location
68 where pmem file will be created. The path has to exist. For
69 other kinds, if set, will cause an error.
70 PMEM_SIZE_LIMIT
72 (optional, only for the FS_DAX kind) - if set, it limits the
73 size of pmem file and the maximum size of total allocations from
74 persistent memory. By default, no limit is introduced. Pass this
75 option only with FS_DAX kind. For other kinds, if set, will
76 cause an error. The accepted formats are: 1, 1K, 1M, 1G. See
77 the memkind(3) manual for information about limitations to this
78 value which are the same as for the max_size value of the
79 memkind_create_pmem() function.
80 RATIO (required) - the part of the ratio tied to the given kind. It's
82 an unsigned type in a range from 1 to UINT_MAX. See EXAMPLES
83 section.
84 POLICY (required, only one in the whole environment variable) - deter‐
86 mines the algorithm used to distribute allocations between pro‐
87 vided memory kinds. This parameter has to be the last parameter
88 in MEMKIND_MEM_TIERS configuration string. Currently only
89 STATIC_RATIO and DYNAMIC_THRESHOLD policies are valid. See the
90 POLICIES section.
91 NOTE: The application will fail when provided environment variable
93 string is not in the correct format.
94
97 STATIC_RATIO
98 All allocations are made in such a way that the ratio between
99 memory tiers is constant. No threshold is used.
100 DYNAMIC_THRESHOLD
102 The ratio between memory tiers is kept with a help of a thresh‐
103 old between kinds which value changes in time. Minimum two
104 tiers are required for this policy, otherwise the application
105 will be aborted. The threshold value can change in time to keep
106 the desired ratio between tiers, but it will not be lesser than
107 MIN_VAL and it will not be greater than MAX_VAL. For every al‐
108 location, if its size is greater than or equal to INIT_VAL, it
109 will come from the (N+1)th tier.
110 Default values for a threshold between first two tiers in the
112 MEMKIND_MEM_TIERS environment variable are:
113 INIT_VAL = 1024, MIN_VAL = 513, MAX_VAL = 1536.
115 If there are more tiers defined, each next undefined threshold
117 will have all parameters increased by 1024, so the next unde‐
118 fined threshold between the next two tiers will have:
119 INIT_VAL = 2048, MIN_VAL = 1537, MAX_VAL = 2560.
121
124 INIT_VAL
125 (optional) - the initial value of the threshold between two ad‐
126 jacent Nth and (N+1)th tiers. It must be greater than or equal
127 to MIN_VAL and less than or equal to MAX_VAL.
128 MIN_VAL
130 (optional) - the minimum value of the threshold.
131 MAX_VAL
133 (optional) - the maximum value of the threshold.
134 NOTE: Because setting the above parameters is optional, they will be
136 set to default values in case they are not defined.
137
140 With the usage of both DRAM and KMEM_DAX tiers, if there is not enough
141 memory to satisfy the DRAM tier memory allocation request, the alloca‐
142 tion will fall back to PMEM memory.
143 With the usage of both DRAM and FS_DAX tiers, if there is not enough
145 memory to satisfy the DRAM tier memory allocation request, the alloca‐
146 tion will fail.
147 If there is not enough memory to satisfy the FS_DAX or KMEM_DAX tier
149 memory allocation request, the allocation will fail.
150
153 The following example will run ls with the memkind memory tiering li‐
154 brary. Make sure that paths to both libmemtier.so and libmemkind.so are
155 included in LD_LIBRARY_PATH. During the application run, 20% of the
156 allocated memory will come from PMEM memory and 80% will come from DRAM
157 ( FS_DAX:DRAM ratio is 1:4):
158 LD_PRELOAD=libmemtier.so
160 MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,PMEM_SIZE_LIMIT:10G,RA‐
161 TIO:1;KIND:DRAM,RATIO:4;POLICY:STATIC_RATIO" /bin/ls -l
162 The example value of MEMKIND_MEM_TIERS environment variable where all
164 allocations will come from PMEM memory with filesystem created with the
165 path /mnt/pmem0 (PMEM file size is limited only by the specified
166 filesystem):
167 LD_PRELOAD=libmemtier.so
169 MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,RATIO:1;POL‐
170 ICY:STATIC_RATIO"
171 The example value of MEMKIND_MEM_THRESHOLDS environment variable. With
173 INIT_VAL=64, on the application start all allocations lower than 64
174 bytes threshold will come from DRAM and equal to or greater than this
175 value will come from PMEM memory NUMA nodes. The threshold value
176 changes during the runtime in order to maintain the ratio. MIN_VAL=1
177 and MAX_VAL=10000 set the lower and upper limits of the threshold
178 value. Note that the DYNAMIC_THRESHOLD policy has to be set in
179 MEMKIND_MEM_TIERS environment variable:
180 LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="KIND:DRAM,RA‐
182 TIO:1;KIND:KMEM_DAX,RATIO:4;POLICY:DYNAMIC_THRESHOLD"
183 MEMKIND_MEM_THRESHOLDS="INIT_VAL=64,MIN_VAL=1,MAX_VAL=10000"
184
187 libmemtier works for applications that do not statically link a malloc
188 implementation.
189
192 Copyright (C) 2021 Intel Corporation. All rights reserved.
193
196 memkind(3), malloc(3)
197Intel Corporation 2021-03-01 MEMTIER(7)