1MEMTIER(7)                          MEMTIER                         MEMTIER(7)
2
3
4

NAME

6       libmemtier.so  -  interposer  library  which enables the memkind memory
7       tiering
8
9

SYNOPSIS

11       In order to use the memkind memory tiering, pass the following environ‐
12       ment variables along with the command:
13
14              LD_PRELOAD=libmemtier.so MEMKIND_MEM_TIERS="..."  command {argu‐
15              ments ...}
16
17

DESCRIPTION

19       This library enables the memkind memory tiering  mechanism.  With  this
20       funcionality, allocations will be split between different types of mem‐
21       ory automatically. The library allows making allocations with the usage
22       of  multiple  kinds  keeping a specified ratio between them. This ratio
23       determines how much of the total allocated memory should  be  allocated
24       with each kind. Both LD_PRELOAD and MEMKIND_MEM_TIERS environment vari‐
25       ables are mandatory for enabling the memkind memory tiering.
26
27

ENVIRONMENT

29       MEMKIND_MEM_TIERS
30              A semicolon-separated list of tier configurations ended  with  a
31              policy  parameter.  Each  of the tier configurations is a comma-
32              separated   list   of   tier   parameters   in    the    format:
33              param_name:value,(...)   For  the list of available tier parame‐
34              ters, please see the TIER PARAMETERS section below.
35
36       MEMKIND_MEM_THRESHOLDS
37              Semicolon-separated list of threshold configurations. Each  con‐
38              figuration  consists  of a comma-separated list of threshold pa‐
39              rameters in the same format as  MEMKIND_MEM_TIERS  ,  where  the
40              param_name  is  one  of the INIT_VAL, MIN_VAL and MAX_VAL.  Each
41              parameter can be set only once. When any of the  parameters  are
42              missing, they will be set to a default value.  If threshold con‐
43              figuration is missing, threshold with default configuration will
44              be used. If any of the parameters is set more than once, the ap‐
45              plication will be aborted. Order of provided  configurations  is
46              important - configuration of Nth threshold defines threshold be‐
47              tween Nth and (N+1)th tier. Note that for N+1 tiers there should
48              be  at  most  N thresholds defined.  Set this variable only when
49              DYNAMIC_THRESHOLD  policy  is  used.  For  more  details   about
50              param_name  see the THRESHOLD PARAMETERS section below. See also
51              EXAMPLES section for the usage of  MEMKIND_MEM_THRESHOLDS  envi‐
52              ronment variable.
53
54

TIER PARAMETERS

56       KIND   (required)  -  kind  of memory used in memory tier. Allowed kind
57              names are 'DRAM', 'KMEM_DAX' and 'FS_DAX'.  There can be  multi‐
58              ple different kind names provided in the MEMKIND_MEM_TIERS envi‐
59              ronment variable - one in each tier configuration. Additionally,
60              in  this  variable  more than one 'FS_DAX' kind name is allowed.
61              More information about available  kinds  can  be  found  in  the
62              memkind(3)  manual,  where the 'DRAM' kind name allocates memory
63              with the usage of the MEMKIND_DEFAULT kind, the 'KMEM_DAX'  with
64              the  MEMKIND_DAX_KMEM  kind  and the 'FS_DAX' with a file-backed
65              kind of memory created with the memkind_create_pmem() function.
66
67       PATH   (required only for the FS_DAX kind) - the path to  the  location
68              where  pmem  file  will  be  created. The path has to exist. For
69              other kinds, if set, will cause an error.
70
71       PMEM_SIZE_LIMIT
72              (optional, only for the FS_DAX kind) - if  set,  it  limits  the
73              size of pmem file and the maximum size of total allocations from
74              persistent memory. By default, no limit is introduced. Pass this
75              option  only  with  FS_DAX  kind.  For other kinds, if set, will
76              cause an error.  The accepted formats are: 1, 1K,  1M,  1G.  See
77              the  memkind(3) manual for information about limitations to this
78              value which are the same  as  for  the  max_size  value  of  the
79              memkind_create_pmem() function.
80
81       RATIO  (required)  - the part of the ratio tied to the given kind. It's
82              an unsigned type in a range from 1 to  UINT_MAX.   See  EXAMPLES
83              section.
84
85       POLICY (required,  only one in the whole environment variable) - deter‐
86              mines the algorithm used to distribute allocations between  pro‐
87              vided  memory kinds. This parameter has to be the last parameter
88              in  MEMKIND_MEM_TIERS  configuration  string.   Currently   only
89              STATIC_RATIO  and  DYNAMIC_THRESHOLD policies are valid. See the
90              POLICIES section.
91
92       NOTE: The application will  fail  when  provided  environment  variable
93       string is not in the correct format.
94
95

POLICIES

97       STATIC_RATIO
98              All  allocations  are  made in such a way that the ratio between
99              memory tiers is constant. No threshold is used.
100
101       DYNAMIC_THRESHOLD
102              The ratio between memory tiers is kept with a help of a  thresh‐
103              old  between  kinds  which  value  changes in time.  Minimum two
104              tiers are required for this policy,  otherwise  the  application
105              will  be aborted. The threshold value can change in time to keep
106              the desired ratio between tiers, but it will not be lesser  than
107              MIN_VAL  and it will not be greater than MAX_VAL.  For every al‐
108              location, if its size is greater than or equal to  INIT_VAL,  it
109              will come from the (N+1)th tier.
110
111              Default  values  for  a threshold between first two tiers in the
112              MEMKIND_MEM_TIERS environment variable are:
113
114              INIT_VAL = 1024, MIN_VAL = 513, MAX_VAL = 1536.
115
116              If there are more tiers defined, each next  undefined  threshold
117              will  have  all  parameters increased by 1024, so the next unde‐
118              fined threshold between the next two tiers will have:
119
120              INIT_VAL = 2048, MIN_VAL = 1537, MAX_VAL = 2560.
121
122

THRESHOLD PARAMETERS

124       INIT_VAL
125              (optional) - the initial value of the threshold between two  ad‐
126              jacent  Nth  and (N+1)th tiers. It must be greater than or equal
127              to MIN_VAL and less than or equal to MAX_VAL.
128
129       MIN_VAL
130              (optional) - the minimum value of the threshold.
131
132       MAX_VAL
133              (optional) - the maximum value of the threshold.
134
135       NOTE: Because setting the above parameters is optional,  they  will  be
136       set to default values in case they are not defined.
137
138

DRAM FALLBACK POLICY

140       With  the usage of both DRAM and KMEM_DAX tiers, if there is not enough
141       memory to satisfy the DRAM tier memory allocation request, the  alloca‐
142       tion will fall back to PMEM memory.
143
144       With  the  usage  of both DRAM and FS_DAX tiers, if there is not enough
145       memory to satisfy the DRAM tier memory allocation request, the  alloca‐
146       tion will fail.
147
148       If  there  is  not enough memory to satisfy the FS_DAX or KMEM_DAX tier
149       memory allocation request, the allocation will fail.
150
151

EXAMPLES

153       The following example will run ls with the memkind memory  tiering  li‐
154       brary. Make sure that paths to both libmemtier.so and libmemkind.so are
155       included in LD_LIBRARY_PATH.  During the application run,  20%  of  the
156       allocated memory will come from PMEM memory and 80% will come from DRAM
157       ( FS_DAX:DRAM ratio is 1:4):
158
159              LD_PRELOAD=libmemtier.so
160              MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,PMEM_SIZE_LIMIT:10G,RA‐
161              TIO:1;KIND:DRAM,RATIO:4;POLICY:STATIC_RATIO" /bin/ls -l
162
163       The example value of MEMKIND_MEM_TIERS environment variable  where  all
164       allocations will come from PMEM memory with filesystem created with the
165       path /mnt/pmem0 (PMEM file  size  is  limited  only  by  the  specified
166       filesystem):
167
168              LD_PRELOAD=libmemtier.so
169              MEMKIND_MEM_TIERS="KIND:FS_DAX,PATH:/mnt/pmem0,RATIO:1;POL‐
170              ICY:STATIC_RATIO"
171
172       The  example value of MEMKIND_MEM_THRESHOLDS environment variable. With
173       INIT_VAL=64, on the application start all  allocations  lower  than  64
174       bytes  threshold  will come from DRAM and equal to or greater than this
175       value will come from  PMEM  memory  NUMA  nodes.  The  threshold  value
176       changes  during  the runtime in order to maintain the ratio.  MIN_VAL=1
177       and MAX_VAL=10000 set the lower  and  upper  limits  of  the  threshold
178       value.  Note  that  the  DYNAMIC_THRESHOLD  policy  has  to  be  set in
179       MEMKIND_MEM_TIERS environment variable:
180
181              LD_PRELOAD=libmemtier.so        MEMKIND_MEM_TIERS="KIND:DRAM,RA‐
182              TIO:1;KIND:KMEM_DAX,RATIO:4;POLICY:DYNAMIC_THRESHOLD"
183              MEMKIND_MEM_THRESHOLDS="INIT_VAL=64,MIN_VAL=1,MAX_VAL=10000"
184
185

NOTES

187       libmemtier works for applications that do not statically link a  malloc
188       implementation.
189
190
192       Copyright (C) 2021 Intel Corporation. All rights reserved.
193
194

SEE ALSO

196       memkind(3), malloc(3)
197
198
199
200Intel Corporation                 2021-03-01                        MEMTIER(7)
Impressum