1sh5util(1) Slurm Commands sh5util(1)
2
6 sh5util - Tool for merging HDF5 files from the acct_gather_profile
7 plugin that gathers detailed data for jobs running under Slurm
8
11 sh5util
12
15 sh5util merges HDF5 files produced on each node for each step of a job
16 into one HDF5 file for the job. The resulting file can be viewed and
17 manipulated by common HDF5 tools such as HDF5View, h5dump, h5edit, or
18 h5ls.
19 sh5util also has two extract modes. The first, writes a limited set of
21 data for specific nodes, steps, and data series in "comma separated
22 value" form to a file which can be imported into other analysis tools
23 such as spreadsheets.
24 The second, (Item-Extract) extracts one data time from one time series
26 for all the samples on all the nodes from a jobs HDF5 profile.
27 - Finds sample with maximum value of the item.
29 - Write CSV file with min, ave, max, and item totals for each node for
31 each sample
32
36 -E, --extract
37 Extract data series from a merged job file.
39 Extract mode options
42 -i, --input=path
45 merged file to extract from (default ./job_$jobid.h5)
46 -N, --node=nodename
49 Node name to extract (default is all)
50 -l, --level=[Node:Totals | Node:TimeSeries]
53 Level to which series is attached. (default Node:To‐
54 tals)
55 -s, --series=[Energy | Filesystem | Network | Task | Task_#]
58 Task is all tasks, Task_# (# is a task id) (default is
59 everything)
60 -h, --help
63 Print this description of use.
64 -I, --item-extract
67 Extract one data item from all samples of one data series from
69 all nodes in a merged job file.
70 Item-Extract mode options
73 -s, --series=[Energy | Filesystem | Network | Task]
76 -d, --data
79 Name of data item in series (See note below).
80 -j, --jobs=<job[.step]>
84 Format is <job[.step]>. Merge this job/step (or a comma-sepa‐
85 rated list of job steps). This option is required. Not specify‐
86 ing a step will result in all steps found to be processed.
87 -L, --list
90 Print the items of a series contained in a job file.
92 List mode options
94 -i, --input=path
97 Merged file to extract from (default ./job_$jobid.h5)
98 -s, --series=[Energy | Filesystem | Network | Task]
101 -o, --output=<path>
104 Path to a file into which to write.
105 Default for merge is ./job_$jobid.h5
106 Default for extract is ./extract_$jobid.csv
107 -p, --profiledir=<dir>
110 Directory location where node-step files exist default is set in
111 acct_gather.conf.
112 -S, --savefiles
115 Instead of removing node-step files after merging them into the
116 job file, keep them around.
117 --usage
120 Display brief usage message.
121 --user=<user>
124 User who profiled job. (Handy for root user, defaults to user
125 running this command.)
126
129 Energy
130 Power
131 CPU_Frequency
132 Filesystem
135 Reads
136 Megabytes_Read
137 Writes
138 Megabytes_Write
139 Network
142 Packets_In
143 Megabytes_In
144 Packets_Out
145 Megabytes_Out
146 Task
149 CPU_Frequency
150 CPU_Time
151 CPU_Utilization
152 RSS
153 VM_Size
154 Pages
155 Read_Megabytes
156 Write_Megabytes
157
160 Executing sh5util sends a remote procedure call to slurmctld. If enough
161 calls from sh5util or other Slurm client commands that send remote pro‐
162 cedure calls to the slurmctld daemon come in at once, it can result in
163 a degradation of performance of the slurmctld daemon, possibly result‐
164 ing in a denial of service.
165 Do not run sh5util or other Slurm client commands that send remote pro‐
167 cedure calls to slurmctld from loops in shell scripts or other pro‐
168 grams. Ensure that programs limit calls to sh5util to the minimum nec‐
169 essary for the information you are trying to gather.
170
173 Merge node-step files (as part of a sbatch script):
174 $ sbatch -n1 -d$SLURM_JOB_ID --wrap="sh5util --savefiles -j $SLURM_JOB_ID"
176 Extract all task data from a node:
179 $ sh5util -j 42 -N snowflake01 --level=Node:TimeSeries --series=Tasks
181 Extract all energy data:
184 $ sh5util -j 42 --series=Energy --data=power
186
189 Copyright (C) 2013 Bull.
190 Copyright (C) 2013-2021 SchedMD LLC. Slurm is free software; you can
191 redistribute it and/or modify it under the terms of the GNU General
192 Public License as published by the Free Software Foundation; either
193 version 2 of the License, or (at your option) any later version.
194 Slurm is distributed in the hope that it will be useful, but WITHOUT
196 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
197 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
198 for more details.
199
202February 2021 Slurm Commands sh5util(1)