1CONDOR_ADSTASH(1) HTCondor Manual CONDOR_ADSTASH(1)
2
6 condor_adstash - HTCondor Manual
7 Gather schedd and/or startd job history ClassAds and push them via a
9 search engine or file interface.
10
13 condor_adstash [--help ]
14 condor_adstash [--process_name NAME] [--standalone ] [--sample_interval
16 SECONDS] [--checkpoint_file PATH] [--log_file PATH] [--log_level LEVEL]
17 [--threads THREADS] [--interface {null,elasticsearch,jsonfile}] [--col‐
18 lectors COLLECTORS] [--schedds SCHEDDS] [--startds STARTDS]
19 [--schedd_history ] [--startd_history ] [--ad_file PATH] [--schedd_his‐
20 tory_max_ads NUM_ADS] [--startd_history_max_ads NUM_ADS] [--schedd_his‐
21 tory_timeout SECONDS] [--startd_history_timeout SECONDS] [--se_host
22 HOST[:PORT]] [--se_url_prefix PREFIX] [--se_username USERNAME]
23 [--se_use_https ] [--se_timeout SECONDS] [--se_bunch_size NUM_DOCS]
24 [--es_index_name INDEX_NAME] [--se_no_log_mappings] [--se_ca_certs
25 PATH] [--json_dir PATH]
26
28 condor_adstash is a tool that assists in monitoring usage by gathering
29 job ClassAds (typically from condor_schedd and/or condor_startd history
30 queries) and pushing the ClassAds as documents to some target (typi‐
31 cally Elasticsearch).
32 Unless run in --standalone mode, condor_adstash expects to be invoked
34 as a daemon by a condor_master, i.e. condor_adstash should be invoked
35 in standalone mode when run on the command-line. Whether invoked by
36 condor_master or run standalone, condor_adstash gets its configuration,
37 in increasing priority, from the HTCondor configuration macros begin‐
38 ning with ADSTASH_ (when --process_name is not provided), then environ‐
39 ment variables, and finally command-line options.
40 condor_adstash must be able to write its --checkpoint_file to a persis‐
42 tent location so that duplicate job ClassAds are not fetched from the
43 daemons' histories in consecutive polls.
44 A named Elasticsearch index will be created if it doesn't exist, and
46 may be modified if new fields (corresponding to ClassAd attribute
47 names) need to be added. It is up to the administrator of the Elastic‐
48 search instance to install rollover policies (e.g. ILM) on the named
49 index and/or to set up the index as an alias.
50
52 -h, --help
53 Display the help message and exit.
54 --process_name PREFIX
56 Give condor_adstash a different name for looking up HTCondor
57 configuration and environment variable values (see examples).
58 --standalone
60 Run condor_adstash in standalone mode (runs once, does not
61 attempt to contact condor_master)
62 --sample_interval SECONDS
64 Number of seconds between polling the list(s) of daemons (ig‐
65 nored in standalone mode)
66 --checkpoint_file PATH
68 Location of checkpoint file (will be created if missing)
69 --log_file PATH
71 Location of log file
72 --log_level LEVEL
74 Log level (uses Python logging library levels: CRITICAL/ER‐
75 ROR/WARNING/INFO/DEBUG)
76 --threads THREADS
78 Number of parallel threads to use when polling for job Clas‐
79 sAds and when pushing documents to Elasticsearch
80 --interface {null,elasticsearch,opensearch,jsonfile}
82 Push ads via the chosen interface
83
85 --schedd_history
86 Poll and push condor_schedd job histories
87 --startd_history
89 Poll and push condor_startd job histories
90 --ad_file PATH
92 Load Job ClassAds from a file instead of querying daemons
93 (Ignores --schedd_history and --startd_history.)
94
96 --collectors COLLECTORS
97 Comma-separated list of condor_collector addresses to contact
98 to locate condor_schedd and condor_startd daemons
99 --schedds SCHEDDS
101 Comma-separated list of condor_schedd names to poll job his‐
102 tories from
103 --startds STARTDS
105 Comma-separated list of condor_startd machines to poll job
106 histories from
107 --schedd_history_max_ads NUM_ADS
109 Abort after reading NUM_ADS from a condor_schedd
110 --startd_history_max_ads NUM_ADS
112 Abort after reading NUM_ADS from a condor_startd
113 --schedd_history_timeout SECONDS
115 Abort if reading from a condor_schedd takes more than this
116 many seconds
117 --startd_history_timeout SECONDS
119 Abort if reading from a condor_startd takes more than this
120 many seconds
121
123 --se_host HOST[:PORT]
124 Search engine host:port
125 --se_url_prefix PREFIX
127 Search engine URL prefix
128 --se_username USERNAME
130 Search engine username
131 --se_use_https
133 Use HTTPS when connecting to search engine
134 --se_timeout SECONDS
136 Max time to wait for search engine queries
137 --se_bunch_size NUM_DOCS
139 Group ads in bunches of this size to send to search engine
140 --se_index_name INDEX_NAME
142 Push ads to this search engine index or alias
143 --se_no_log_mappings
145 Don't write a JSON file with mappings to the log directory
146 --se_ca_certs PATH
148 Path to root certificate authority file (will use certifi's
149 CA if not set)
150
152 --json_dir PATH
153 Directory to store JSON files, which are named by timestamp
154
156 Running condor_adstash in standalone mode on the command-line will re‐
157 sult in condor_adstash reading its configuration from the current HT‐
158 Condor configuration:
159 $ condor_adstash --standalone
161 By default, condor_adstash looks for HTCondor configuration variables
163 with names are prefixed with ADSTASH_, e.g. ADSTASH_READ_SCHEDDS = *.
164 These values can be overridden on the command-line:
165 $ condor_adstash --standalone --schedds=myschedd.localdomain
167 condor_adstash configuration variables can be also be named using cus‐
169 tom prefixes, with the prefix passed in using -\-process_name=PREFIX.
170 For example, if the HTCondor configuration contained FOO_SCHEDD_HISTORY
171 = False and FOO_STARTD_HISTORY = True, condor_adstash can be invoked to
172 read these instead of ADSTASH_SCHEDD_HISTORY and ADSTASH_STARTD_HIS‐
173 TORY:
174 $ condor_adstash --standalone --process_name=FOO
176 Providing a PREFIX to --process_name that does not match any HTCondor
178 configuration variables will cause condor_adstash to fallback to a de‐
179 fault set of configuration values, which may be useful in debugging.
180 The configuration values that condor_adstash reads from the current HT‐
182 Condor configuration can be previewed by printing the help message.
183 The values will be listed as the default values for each command-line
184 option:
185 $ condor_adstash --help
187 $ condor_adstash --process_name=FOO --help
188
190 HTCondor Team
191
193 1990-2023, Center for High Throughput Computing, Computer Sciences De‐
194 partment, University of Wisconsin-Madison, Madison, WI, US. Licensed
195 under the Apache License, Version 2.0.
196 Oct 02, 2023 CONDOR_ADSTASH(1)