condor_adstash - HTCondor Manual

       HTCondor Team

--schedd_history
                 Poll and push condor_schedd job histories

          --startd_history
                 Poll and push condor_startd job histories

          --ad_filePATH
                 Load  Job  ClassAds  from  a  file  instead  of  querying daemons (Ignores --schedd_history and
                 --startd_history.)

       1990-2024,  Center  for High Throughput Computing, Computer Sciences Department, University of Wisconsin-
       Madison, Madison, WI, US. Licensed under the Apache License, Version 2.0.

                                                  Jan 04, 2025                                 CONDOR_ADSTASH(1)

condor_adstash is a tool that assists in monitoring usage  by  gathering  job  ClassAds  (typically  from
       condor_schedd  and/or condor_startd history queries) and pushing the ClassAds as documents to some target
       (typically Elasticsearch).

       Unless run in --standalone mode, condor_adstash expects to be invoked as a  daemon  by  a  condor_master,
       i.e.  condor_adstash  should be invoked in standalone mode when run on the command-line.  Whether invoked
       by condor_master or run standalone, condor_adstash gets its configuration, in increasing  priority,  from
       the  HTCondor  configuration  macros  beginning with ADSTASH_ (when --process_name is not provided), then
       environment variables, and finally command-line options.

       condor_adstash must be able to write its --checkpoint_file to a persistent location so that duplicate job
       ClassAds are not fetched from the daemons' histories in consecutive polls.

       A named Elasticsearch index will be created if it doesn't exist,  and  may  be  modified  if  new  fields
       (corresponding  to  ClassAd  attribute  names)  need  to  be added.  It is up to the administrator of the
       Elasticsearch instance to install rollover policies (e.g. ILM) on the named index and/or to  set  up  the
       index as an alias.

       Running  condor_adstash  in standalone mode on the command-line will result in condor_adstash reading its
       configuration from the current HTCondor configuration:

          $ condor_adstash --standalone

       By default, condor_adstash looks for HTCondor  configuration  variables  with  names  are  prefixed  with
       ADSTASH_, e.g. ADSTASH_READ_SCHEDDS=*.  These values can be overridden on the command-line:

          $ condor_adstash --standalone --schedds=myschedd.localdomain

       condor_adstash configuration variables can be also be named using custom prefixes, with the prefix passed
       in using -\-process_name=PREFIX.  For example, if the HTCondor configuration contained FOO_SCHEDD_HISTORY=False  and  FOO_STARTD_HISTORY=True,  condor_adstash  can  be  invoked  to  read  these instead of
       ADSTASH_SCHEDD_HISTORY and ADSTASH_STARTD_HISTORY:

          $ condor_adstash --standalone --process_name=FOO

       Providing a PREFIX to --process_name that does not match any HTCondor configuration variables will  cause
       condor_adstash to fallback to a default set of configuration values, which may be useful in debugging.

       The  configuration  values  that  condor_adstash  reads  from  the  current HTCondor configuration can be
       previewed by printing the help message.  The values will  be  listed  as  the  default  values  for  each
       command-line option:

          $ condor_adstash --help
          $ condor_adstash --process_name=FOO --help

--json_dirPATH
                 Directory to store JSON files, which are named by timestamp

       condor_adstash - HTCondor Manual

       Gather schedd and/or startd job history ClassAds and push them via a search engine or file interface.

-h, --help
                 Display the help message and exit.

          --process_namePREFIX
                 Give  condor_adstash  a  different  name  for looking up HTCondor configuration and environment
                 variable values (see examples).

          --standalone
                 Run condor_adstash in standalone mode (runs once, does not attempt to contact condor_master)

          --sample_intervalSECONDS
                 Number of seconds between polling the list(s) of daemons (ignored in standalone mode)

          --checkpoint_filePATH
                 Location of checkpoint file (will be created if missing)

          --log_filePATH
                 Location of log file

          --log_levelLEVEL
                 Log level (uses Python logging library levels: CRITICAL/ERROR/WARNING/INFO/DEBUG)

          --threadsTHREADS
                 Number of parallel threads to use when polling for job ClassAds and when pushing  documents  to
                 Elasticsearch

          --interface{null,elasticsearch,opensearch,jsonfile}
                 Push ads via the chosen interface

--collectorsCOLLECTORS
                 Comma-separated list of condor_collector addresses  to  contact  to  locate  condor_schedd  and
                 condor_startd daemons

          --scheddsSCHEDDS
                 Comma-separated list of condor_schedd names to poll job histories from

          --startdsSTARTDS
                 Comma-separated list of condor_startd machines to poll job histories from

          --schedd_history_max_adsNUM_ADS
                 Abort after reading NUM_ADS from a condor_schedd--startd_history_max_adsNUM_ADS
                 Abort after reading NUM_ADS from a condor_startd--schedd_history_timeoutSECONDS
                 Abort if reading from a condor_schedd takes more than this many seconds

          --startd_history_timeoutSECONDS
                 Abort if reading from a condor_startd takes more than this many seconds

--se_hostHOST[:PORT]
                 Search engine host:port

          --se_url_prefixPREFIX
                 Search engine URL prefix

          --se_usernameUSERNAME
                 Search engine username

          --se_use_https
                 Use HTTPS when connecting to search engine

          --se_timeoutSECONDS
                 Max time to wait for search engine queries

          --se_bunch_sizeNUM_DOCS
                 Group ads in bunches of this size to send to search engine

          --se_index_nameINDEX_NAME
                 Push ads to this search engine index or alias

          --se_no_log_mappings
                 Don't write a JSON file with mappings to the log directory

          --se_ca_certsPATH
                 Path to root certificate authority file (will use certifi's CA if not set)

condor_adstash [--help ]

       condor_adstash [--process_nameNAME] [--standalone ] [--sample_intervalSECONDS] [--checkpoint_filePATH]
       [--log_filePATH]  [--log_levelLEVEL]  [--threadsTHREADS] [--interface{null,elasticsearch,jsonfile}]
       [--collectorsCOLLECTORS] [--scheddsSCHEDDS] [--startdsSTARTDS] [--schedd_history ] [--startd_history ]
       [--ad_filePATH]     [--schedd_history_max_adsNUM_ADS]     [--startd_history_max_adsNUM_ADS]
       [--schedd_history_timeoutSECONDS]    [--startd_history_timeoutSECONDS]   [--se_hostHOST[:PORT]]
       [--se_url_prefixPREFIX]   [--se_usernameUSERNAME]   [--se_use_https   ]   [--se_timeout    SECONDS]
       [--se_bunch_sizeNUM_DOCS]  [--es_index_nameINDEX_NAME]  [--se_no_log_mappings]  [--se_ca_certsPATH]
       [--json_dirPATH]