DESCRIPTION

The file location can be modified at system build time using the DEFAULT_SLURM_CONF parameter or at execution time by setting the SLURM_CONF environment variable. The Slurm daemons also allow you to override both the built-in and environment-provided location using the "-f" option on the command line.

The contents of the file are case insensitive except for the names of nodes and partitions. Any text following a "#" in the configuration file is treated as a comment through the end of that line. Changes to the configuration file take effect upon restart of Slurm daemons, daemon receipt of the SIGHUP signal, or execution of the command "scontrol reconfigure" unless otherwise noted.

If a line begins with the word "Include" followed by whitespace and then a file name, that file will be included inline with the current configuration file. For large or complex systems, multiple configuration files may prove easier to manage and enable reuse of some files (See INCLUDE MODIFIERS for more details).

Note on file permissions:

The slurm.conf file must be readable by all users of Slurm, since it is used by many of the Slurm commands. Other files that are defined in the slurm.conf file, such as log files and job accounting files, may need to be created/owned by the user "SlurmUser" to be successfully accessed. Use the "chown" and "chmod" commands to set the ownership and permissions appropriately. See the section FILE AND DIRECTORY PERMISSIONS for information about the various files and directories used by Slurm.

PARAMETERS

The overall configuration parameters available include:

AccountingStorageBackupHost

The name of the backup machine hosting the accounting storage database. If used with the accounting_storage/slurmdbd plugin, this is where the backup slurmdbd would be running. Only used for database type storage plugins, ignored otherwise.

AccountingStorageEnforce

This controls what level of association-based enforcement to impose on job submissions. Valid options are any combination of associations, limits, nojobs, nosteps, qos, safe, and wckeys, or all for all things (expect nojobs and nosteps, they must be requested as well).

If limits, qos, or wckeys are set, associations will automatically be set.

If wckeys is set, TrackWCKey will automatically be set.

If safe is set, limits and associations will automatically be set.

If nojobs is set nosteps will automatically be set.

By enforcing Associations no new job is allowed to run unless a corresponding association exists in the system. If limits are enforced users can be limited by association to whatever job size or run time limits are defined.

If nojobs is set Slurm will not account for any jobs or steps on the system, like wise if nosteps is set Slurm will not account for any steps ran limits will still be enforced.

If safe is enforced a job will only be launched against an association or qos that has a GrpCPUMins limit set if the job will be able to run to completion. Without this option set, jobs will be launched as long as their usage hasn't reached the cpu-minutes limit which can lead to jobs being launched but then killed when the limit is reached.

With qos and/or wckeys enforced jobs will not be scheduled unless a valid qos and/or workload characterization key is specified.

When AccountingStorageEnforce is changed, a restart of the slurmctld daemon is required (not just a "scontrol reconfig").

AccountingStorageHost

The name of the machine hosting the accounting storage database. Only used for database type storage plugins, ignored otherwise. Also see DefaultStorageHost.

AccountingStorageLoc

The fully qualified file name where accounting records are written when the AccountingStorageType is "accounting_storage/filetxt" or else the name of the database where accounting records are stored when the AccountingStorageType is a database. Also see DefaultStorageLoc.

AccountingStoragePass

The password used to gain access to the database to store the accounting data. Only used for database type storage plugins, ignored otherwise. In the case of Slurm DBD (Database Daemon) with MUNGE authentication this can be configured to use a MUNGE daemon specifically configured to provide authentication between clusters while the default MUNGE daemon provides authentication within a cluster. In that case, AccountingStoragePass should specify the named port to be used for communications with the alternate MUNGE daemon (e.g. "/var/run/munge/global.socket.2"). The default value is NULL. Also see DefaultStoragePass.

AccountingStoragePort

The listening port of the accounting storage database server. Only used for database type storage plugins, ignored otherwise. Also see DefaultStoragePort.

AccountingStorageTRES

Comma separated list of resources you wish to track on the cluster. These are the resources requested by the sbatch/srun job when it is submitted. Currently this consists of any GRES, BB (burst buffer) or license along with CPU, Memory, Node, and Energy. By default CPU, Energy, Memory, and Node are tracked. AccountingStorageTRES=gres/craynetwork,license/iop1 will track cpu, energy, memory, nodes along with a gres called craynetwork as well as a license called iop1. Whenever these resources are used on the cluster they are recorded. The TRES are automatically set up in the database on the start of the slurmctld.

AccountingStorageType

The accounting storage mechanism type. Acceptable values at present include "accounting_storage/filetxt", "accounting_storage/mysql", "accounting_storage/none" and "accounting_storage/slurmdbd". The "accounting_storage/filetxt" value indicates that accounting records will be written to the file specified by the AccountingStorageLoc parameter. The "accounting_storage/mysql" value indicates that accounting records will be written to a MySQL or MariaDB database specified by the AccountingStorageLoc parameter. The "accounting_storage/slurmdbd" value indicates that accounting records will be written to the Slurm DBD, which manages an underlying MySQL database. See "man slurmdbd" for more information. The default value is "accounting_storage/none" and indicates that account records are not maintained. Note: The filetxt plugin records only a limited subset of accounting information and will prevent some sacct options from proper operation. Also see DefaultStorageType.

AccountingStorageUser

The user account for accessing the accounting storage database. Only used for database type storage plugins, ignored otherwise. Also see DefaultStorageUser.

AccountingStoreJobComment

If set to "YES" then include the job's comment field in the job complete message sent to the Accounting Storage database. The default is "YES".

AcctGatherNodeFreq

The AcctGather plugins sampling interval for node accounting. For AcctGather plugin values of none, this parameter is ignored. For all other values this parameter is the number of seconds between node accounting samples. For the acct_gather_energy/rapl plugin, set a value less than 300 because the counters may overflow beyond this rate. The default value is zero. This value disables accounting sampling for nodes. Note: The accounting sampling interval for jobs is determined by the value of JobAcctGatherFrequency.

AcctGatherEnergyType

Identifies the plugin to be used for energy consumption accounting. The jobacct_gather plugin and slurmd daemon call this plugin to collect energy consumption data for jobs and nodes. The collection of energy consumption data takes place on node level, hence only in case of exclusive job allocation the energy consumption measurements will reflect the jobs real consumption. In case of node sharing between jobs the reported consumed energy per job (through sstat or sacct) will not reflect the real energy consumed by the jobs.

Configurable values at present are:

acct_gather_energy/none

No energy consumption data is collected.

acct_gather_energy/ipmi

Energy consumption data is collected from the Baseboard Management Controller (BMC) using the Intelligent Platform Management Interface (IPMI).

acct_gather_energy/rapl

Energy consumption data is collected from hardware sensors using the Running Average Power Limit (RAPL) mechanism. Note that enabling RAPL may require the execution of the command "sudo modprobe msr".

AcctGatherInfinibandType

Identifies the plugin to be used for infiniband network traffic accounting. The plugin is activated only when profiling on hdf5 files is activated and the user asks for network data collection for jobs through --profile=Network (or =All). The collection of network traffic data takes place on node level, hence only in case of exclusive job allocation the collected values will reflect the jobs real traffic. All network traffic data are logged on hdf5 files per job on each node. No storage on the Slurm database takes place.

Configurable values at present are:

acct_gather_infiniband/none

No infiniband network data are collected.

acct_gather_infiniband/ofed

Infiniband network traffic data are collected from the hardware monitoring counters of Infiniband devices through the OFED library.

AcctGatherFilesystemType

Identifies the plugin to be used for filesystem traffic accounting. The plugin is activated only when profiling on hdf5 files is activated and the user asks for filesystem data collection for jobs through --profile=Lustre (or =All). The collection of filesystem traffic data takes place on node level, hence only in case of exclusive job allocation the collected values will reflect the jobs real traffic. All filesystem traffic data are logged on hdf5 files per job on each node. No storage on the Slurm database takes place.

Configurable values at present are:

acct_gather_filesystem/none

No filesystem data are collected.

acct_gather_filesystem/lustre

Lustre filesystem traffic data are collected from the counters found in /proc/fs/lustre/.

AcctGatherProfileType

Identifies the plugin to be used for detailed job profiling. The jobacct_gather plugin and slurmd daemon call this plugin to collect detailed data such as I/O counts, memory usage, or energy consumption for jobs and nodes. There are interfaces in this plugin to collect data as step start and completion, task start and completion, and at the account gather frequency. The data collected at the node level is related to jobs only in case of exclusive job allocation.

Configurable values at present are:

acct_gather_profile/none

No profile data is collected.

acct_gather_profile/hdf5

This enables the HDF5 plugin. The directory where the profile files are stored and which values are collected are configured in the acct_gather.conf file.

AllowSpecResourcesUsage

If set to 1, Slurm allows individual jobs to override node's configured CoreSpecCount value. For a job to take advantage of this feature, a command line option of --core-spec must be specified. The default value for this option is 1 for Cray systems and 0 for other system types.

AuthInfo

Additional information to be used for authentication of communications between the Slurm daemons (slurmctld and slurmd) and the Slurm clients. The interpretation of this option is specific to the configured AuthType. Multiple options may be specified in a comma delimited list. If not specified, the default authentication information will be used.

cred_expire

Default job step credential lifetime, in seconds (e.g. "cred_expire=1200"). It must be sufficiently long enough to load user environment, run prolog, deal with the slurmd getting paged out of memory, etc. This also controls how long a requeued job must wait before starting again. The default value is 120 seconds.

socket

Path name to a MUNGE daemon socket to use (e.g. "socket=/var/run/munge/munge.socket.2"). The default value is "/var/run/munge/munge.socket.2". Used by auth/munge and crypto/munge.

ttl

Credential lifetime, in seconds (e.g. "ttl=300"). The default value is dependent upon the Munge installation, but is typically 300 seconds.

AuthType

The authentication method for communications between Slurm components. Acceptable values at present include "auth/none" and "auth/munge". The default value is "auth/munge". "auth/none" includes the UID in each communication, but it is not verified. This may be fine for testing purposes, but do not use "auth/none" if you desire any security. "auth/munge" indicates that LLNL's MUNGE is to be used (this is the best supported authentication mechanism for Slurm, see "http://munge.googlecode.com/" for more information). All Slurm daemons and commands must be terminated prior to changing the value of AuthType and later restarted (Slurm jobs can be preserved).

BackupAddr

The name that BackupController should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. For example, "elx0000" might be used to designate the Ethernet address for node "lx0000". By default the BackupAddr will be identical in value to BackupController.

BackupController

The short, or long, name of the machine where Slurm control functions are to be executed in the event that ControlMachine fails (i.e. the name returned by the command "hostname -s"). This node may also be used as a compute server if so desired. It will come into service as a controller only upon the failure of ControlMachine and will revert to a "standby" mode when the ControlMachine becomes available once again.

The backup controller recovers state information from the StateSaveLocation directory, which must be readable and writable from both the primary and backup controllers. While not essential, it is recommended that you specify a backup controller. See the RELOCATING CONTROLLERS section if you change this.

BatchStartTimeout

The maximum time (in seconds) that a batch job is permitted for launching before being considered missing and releasing the allocation. The default value is 10 (seconds). Larger values may be required if more time is required to execute the Prolog, load user environment variables (for Moab spawned jobs), or if the slurmd daemon gets paged from memory.

Note: The test for a job being successfully launched is only performed when the Slurm daemon on the compute node registers state with the slurmctld daemon on the head node, which happens fairly rarely. Therefore a job will not necessarily be terminated if its start time exceeds BatchStartTimeout. This configuration parameter is also applied to launch tasks and avoid aborting srun commands due to long running Prolog scripts.

BurstBufferType

The plugin used to manage burst buffers. Acceptable values at present include "burst_buffer/none". More information later...

CheckpointType

The system-initiated checkpoint method to be used for user jobs. The slurmctld daemon must be restarted for a change in CheckpointType to take effect. Supported values presently include:

checkpoint/aix

for IBM AIX systems only

checkpoint/blcr

Berkeley Lab Checkpoint Restart (BLCR). NOTE: If a file is found at sbin/scch (relative to the Slurm installation location), it will be executed upon completion of the checkpoint. This can be a script used for managing the checkpoint files. NOTE: Slurm's BLCR logic only supports batch jobs.

checkpoint/none

no checkpoint support (default)

checkpoint/ompi

OpenMPI (version 1.3 or higher)

checkpoint/poe

for use with IBM POE (Parallel Operating Environment) only

ChosLoc

If configured, then any processes invoked on the user behalf (namely the SPANK prolog/epilog scripts and the slurmstepd processes, which in turn spawn the user batch script and applications) are not directly executed by the slurmd daemon, but instead the ChosLoc program is executed. Both are spawned with the same user ID as the configured SlurmdUser (typically user root). That program's argument are the program and arguments that would otherwise be invoked directly by the slurmd daemon. The intent of this feature is to be able to run a user application in some sort of container. This option specified the fully qualified pathname of the chos command (see https://github.com/scanon/chos for details).

ClusterName

The name by which this Slurm managed cluster is known in the accounting database. This is needed distinguish accounting records when multiple clusters report to the same database. Because of limitations in some databases, any upper case letters in the name will be silently mapped to lower case. In order to avoid confusion, it is recommended that the name be lower case.

CompleteWait

The time, in seconds, given for a job to remain in COMPLETING state before any additional jobs are scheduled. If set to zero, pending jobs will be started as soon as possible. Since a COMPLETING job's resources are released for use by other jobs as soon as the Epilog completes on each individual node, this can result in very fragmented resource allocations. To provide jobs with the minimum response time, a value of zero is recommended (no waiting). To minimize fragmentation of resources, a value equal to KillWait plus two is recommended. In that case, setting KillWait to a small value may be beneficial. The default value of CompleteWait is zero seconds. The value may not exceed 65533.

ControlAddr

Name that ControlMachine should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. For example, "elx0000" might be used to designate the Ethernet address for node "lx0000". By default the ControlAddr will be identical in value to ControlMachine.

ControlMachine

The short, or long, hostname of the machine where Slurm control functions are executed (i.e. the name returned by the command "hostname -s"). This value must be specified. In order to support some high availability architectures, multiple hostnames may be listed with comma separators and one ControlAddr must be specified. The high availability system must insure that the slurmctld daemon is running on only one of these hosts at a time. See the RELOCATING CONTROLLERS section if you change this.

CoreSpecPlugin

Identifies the plugins to be used for enforcement of core specialization. The slurmd daemon must be restarted for a change in CoreSpecPlugin to take effect. Acceptable values at present include:

core_spec/cray

used only for Cray systems

core_spec/none

used for all other system types

CpuFreqDef

Default CPU frequency governor to use when running a job step if it has not been explicitly set with the --cpu-freq option. Acceptable values at present include:

Conservative

attempts to use the Conservative CPU governor

OnDemand

attempts to use the OnDemand CPU governor

Performance

attempts to use the Performance CPU governor

PowerSave

attempts to use the PowerSave CPU governor

There is no default value. If unset, no attempt to set the governor is made if the --cpu-freq option has not been set.

CpuFreqGovernors

List of CPU frequency governors allowed to be set with the salloc, sbatch, or srun option --cpu-freq. Acceptable values at present include:

Conservative

attempts to use the Conservative CPU governor

OnDemand

attempts to use the OnDemand CPU governor (the default value)

Performance

attempts to use the Performance CPU governor (the default value)

PowerSave

attempts to use the PowerSave CPU governor

UserSpace

attempts to use the UserSpace CPU governor

The default is OnDemand, Performance.

CryptoType

The cryptographic signature tool to be used in the creation of job step credentials. The slurmctld daemon must be restarted for a change in CryptoType to take effect. Acceptable values at present include "crypto/munge" and "crypto/openssl". The default value is "crypto/munge".

DebugFlags

Defines specific subsystems which should provide more detailed event logging. Multiple subsystems can be specified with comma separators. Most DebugFlags will result in verbose logging for the identified subsystems and could impact performance. The below DB_* flags are only useful when writing directly to the database. If using the DBD put these debug flags in the slurmdbd.conf. Valid subsystems available today (with more to come) include:

Backfill

Backfill scheduler details

BackfillMap

Backfill scheduler to log a very verbose map of reserved resources through time. Combine with Backfill for a verbose and complete view of the backfill scheduler's work.

BGBlockAlgo

BlueGene block selection details

BGBlockAlgoDeep

BlueGene block selection, more details

BGBlockPick

BlueGene block selection for jobs

BGBlockWires

BlueGene block wiring (switch state details)

BurstBuffer

Burst Buffer plugin

CPU_Bind

CPU binding details for jobs and steps

CpuFrequency

Cpu frequency details for jobs and steps using the --cpu-freq option.

DB_ASSOC

SQL statements/queries when dealing with associations in the database.

DB_EVENT

SQL statements/queries when dealing with (node) events in the database.

DB_JOB

SQL statements/queries when dealing with jobs in the database.

DB_QOS

SQL statements/queries when dealing with QOS in the database.

DB_QUERY

SQL statements/queries when dealing with transactions and such in the database.

DB_RESERVATION

SQL statements/queries when dealing with reservations in the database.

DB_RESOURCE

SQL statements/queries when dealing with resources like licenses in the database.

DB_STEP

SQL statements/queries when dealing with steps in the database.

DB_USAGE

SQL statements/queries when dealing with usage queries and inserts in the database.

DB_WCKEY

SQL statements/queries when dealing with wckeys in the database.

Elasticsearch

Elasticsearch debug info

Energy

AcctGatherEnergy debug info

ExtSensors

External Sensors debug info

FrontEnd

Front end node details

Gres

Generic resource details

Gang

Gang scheduling details

JobContainer

Job container plugin details

License

License management details

NodeFeatures

Node Features plugin debug info

NO_CONF_HASH

Do not log when the slurm.conf files differs between Slurm daemons

Power

Power management plugin

Priority

Job prioritization

Protocol

Communication protocol details

Reservation

Advanced reservations

SelectType

Resource selection plugin

Steps

Slurmctld resource allocation for job steps

Switch

Switch plugin

TraceJobs

Trace jobs in slurmctld. It will print detailed job information including state, job ids and allocated nodes counter.

Triggers

Slurmctld triggers

Wiki

Sched/wiki and wiki2 communications

DefMemPerCPU

Default real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging. DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). The default value is 0 (unlimited). Also see DefMemPerNode and MaxMemPerCPU. DefMemPerCPU and DefMemPerNode are mutually exclusive.

NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

DefMemPerNode

Default real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging. DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). The default value is 0 (unlimited). Also see DefMemPerCPU and MaxMemPerNode. DefMemPerCPU and DefMemPerNode are mutually exclusive.

NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

DefaultStorageHost

The default name of the machine hosting the accounting storage and job completion databases. Only used for database type storage plugins and when the AccountingStorageHost and JobCompHost have not been defined.

DefaultStorageLoc

The fully qualified file name where accounting records and/or job completion records are written when the DefaultStorageType is "filetxt" or the name of the database where accounting records and/or job completion records are stored when the DefaultStorageType is a database. Also see AccountingStorageLoc and JobCompLoc.

DefaultStoragePass

The password used to gain access to the database to store the accounting and job completion data. Only used for database type storage plugins, ignored otherwise. Also see AccountingStoragePass and JobCompPass.

DefaultStoragePort

The listening port of the accounting storage and/or job completion database server. Only used for database type storage plugins, ignored otherwise. Also see AccountingStoragePort and JobCompPort.

DefaultStorageType

The accounting and job completion storage mechanism type. Acceptable values at present include "filetxt", "mysql" and "none". The value "filetxt" indicates that records will be written to a file. The value "mysql" indicates that accounting records will be written to a MySQL or MariaDB database. The default value is "none", which means that records are not maintained. Also see AccountingStorageType and JobCompType.

DefaultStorageUser

The user account for accessing the accounting storage and/or job completion database. Only used for database type storage plugins, ignored otherwise. Also see AccountingStorageUser and JobCompUser.

DisableRootJobs

If set to "YES" then user root will be prevented from running any jobs. The default value is "NO", meaning user root will be able to execute jobs. DisableRootJobs may also be set by partition.

EioTimeout

The number of seconds srun waits for slurmstepd to close the TCP/IP connection used to relay data between the user application and srun when the user application terminates. The default value is 60 seconds. May not exceed 65533.

EnforcePartLimits

If set to "ALL" then jobs which exceed a partition's size and/or time limits will be rejected at submission time. If job is submitted to multiple partitions, the job must satisfy the limits on all the requested paritions. If set to "NO" then the job will be accepted and remain queued until the partition limits are altered(Time and Node Limits). If set to "ANY" or "YES" a job must satisfy any of the requested partitions to be submitted. The default value is "NO". NOTE: If set, then a job's QOS can not be used to exceed partition limits.

Epilog

Fully qualified pathname of a script to execute as user root on every node when a user's job completes (e.g. "/usr/local/slurm/epilog"). A glob pattern (See glob (7)) may also be used to run more than one epilog script (e.g. "/etc/slurm/epilog.d/*"). The Epilog script or scripts may be used to purge files, disable user login, etc. By default there is no epilog. See Prolog and Epilog Scripts for more information.

EpilogMsgTime

The number of microseconds that the slurmctld daemon requires to process an epilog completion message from the slurmd daemons. This parameter can be used to prevent a burst of epilog completion messages from being sent at the same time which should help prevent lost messages and improve throughput for large jobs. The default value is 2000 microseconds. For a 1000 node job, this spreads the epilog completion messages out over two seconds.

EpilogSlurmctld

Fully qualified pathname of a program for the slurmctld to execute upon termination of a job allocation (e.g. "/usr/local/slurm/epilog_controller"). The program executes as SlurmUser, which gives it permission to drain nodes and requeue the job if a failure occurs (See scontrol(1)). Exactly what the program does and how it accomplishes this is completely at the discretion of the system administrator. Information about the job being initiated, it's allocated nodes, etc. are passed to the program using environment variables. See Prolog and Epilog Scripts for more information.

ExtSensorsFreq

The external sensors plugin sampling interval. If ExtSensorsType=ext_sensors/none, this parameter is ignored. For all other values of ExtSensorsType, this parameter is the number of seconds between external sensors samples for hardware components (nodes, switches, etc.) The default value is zero. This value disables external sensors sampling. Note: This parameter does not affect external sensors data collection for jobs/steps.

ExtSensorsType

Identifies the plugin to be used for external sensors data collection. Slurmctld calls this plugin to collect external sensors data for jobs/steps and hardware components. In case of node sharing between jobs the reported values per job/step (through sstat or sacct) may not be accurate. See also "man ext_sensors.conf".

Configurable values at present are:

ext_sensors/none

No external sensors data is collected.

ext_sensors/rrd

External sensors data is collected from the RRD database.

FairShareDampeningFactor

Dampen the effect of exceeding a user or group's fair share of allocated resources. Higher values will provides greater ability to differentiate between exceeding the fair share at high levels (e.g. a value of 1 results in almost no difference between overconsumption by a factor of 10 and 100, while a value of 5 will result in a significant difference in priority). The default value is 1.

FastSchedule

Controls how a node's configuration specifications in slurm.conf are used. If the number of node configuration entries in the configuration file is significantly lower than the number of nodes, setting FastSchedule to 1 will permit much faster scheduling decisions to be made. (The scheduler can just check the values in a few configuration records instead of possibly thousands of node records.) Note that on systems with hyper-threading, the processor count reported by the node will be twice the actual processor count. Consider which value you want to be used for scheduling purposes.

0

Base scheduling decisions upon the actual configuration of each individual node except that the node's processor count in Slurm's configuration must match the actual hardware configuration if PreemptMode=suspend,gang or SelectType=select/cons_res are configured (both of those plugins maintain resource allocation information using bitmaps for the cores in the system and must remain static, while the node's memory and disk space can be established later).

1 (default)

Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than the configured resources will be set to DRAIN.

2

Consider the configuration of each node to be that specified in the slurm.conf configuration file and any node with less than the configured resources will not be set DRAIN. This option is generally only useful for testing purposes.

FirstJobId

The job id to be used for the first submitted to Slurm without a specific requested value. Job id values generated will incremented by 1 for each subsequent job. This may be used to provide a meta-scheduler with a job id space which is disjoint from the interactive jobs. The default value is 1. Also see MaxJobId

GetEnvTimeout

Used for Moab scheduled jobs only. Controls how long job should wait in seconds for loading the user's environment before attempting to load it from a cache file. Applies when the srun or sbatch --get-user-env option is used. If set to 0 then always load the user's environment from the cache file. The default value is 2 seconds.

GresTypes

A comma delimited list of generic resources to be managed. These generic resources may have an associated plugin available to provide additional functionality. No generic resources are managed by default. Insure this parameter is consistent across all nodes in the cluster for proper operation. The slurmctld daemon must be restarted for changes to this parameter to become effective.

GroupUpdateForce

If set to a non-zero value, then information about which users are members of groups allowed to use a partition will be updated periodically, even when there have been no changes to the /etc/group file. Otherwise group member information will be updated periodically only after the /etc/group file is updated The default value is 1. Also see the GroupUpdateTime parameter.

GroupUpdateTime

Controls how frequently information about which users are members of groups allowed to use a partition will be updated, and how long user group membership lists will be cached. The time interval is given in seconds with a default value of 600 seconds and a maximum value of 4095 seconds. A value of zero will prevent periodic updating of group membership information. Also see the GroupUpdateForce parameter.

HealthCheckInterval

The interval in seconds between executions of HealthCheckProgram. The default value is zero, which disables execution.

HealthCheckNodeState

Identify what node states should execute the HealthCheckProgram. Multiple state values may be specified with a comma separator. The default value is ANY to execute on nodes in any state.

ALLOC

Run on nodes in the ALLOC state (all CPUs allocated).

ANY

Run on nodes in any state.

CYCLE

Rather than running the health check program on all nodes at the same time, cycle through running on all compute nodes through the course of the HealthCheckInterval. May be combined with the various node state options.

IDLE

Run on nodes in the IDLE state.

MIXED

Run on nodes in the MIXED state (some CPUs idle and other CPUs allocated).

HealthCheckProgram

Fully qualified pathname of a script to execute as user root periodically on all compute nodes that are not in the NOT_RESPONDING state. This program may be used to verify the node is fully operational and DRAIN the node or send email if a problem is detected. Any action to be taken must be explicitly performed by the program (e.g. execute "scontrol update NodeName=foo State=drain Reason=tmp_file_system_full" to drain a node). The execution interval is controlled using the HealthCheckInterval parameter. Note that the HealthCheckProgram will be executed at the same time on all nodes to minimize its impact upon parallel programs. This program is will be killed if it does not terminate normally within 60 seconds. By default, no program will be executed.

InactiveLimit

The interval, in seconds, after which a non-responsive job allocation command (e.g. srun or salloc) will result in the job being terminated. If the node on which the command is executed fails or the command abnormally terminates, this will terminate its job allocation. This option has no effect upon batch jobs. When setting a value, take into consideration that a debugger using srun to launch an application may leave the srun command in a stopped state for extended periods of time. This limit is ignored for jobs running in partitions with the RootOnly flag set (the scheduler running as root will be responsible for the job). The default value is unlimited (zero) and may not exceed 65533 seconds.

JobAcctGatherType

The job accounting mechanism type. Acceptable values at present include "jobacct_gather/aix" (for AIX operating system), "jobacct_gather/linux" (for Linux operating system), "jobacct_gather/cgroup" and "jobacct_gather/none" (no accounting data collected). The default value is "jobacct_gather/none". "jobacct_gather/cgroup" is a plugin for the Linux operating system that uses cgroups to collect accounting statistics. The plugin collects the following statistics: From the cgroup memory subsystem: memory.usage_in_bytes (reported as 'pages') and rss from memory.stat (reported as 'rss'). From the cgroup cpuacct subsystem: user cpu time and system cpu time. No value is provided by cgroups for virtual memory size ('vsize'). In order to use the sstat tool, "jobacct_gather/aix", "jobacct_gather/linux", or "jobacct_gather/cgroup" must be configured.
NOTE: Changing this configuration parameter changes the contents of the messages between Slurm daemons. Any previously running job steps are managed by a slurmstepd daemon that will persist through the lifetime of that job step and not change it's communication protocol. Only change this configuration parameter when there are no running job steps.

JobAcctGatherFrequency

The job accounting and profiling sampling intervals. The supported format is follows:

JobAcctGatherFrequency=<datatype>=<interval>

where <datatype>=<interval> specifies the task sampling interval for the jobacct_gather plugin or a sampling interval for a profiling type by the acct_gather_profile plugin. Multiple, comma-separated <datatype>=<interval> intervals may be specified. Supported datatypes are as follows:

task=<interval>

where <interval> is the task sampling interval in seconds for the jobacct_gather plugins and for task profiling by the acct_gather_profile plugin.

energy=<interval>

where <interval> is the sampling interval in seconds for energy profiling using the acct_gather_energy plugin

network=<interval>

where <interval> is the sampling interval in seconds for infiniband profiling using the acct_gather_infiniband plugin.

filesystem=<interval>

where <interval> is the sampling interval in seconds for filesystem profiling using the acct_gather_filesystem plugin.

e.g.
PriorityWeightTRES=CPU=1000,Mem=2000,GRES/gpu=3000

    SallocDefaultCommand = "$SHELL"

    SallocDefaultCommand = "srun -n1 -N1 --mem-per-cpu=0 --pty --preserve-env --mpi=none $SHELL"

    SallocDefaultCommand = "srun -n1 -N1 --mem-per-cpu=0 --gres=craynetwork:0 --pty --preserve-env --mpi=none $SHELL"

The configuration of nodes (or machines) to be managed by Slurm is also specified in /etc/slurm.conf. Changes in node configuration (e.g. adding nodes, changing their processor count, etc.) require restarting both the slurmctld daemon and the slurmd daemons. All slurmd daemons must know each node in the system to forward messages in support of hierarchical communications. Only the NodeName must be supplied in the configuration file. All other node configuration information is optional. It is advisable to establish baseline node configurations, especially if the cluster is heterogeneous. Nodes which register to the system with less than the configured resources (e.g. too little memory), will be placed in the "DOWN" state to avoid scheduling jobs on them. Establishing baseline configurations will also speed Slurm's scheduling process by permitting it to compare job requirements against these (relatively few) configuration parameters and possibly avoid having to check job requirements against every individual node's configuration. The resources checked at node registration time are: CPUs, RealMemory and TmpDisk. While baseline values for each of these can be established in the configuration file, the actual values upon node registration are recorded and these actual values may be used for scheduling purposes (depending upon the value of FastSchedule in the configuration file.

Default values can be specified with a record in which NodeName is "DEFAULT". The default entry values will apply only to lines following it in the configuration file and the default values can be reset multiple times in the configuration file with multiple entries where "NodeName=DEFAULT". Each line where NodeName is "DEFAULT" will replace or add to previous default values and not a reinitialize the default values. The "NodeName=" specification must be placed on every line describing the configuration of nodes. A single node name can not appear as a NodeName value in more than one line (duplicate node name records will be ignored). In fact, it is generally possible and desirable to define the configurations of all nodes in only a few lines. This convention permits significant optimization in the scheduling of larger clusters. In order to support the concept of jobs requiring consecutive nodes on some architectures, node specifications should be place in this file in consecutive order. No single node name may be listed more than once in the configuration file. Use "DownNodes=" to record the state of nodes which are temporarily in a DOWN, DRAIN or FAILING state without altering permanent configuration information. A job step's tasks are allocated to nodes in order the nodes appear in the configuration file. There is presently no capability within Slurm to arbitrarily order a job step's tasks.

Multiple node names may be comma separated (e.g. "alpha,beta,gamma") and/or a simple node range expression may optionally be used to specify numeric ranges of nodes to avoid building a configuration file with large numbers of entries. The node range expression can contain one pair of square brackets with a sequence of comma separated numbers and/or ranges of numbers separated by a "-" (e.g. "linux[0-64,128]", or "lx[15,18,32-33]"). Note that the numeric ranges can include one or more leading zeros to indicate the numeric portion has a fixed number of digits (e.g. "linux[0000-1023]"). Up to two numeric ranges can be included in the expression (e.g. "rack[0-63]_blade[0-41]"). If one or more numeric expressions are included, one of them must be at the end of the name (e.g. "unit[0-31]rack" is invalid), but arbitrary names can always be used in a comma separated list.

On BlueGene systems only, the square brackets should contain pairs of three digit numbers separated by a "x". These numbers indicate the boundaries of a rectangular prism (e.g. "bgl[000x144,400x544]"). See BlueGene documentation for more details. The node configuration specified the following information:

NodeName

Name that Slurm uses to refer to a node (or base partition for BlueGene systems). Typically this would be the string that "/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). Only short hostname forms are compatible with the switch/nrt plugin at this time. It may also be an arbitrary string if NodeHostname is specified. If the NodeName is "DEFAULT", the values specified with that record will apply to subsequent node specifications unless explicitly set to other values in that node record or replaced with a different set of default values. Each line where NodeName is "DEFAULT" will replace or add to previous default values and not a reinitialize the default values. For architectures in which the node order is significant, nodes will be considered consecutive in the order defined. For example, if the configuration for "NodeName=charlie" immediately follows the configuration for "NodeName=baker" they will be considered adjacent in the computer.

NodeHostname

Typically this would be the string that "/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). Only short hostname forms are compatible with the switch/nrt plugin at this time. A node range expression can be used to specify a set of nodes. If an expression is used, the number of nodes identified by NodeHostname on a line in the configuration file must be identical to the number of nodes identified by NodeName. By default, the NodeHostname will be identical in value to NodeName.

NodeAddr

Name that a node should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. If a node range expression is used to designate multiple nodes, they must exactly match the entries in the NodeName (e.g. "NodeName=lx[0-7] NodeAddr=elx[0-7]"). NodeAddr may also contain IP addresses. By default, the NodeAddr will be identical in value to NodeHostname.

Boards

Number of Baseboards in nodes with a baseboard controller. Note that when Boards is specified, SocketsPerBoard, CoresPerSocket, and ThreadsPerCore should be specified. Boards and CPUs are mutually exclusive. The default value is 1.

CoreSpecCount

Number of cores on which Slurm compute node daemons (slurmd, slurmstepd) will be confined. These cores will not be available for allocation to user jobs. Isolation of the Slurm daemons from user jobs may improve performance. If this option and CPUSpecList are both designated for a node, an error is generated. For information on the algorithm used by Slurm to select the cores refer to the core specialization documentation ( http://slurm.schedmd.com/core_spec.html ). This option has no effect unless cgroup job confinement is also configured (TaskPlugin=task/cgroup with ConstrainCores=yes in cgroup.conf).

CoresPerSocket

Number of cores in a single physical processor socket (e.g. "2"). The CoresPerSocket value describes physical cores, not the logical number of processors per socket. NOTE: If you have multi-core processors, you will likely need to specify this parameter in order to optimize scheduling. The default value is 1.

CPUs

Number of logical processors on the node (e.g. "2"). CPUs and Boards are mutually exclusive. It can be set to the total number of sockets, cores or threads. This can be useful when you want to schedule only the cores on a hyper-threaded node. If CPUs is omitted, it will be set equal to the product of Sockets, CoresPerSocket, and ThreadsPerCore. The default value is 1.

CPUSpecList

A comma delimited list of Slurm abstract CPU IDs on which Slurm compute node daemons (slurmd, slurmstepd) will be confined. The list will be expanded to include all other CPUs, if any, on the same cores. These cores will not be available for allocation to user jobs. Isolation of the Slurm daemons from user jobs may improve performance. If this option and CoreSpecCount are both designated for a node, an error is generated. This option has no effect unless cgroup job confinement is also configured (TaskPlugin=task/cgroup with ConstrainCores=yes in cgroup.conf).

Feature

A comma delimited list of arbitrary strings indicative of some characteristic associated with the node. There is no value associated with a feature at this time, a node either has a feature or it does not. If desired a feature may contain a numeric component indicating, for example, processor speed. By default a node has no features. Also see Gres.

Gres

A comma delimited list of generic resources specifications for a node. The format is: "<name>[:<type>][:no_consume]:<number>[K|M|G]". The first field is the resource name, which matches the GresType configuration parameter name. The optional type field might be used to identify a model of that generic resource. A generic resource can also be specified as non-consumable (i.e. multiple jobs can use the same generic resource) with the optional field ":no_consume". The final field must specify a generic resources count. A suffix of "K", "M", "G", "T" or "P" may be used to multiply the number by 1024, 1048576, 1073741824, etc. respectively. (e.g."Gres=gpu:tesla:1,gpu:kepler:1,bandwidth:lustre:no_consume:4G"). By default a node has no generic resources and its maximum count is that of an unsigned 64bit integer. Also see Feature.

MemSpecLimit

Limit on combined real memory allocation for compute node daemons (slurmd, slurmstepd), in megabytes. This memory is not available to job allocations. The daemons won't be killed when they exhaust the memory allocation (ie. the OOM Killer is disabled for the daemon's memory cgroup). This option has no effect unless cgroup job confinement is also configured (TaskPlugin=task/cgroup with ConstrainRAMSpace=yes in cgroup.conf).

Port

The port number that the Slurm compute node daemon, slurmd, listens to for work on this particular node. By default there is a single port number for all slurmd daemons on all compute nodes as defined by the SlurmdPort configuration parameter. Use of this option is not generally recommended except for development or testing purposes. If multiple slurmd daemons execute on a node this can specify a range of ports.

Note: On Cray systems, Realm-Specific IP Addressing (RSIP) will automatically try to interact with anything opened on ports 8192-60000. Configure Port to use a port outside of the configured SrunPortRange and RSIP's port range.

Procs

See CPUs.

RealMemory

Size of real memory on the node in MegaBytes (e.g. "2048"). The default value is 1.

Reason

Identifies the reason for a node being in state "DOWN", "DRAINED" "DRAINING", "FAIL" or "FAILING". Use quotes to enclose a reason having more than one word.

Sockets

Number of physical processor sockets/chips on the node (e.g. "2"). If Sockets is omitted, it will be inferred from CPUs, CoresPerSocket, and ThreadsPerCore. NOTE: If you have multi-core processors, you will likely need to specify these parameters. Sockets and SocketsPerBoard are mutually exclusive. If Sockets is specified when Boards is also used, Sockets is interpreted as SocketsPerBoard rather than total sockets. The default value is 1.

SocketsPerBoard

Number of physical processor sockets/chips on a baseboard. Sockets and SocketsPerBoard are mutually exclusive. The default value is 1.

State

State of the node with respect to the initiation of user jobs. Acceptable values are "CLOUD", "DOWN", "DRAIN", "FAIL", "FAILING", "FUTURE" and "UNKNOWN". Node states of "BUSY" and "IDLE" should not be specified in the node configuration, but set the node state to "UNKNOWN" instead. Setting the node state to "UNKNOWN" will result in the node state being set to "BUSY", "IDLE" or other appropriate state based upon recovered system state information. The default value is "UNKNOWN". Also see the DownNodes parameter below.

CLOUD

Indicates the node exists in the cloud. It's initial state will be treated as powered down. The node will be available for use after it's state is recovered from Slurm's state save file or the slurmd daemon starts on the compute node.

DOWN

Indicates the node failed and is unavailable to be allocated work.

DRAIN

Indicates the node is unavailable to be allocated work.on.

FAIL

Indicates the node is expected to fail soon, has no jobs allocated to it, and will not be allocated to any new jobs.

FAILING

Indicates the node is expected to fail soon, has one or more jobs allocated to it, but will not be allocated to any new jobs.

FUTURE

Indicates the node is defined for future use and need not exist when the Slurm daemons are started. These nodes can be made available for use simply by updating the node state using the scontrol command rather than restarting the slurmctld daemon. After these nodes are made available, change their State in the slurm.conf file. Until these nodes are made available, they will not be seen using any Slurm commands or nor will any attempt be made to contact them.

UNKNOWN

Indicates the node's state is undefined (BUSY or IDLE), but will be established when the slurmd daemon on that node registers. The default value is "UNKNOWN".

ThreadsPerCore

Number of logical threads in a single physical core (e.g. "2"). Note that the Slurm can allocate resources to jobs down to the resolution of a core. If your system is configured with more than one thread per core, execution of a different job on each thread is not supported unless you configure SelectTypeParameters=CR_CPU plus CPUs; do not configure Sockets, CoresPerSocket or ThreadsPerCore. A job can execute a one task per thread from within one job step or execute a distinct job step on each of the threads. Note also if you are running with more than 1 thread per core and running the select/cons_res plugin you will want to set the SelectTypeParameters variable to something other than CR_CPU to avoid unexpected results. The default value is 1.

TmpDisk

Total size of temporary disk storage in TmpFS in MegaBytes (e.g. "16384"). TmpFS (for "Temporary File System") identifies the location which jobs should use for temporary storage. Note this does not indicate the amount of free space available to the user on the node, only the total file system size. The system administration should insure this file system is purged as needed so that user jobs have access to most of this space. The Prolog and/or Epilog programs (specified in the configuration file) might be used to insure the file system is kept clean. The default value is 0.

Weight

The priority of the node for scheduling purposes. All things being equal, jobs will be allocated the nodes with the lowest weight which satisfies their requirements. For example, a heterogeneous collection of nodes might be placed into a single partition for greater system utilization, responsiveness and capability. It would be preferable to allocate smaller memory nodes rather than larger memory nodes if either will satisfy a job's requirements. The units of weight are arbitrary, but larger weights should be assigned to nodes with more processors, memory, disk space, higher processor speed, etc. Note that if a job allocation request can not be satisfied using the nodes with the lowest weight, the set of nodes with the next lowest weight is added to the set of nodes under consideration for use (repeat as needed for higher weight values). If you absolutely want to minimize the number of higher weight nodes allocated to a job (at a cost of higher scheduling overhead), give each node a distinct Weight value and they will be added to the pool of nodes being considered for scheduling individually. The default value is 1.

The "DownNodes=" configuration permits you to mark certain nodes as in a DOWN, DRAIN, FAIL, or FAILING state without altering the permanent configuration information listed under a "NodeName=" specification.

DownNodes

Any node name, or list of node names, from the "NodeName=" specifications.

Reason

Identifies the reason for a node being in state "DOWN", "DRAIN", "FAIL" or "FAILING. Use quotes to enclose a reason having more than one word.

State

State of the node with respect to the initiation of user jobs. Acceptable values are "DOWN", "DRAIN", "FAIL", "FAILING" and "UNKNOWN". Node states of "BUSY" and "IDLE" should not be specified in the node configuration, but set the node state to "UNKNOWN" instead. Setting the node state to "UNKNOWN" will result in the node state being set to "BUSY", "IDLE" or other appropriate state based upon recovered system state information. The default value is "UNKNOWN".

DOWN

Indicates the node failed and is unavailable to be allocated work.

DRAIN

Indicates the node is unavailable to be allocated work.on.

FAIL

Indicates the node is expected to fail soon, has no jobs allocated to it, and will not be allocated to any new jobs.

FAILING

Indicates the node is expected to fail soon, has one or more jobs allocated to it, but will not be allocated to any new jobs.

UNKNOWN

Indicates the node's state is undefined (BUSY or IDLE), but will be established when the slurmd daemon on that node registers. The default value is "UNKNOWN".

On computers where frontend nodes are used to execute batch scripts rather than compute nodes (BlueGene or Cray systems), one may configure one or more frontend nodes using the configuration parameters defined below. These options are very similar to those used in configuring compute nodes. These options may only be used on systems configured and built with the appropriate parameters (--have-front-end, --enable-bluegene-emulation) or a system determined to have the appropriate architecture by the configure script (BlueGene or Cray systems). The front end configuration specifies the following information:

AllowGroups

Comma separated list of group names which may execute jobs on this front end node. By default, all groups may use this front end node. If at least one group associated with the user attempting to execute the job is in AllowGroups, he will be permitted to use this front end node. May not be used with the DenyGroups option.

AllowUsers

Comma separated list of user names which may execute jobs on this front end node. By default, all users may use this front end node. May not be used with the DenyUsers option.

DenyGroups

Comma separated list of group names which are prevented from executing jobs on this front end node. May not be used with the AllowGroups option.

DenyUsers

Comma separated list of user names which are prevented from executing jobs on this front end node. May not be used with the AllowUsers option.

FrontendName

Name that Slurm uses to refer to a frontend node. Typically this would be the string that "/bin/hostname -s" returns. It may also be the fully qualified domain name as returned by "/bin/hostname -f" (e.g. "foo1.bar.com"), or any valid domain name associated with the host through the host database (/etc/hosts) or DNS, depending on the resolver settings. Note that if the short form of the hostname is not used, it may prevent use of hostlist expressions (the numeric portion in brackets must be at the end of the string). If the FrontendName is "DEFAULT", the values specified with that record will apply to subsequent node specifications unless explicitly set to other values in that frontend node record or replaced with a different set of default values. Each line where FrontendName is "DEFAULT" will replace or add to previous default values and not a reinitialize the default values. Note that since the naming of front end nodes would typically not follow that of the compute nodes (e.g. lacking X, Y and Z coordinates found in the compute node naming scheme), each front end node name should be listed separately and without a hostlist expression (i.e. frontend00,frontend01" rather than "frontend[00-01]").</p>

FrontendAddr

Name that a frontend node should be referred to in establishing a communications path. This name will be used as an argument to the gethostbyname() function for identification. As with FrontendName, list the individual node addresses rather than using a hostlist expression. The number of FrontendAddr records per line must equal the number of FrontendName records per line (i.e. you can't map to node names to one address). FrontendAddr may also contain IP addresses. By default, the FrontendAddr will be identical in value to FrontendName.

Port

The port number that the Slurm compute node daemon, slurmd, listens to for work on this particular frontend node. By default there is a single port number for all slurmd daemons on all frontend nodes as defined by the SlurmdPort configuration parameter. Use of this option is not generally recommended except for development or testing purposes.

Note: On Cray systems, Realm-Specific IP Addressing (RSIP) will automatically try to interact with anything opened on ports 8192-60000. Configure Port to use a port outside of the configured SrunPortRange and RSIP's port range.

Reason

Identifies the reason for a frontend node being in state "DOWN", "DRAINED" "DRAINING", "FAIL" or "FAILING". Use quotes to enclose a reason having more than one word.

State

State of the frontend node with respect to the initiation of user jobs. Acceptable values are "DOWN", "DRAIN", "FAIL", "FAILING" and "UNKNOWN". "DOWN" indicates the frontend node has failed and is unavailable to be allocated work. "DRAIN" indicates the frontend node is unavailable to be allocated work. "FAIL" indicates the frontend node is expected to fail soon, has no jobs allocated to it, and will not be allocated to any new jobs. "FAILING" indicates the frontend node is expected to fail soon, has one or more jobs allocated to it, but will not be allocated to any new jobs. "UNKNOWN" indicates the frontend node's state is undefined (BUSY or IDLE), but will be established when the slurmd daemon on that node registers. The default value is "UNKNOWN". Also see the DownNodes parameter below.

For example: "FrontendName=frontend[00-03] FrontendAddr=efrontend[00-03] State=UNKNOWN" is used to define four front end nodes for running slurmd daemons.

The partition configuration permits you to establish different job limits or access controls for various groups (or partitions) of nodes. Nodes may be in more than one partition, making partitions serve as general purpose queues. For example one may put the same set of nodes into two different partitions, each with different constraints (time limit, job sizes, groups allowed to use the partition, etc.). Jobs are allocated resources within a single partition. Default values can be specified with a record in which PartitionName is "DEFAULT". The default entry values will apply only to lines following it in the configuration file and the default values can be reset multiple times in the configuration file with multiple entries where "PartitionName=DEFAULT". The "PartitionName=" specification must be placed on every line describing the configuration of partitions. Each line where PartitionName is "DEFAULT" will replace or add to previous default values and not a reinitialize the default values. A single partition name can not appear as a PartitionName value in more than one line (duplicate partition name records will be ignored). If a partition that is in use is deleted from the configuration and slurm is restarted or reconfigured (scontrol reconfigure), jobs using the partition are canceled. NOTE: Put all parameters for each partition on a single line. Each line of partition configuration information should represent a different partition. The partition configuration file contains the following information:

AllocNodes

Comma separated list of nodes from which users can submit jobs in the partition. Node names may be specified using the node range expression syntax described above. The default value is "ALL".

AllowAccounts

Comma separated list of accounts which may execute jobs in the partition. The default value is "ALL". NOTE: If AllowAccounts is used then DenyAccounts will not be enforced. Also refer to DenyAccounts.

AllowGroups

Comma separated list of group names which may execute jobs in the partition. If at least one group associated with the user attempting to execute the job is in AllowGroups, he will be permitted to use this partition. Jobs executed as user root can use any partition without regard to the value of AllowGroups. If user root attempts to execute a job as another user (e.g. using srun's --uid option), this other user must be in one of groups identified by AllowGroups for the job to successfully execute. The default value is "ALL". NOTE: For performance reasons, Slurm maintains a list of user IDs allowed to use each partition and this is checked at job submission time. This list of user IDs is updated when the slurmctld daemon is restarted, reconfigured (e.g. "scontrol reconfig") or the partition's AllowGroups value is reset, even if is value is unchanged (e.g. "scontrol update PartitionName=name AllowGroups=group"). For a user's access to a partition to change, both his group membership must change and Slurm's internal user ID list must change using one of the methods described above.

AllowQos

Comma separated list of Qos which may execute jobs in the partition. Jobs executed as user root can use any partition without regard to the value of AllowQos. The default value is "ALL". NOTE: If AllowQos is used then DenyQos will not be enforced. Also refer to DenyQos.

Alternate

Partition name of alternate partition to be used if the state of this partition is "DRAIN" or "INACTIVE."

Default

If this keyword is set, jobs submitted without a partition specification will utilize this partition. Possible values are "YES" and "NO". The default value is "NO".

DefMemPerCPU

Default real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging. DefMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). If not set, the DefMemPerCPU value for the entire cluster will be used. Also see DefMemPerNode and MaxMemPerCPU. DefMemPerCPU and DefMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

DefMemPerNode

Default real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging. DefMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). If not set, the DefMemPerNode value for the entire cluster will be used. Also see DefMemPerCPU and MaxMemPerNode. DefMemPerCPU and DefMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

DenyAccounts

Comma separated list of accounts which may not execute jobs in the partition. By default, no accounts are denied access NOTE: If AllowAccounts is used then DenyAccounts will not be enforced. Also refer to AllowAccounts.

DenyQos

Comma separated list of Qos which may not execute jobs in the partition. By default, no QOS are denied access NOTE: If AllowQos is used then DenyQos will not be enforced. Also refer AllowQos.

DefaultTime

Run time limit used for jobs that don't specify a value. If not set then MaxTime will be used. Format is the same as for MaxTime.

DisableRootJobs

If set to "YES" then user root will be prevented from running any jobs on this partition. The default value will be the value of DisableRootJobs set outside of a partition specification (which is "NO", allowing user root to execute jobs).

ExclusiveUser

If set to "YES" then nodes will be exclusively allocated to users. Multiple jobs may be run for the same user, but only one user can be active at a time. This capability is also available on a per-job basis by using the --exclusive=user option.

GraceTime

Specifies, in units of seconds, the preemption grace time to be extended to a job which has been selected for preemption. The default value is zero, no preemption grace time is allowed on this partition. Once a job has been selected for preemption, it's end time is set to the current time plus GraceTime. The job is immediately sent SIGCONT and SIGTERM signals in order to provide notification of its imminent termination. This is followed by the SIGCONT, SIGTERM and SIGKILL signal sequence upon reaching its new end time. (Meaningful only for PreemptMode=CANCEL)

Hidden

Specifies if the partition and its jobs are to be hidden by default. Hidden partitions will by default not be reported by the Slurm APIs or commands. Possible values are "YES" and "NO". The default value is "NO". Note that partitions that a user lacks access to by virtue of the AllowGroups parameter will also be hidden by default.

LLN

Schedule resources to jobs on the least loaded nodes (based upon the number of idle CPUs). This is generally only recommended for an environment with serial jobs as idle resources will tend to be highly fragmented, resulting in parallel jobs being distributed across many nodes. Also see the SelectParameters configuration parameter CR_LLN to use the least loaded nodes in every partition.

MaxCPUsPerNode

Maximum number of CPUs on any node available to all jobs from this partition. This can be especially useful to schedule GPUs. For example a node can be associated with two Slurm partitions (e.g. "cpu" and "gpu") and the partition/queue "cpu" could be limited to only a subset of the node's CPUs, insuring that one or more CPUs would be available to jobs in the "gpu" partition/queue.

MaxMemPerCPU

Maximum real memory size available per allocated CPU in MegaBytes. Used to avoid over-subscribing memory and causing paging. MaxMemPerCPU would generally be used if individual processors are allocated to jobs (SelectType=select/cons_res). If not set, the MaxMemPerCPU value for the entire cluster will be used. Also see DefMemPerCPU and MaxMemPerNode. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

MaxMemPerNode

Maximum real memory size available per allocated node in MegaBytes. Used to avoid over-subscribing memory and causing paging. MaxMemPerNode would generally be used if whole nodes are allocated to jobs (SelectType=select/linear) and resources are over-subscribed (OverSubscribe=yes or OverSubscribe=force). If not set, the MaxMemPerNode value for the entire cluster will be used. Also see DefMemPerNode and MaxMemPerCPU. MaxMemPerCPU and MaxMemPerNode are mutually exclusive. NOTE: Enforcement of memory limits currently requires enabling of accounting, which samples memory use on a periodic basis (data need not be stored, just collected).

MaxNodes

Maximum count of nodes which may be allocated to any single job. For BlueGene systems this will be a c-nodes count and will be converted to a midplane count with a reduction in resolution. The default value is "UNLIMITED", which is represented internally as -1. This limit does not apply to jobs executed by SlurmUser or user root.

MaxTime

Maximum run time limit for jobs. Format is minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, days-hours:minutes:seconds or "UNLIMITED". Time resolution is one minute and second values are rounded up to the next minute. This limit does not apply to jobs executed by SlurmUser or user root.

MinNodes

Minimum count of nodes which may be allocated to any single job. For BlueGene systems this will be a c-nodes count and will be converted to a midplane count with a reduction in resolution. The default value is 1. This limit does not apply to jobs executed by SlurmUser or user root.

Nodes

Comma separated list of nodes (or base partitions for BlueGene systems) which are associated with this partition. Node names may be specified using the node range expression syntax described above. A blank list of nodes (i.e. "Nodes= ") can be used if one wants a partition to exist, but have no resources (possibly on a temporary basis). A value of "ALL" is mapped to all nodes configured in the cluster.

OverSubscribe

Controls the ability of the partition to execute more than one job at a time on each resource (node, socket or core depending upon the value of SelectTypeParameters). If resources are to be over-subscribed, avoiding memory over-subscription is very important. SelectTypeParameters should be configured to treat memory as a consumable resource and the --mem option should be used for job allocations. Sharing of resources is typically useful only when using gang scheduling (PreemptMode=suspend,gang). Possible values for OverSubscribe are "EXCLUSIVE", "FORCE", "YES", and "NO". Note that a value of "YES" or "FORCE" can negatively impact performance for systems with many thousands of running jobs. The default value is "NO". For more information see the following web pages:
http://slurm.schedmd.com/cons_res.html,
http://slurm.schedmd.com/cons_res_share.html,
http://slurm.schedmd.com/gang_scheduling.html, and
http://slurm.schedmd.com/preempt.html.

EXCLUSIVE

Allocates entire nodes to jobs even with select/cons_res configured. Jobs that run in partitions with "OverSubscribe=EXCLUSIVE" will have exclusive access to all allocated nodes.

FORCE

Makes all resources in the partition available for sharing without any means for users to disable it. May be followed with a colon and maximum number of jobs in running or suspended state. For example "OverSubscribe=FORCE:4" enables each node, socket or core to execute up to four jobs at once. Recommended only for BlueGene systems configured with small blocks or for systems running with gang scheduling (PreemptMode=suspend,gang). NOTE: PreemptType=QOS will permit one additional job to be run on the partition if started due to job preemption. For example, a configuration of OverSubscribe=FORCE:1 will only permit one job per resources normally, but a second job can be started if done so through preemption based upon QOS. The use of PreemptType=QOS and PreemptType=Suspend only applies with SelectType=cons_res.

YES

Makes all resources in the partition available for sharing upon request by the job. Resources will only be over-subscribed when explicitly requested by the user using the "--share" option on job submission. May be followed with a colon and maximum number of jobs in running or suspended state. For example "OverSubscribe=YES:4" enables each node, socket or core to execute up to four jobs at once. Recommended only for systems running with gang scheduling (PreemptMode=suspend,gang).

NO

Selected resources are allocated to a single job. No resource will be allocated to more than one job.

PartitionName

Name by which the partition may be referenced (e.g. "Interactive"). This name can be specified by users when submitting jobs. If the PartitionName is "DEFAULT", the values specified with that record will apply to subsequent partition specifications unless explicitly set to other values in that partition record or replaced with a different set of default values. Each line where PartitionName is "DEFAULT" will replace or add to previous default values and not a reinitialize the default values.

PreemptMode

Mechanism used to preempt jobs from this partition when PreemptType=preempt/partition_prio is configured. This partition specific PreemptMode configuration parameter will override the PreemptMode configuration parameter set for the cluster as a whole. The cluster-level PreemptMode must include the GANG option if PreemptMode is configured to SUSPEND for any partition. The cluster-level PreemptMode must not be OFF if PreemptMode is enabled for any partition. See the description of the cluster-level PreemptMode configuration parameter above for further information.

PriorityJobFactor

Partition factor used by priority/multifactor plugin in calculating job priority. The value may not exceed 65533. Also see PriorityTier.

PriorityTier

Jobs submitted to a partition with a higher priority tier value will be dispatched before pending jobs in partition with lower priority tier value and, if possible, they will preempt running jobs from partitions with lower priority tier values. Note that a partition's priority tier takes precedence over a job's priority. The value may not exceed 65533. Also see PriorityJobFactor.

QOS

Used to extend the limits available to a QOS on a partition. Jobs will not be associated to this QOS outside of being associated to the partition. They will still be associated to their requested QOS. By default, no QOS is used. NOTE: If a limit is set in both the Partition's QOS and the Job's QOS the Partition QOS will be honored unless the Job's QOS has the OverPartQOS flag set in which the Job's QOS will have priority.

ReqResv

Specifies users of this partition are required to designate a reservation when submitting a job. This option can be useful in restricting usage of a partition that may have higher priority or additional resources to be allowed only within a reservation. Possible values are "YES" and "NO". The default value is "NO".

RootOnly

Specifies if only user ID zero (i.e. user root) may allocate resources in this partition. User root may allocate resources for any other user, but the request must be initiated by user root. This option can be useful for a partition to be managed by some external entity (e.g. a higher-level job manager) and prevents users from directly using those resources. Possible values are "YES" and "NO". The default value is "NO".

SelectTypeParameters

Partition-specific resource allocation type. This option replaces the global SelectTypeParameters value. Supported values are CR_Core, CR_Core_Memory, CR_Socket and CR_Socket_Memory. Use requires the system-wide SelectTypeParameters value be set.

Shared

The Shared configuration parameter has been replaced by the OverSubscribe parameter described above.

State

State of partition or availability for use. Possible values are "UP", "DOWN", "DRAIN" and "INACTIVE". The default value is "UP". See also the related "Alternate" keyword.

UP

Designates that new jobs may queued on the partition, and that jobs may be allocated nodes and run from the partition.

DOWN

Designates that new jobs may be queued on the partition, but queued jobs may not be allocated nodes and run from the partition. Jobs already running on the partition continue to run. The jobs must be explicitly canceled to force their termination.

DRAIN

Designates that no new jobs may be queued on the partition (job submission requests will be denied with an error message), but jobs already queued on the partition may be allocated nodes and run. See also the "Alternate" partition specification.

INACTIVE

Designates that no new jobs may be queued on the partition, and jobs already queued may not be allocated nodes and run. See also the "Alternate" partition specification.

TRESBillingWeights

TRESBillingWeights is used to define the billing weights of each TRES type that will be used in calcuating the usage of a job.

Billing weights are specified as a comma-separated list of <TRES Type>=<TRES Billing Weight> pairs.

Any TRES Type is available for billing. Note that base the unit for memory and burst buffers is megabytes.

By default the billing of TRES is calculated as the sum of all TRES types multiplied by their corresponding billing weight.

The weighted amount of a resource can be adjusted by adding a suffix of K,M,G,T or P after the billing weight. For example, a memory weight of "mem=.25" on a job allocated 8GB will be billed 2048 (8192MB *.25) units. A memory weight of "mem=.25G" on the same job will be billed 2 (8192MB * (.25/1024)) units.

When a job is allocated 1 CPU and 8 GB of memory on a partition configured with TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0", the billable TRES will be: (1*1.0) + (8*0.25) + (0*2.0) = 3.0.

If PriorityFlags=MAX_TRES is configured, the billable TRES is calculated as the MAX of individual TRES' on a node (e.g. cpus, mem, gres) plus the sum of all global TRES' (e.g. licenses). Using the same example above the billable TRES will be MAX(1*1.0, 8*0.25) + (0*2.0) = 2.0.

If TRESBillingWeights is not defined then the job is billed against the total number of allocated CPUs.

NOTE: TRESBillingWeights is only used when calcuating fairshare and doesn't affect job priority directly as it is currently not used for the size of the job. If you want TRES' to play a role in the job's priority then refer to the PriorityWeightTRES option.

Prolog and Epilog Scripts

NOTE: Standard output and error messages are normally not preserved. Explicitly write output and error messages to an appropriate location if you wish to preserve that information.

NOTE: By default the Prolog script is ONLY run on any individual node when it first sees a job step from a new allocation; it does not run the Prolog immediately when an allocation is granted. If no job steps from an allocation are run on a node, it will never run the Prolog for that allocation. This Prolog behaviour can be changed by the PrologFlags parameter. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released.

If the Epilog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state. If the EpilogSlurmctld fails (returns a non-zero exit code), this will only be logged. If the Prolog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state and the job being requeued in a held state unless nohold_on_prolog_fail is configured in SchedulerParameters. If the PrologSlurmctld fails (returns a non-zero exit code), this will result in the job requeued to executed on another node if possible. Only batch jobs can be requeued.
 Interactive jobs (salloc and srun) will be cancelled if the PrologSlurmctld fails.

Information about the job is passed to the script using environment variables. Unless otherwise specified, these environment variables are available to all of the programs.

BASIL_RESERVATION_ID

Basil reservation ID. Available on Cray systems with ALPS only.

MPIRUN_PARTITION

BlueGene partition name. Available on BlueGene systems only.

SLURM_ARRAY_JOB_ID

If this job is part of a job array, this will be set to the job ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ..."); Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_ARRAY_TASK_ID

If this job is part of a job array, this will be set to the task ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. "scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ..."); Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_ARRAY_TASK_MAX

If this job is part of a job array, this will be set to the maximum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_ARRAY_TASK_MIN

If this job is part of a job array, this will be set to the minimum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_ARRAY_TASK_STEP

If this job is part of a job array, this will be set to the step size of task IDs. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_CLUSTER_NAME

Name of the cluster executing the job.

SLURM_JOB_ACCOUNT

Account name used for the job. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_CONSTRAINTS

Features required to run the job. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_DERIVED_EC

The highest exit code of all of the job steps. Available in EpilogSlurmctld only.

SLURM_JOB_EXIT_CODE

The exit code of the job script (or salloc). The value is the status as returned by the wait() system call (See wait(2)) Available in EpilogSlurmctld only.

SLURM_JOB_EXIT_CODE2

The exit code of the job script (or salloc). The value has the format <exit>:<sig>. The first number is the exit code, typically as set by the exit() function. The second number of the signal that caused the process to terminante if it was terminated by a signal. Available in EpilogSlurmctld only.

SLURM_JOB_GID

Group ID of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_GPUS

GPU IDs allocated to the job (if any). Available in the Prolog only.

SLURM_JOB_GROUP

Group name of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_ID

Job ID. CAUTION: If this job is the first task of a job array, then Slurm commands using this job ID will refer to the entire job array rather than this specific task of the job array.

SLURM_JOB_NAME

Name of the job. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_NODELIST

Nodes assigned to job. A Slurm hostlist expression. "scontrol show hostnames" can be used to convert this to a list of individual host names. Available in PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_PARTITION

Partition that job runs in. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.

SLURM_JOB_UID

User ID of the job's owner.

SLURM_JOB_USER

User name of the job's owner.

NETWORK TOPOLOGY

Given network topology information, Slurm allocates all of a job's resources onto a single leaf of the network (if possible) using a best-fit algorithm. Otherwise it will allocate a job's resources onto multiple leaf switches so as to minimize the use of higher-level switches. The TopologyPlugin parameter controls which plugin is used to collect network topology information. The only values presently supported are "topology/3d_torus" (default for IBM BlueGene and Cray XT/XE systems, performs best-fit logic over three-dimensional topology), "topology/none" (default for other systems, best-fit logic over one-dimensional topology), "topology/tree" (determine the network topology based upon information contained in a topology.conf file, see "man topology.conf" for more information). Future plugins may gather topology information directly from the network. The topology information is optional. If not provided, Slurm will perform a best-fit algorithm assuming the nodes are in a one-dimensional array as configured and the communications cost is related to the node distance in this array.

RELOCATING CONTROLLERS

1. Stop the Slurm daemons
2. Modify the slurm.conf file appropriately
3. Distribute the updated slurm.conf file to all nodes
4. Restart the Slurm daemons

There should be no loss of any running or pending jobs. Insure that any nodes added to the cluster have the current slurm.conf file installed.

CAUTION: If two nodes are simultaneously configured as the primary controller (two nodes on which ControlMachine specify the local host and the slurmctld daemon is executing on each), system behavior will be destructive. If a compute node has an incorrect ControlMachine or BackupController parameter, that node may be rendered unusable, but no other harm will result.

EXAMPLE

#
# Sample /etc/slurm.conf for dev[0-25].llnl.gov
# Author: John Doe
# Date: 11/06/2001
#
ControlMachine=dev0
ControlAddr=edev0
BackupController=dev1
BackupAddr=edev1
#
AuthType=auth/munge
Epilog=/usr/local/slurm/epilog
Prolog=/usr/local/slurm/prolog
FastSchedule=1
FirstJobId=65536
InactiveLimit=120
JobCompType=jobcomp/filetxt
JobCompLoc=/var/log/slurm/jobcomp
KillWait=30
MaxJobCount=10000
MinJobAge=3600
PluginDir=/usr/local/lib:/usr/local/slurm/lib
ReturnToService=0
SchedulerType=sched/backfill
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmctldPort=7002
SlurmdPort=7003
SlurmdSpoolDir=/usr/local/slurm/slurmd.spool
StateSaveLocation=/usr/local/slurm/slurm.state
SwitchType=switch/none
TmpFS=/tmp
WaitTime=30
JobCredentialPrivateKey=/usr/local/slurm/private.key
JobCredentialPublicCertificate=/usr/local/slurm/public.cert
#
# Node Configurations
#
NodeName=DEFAULT CPUs=2 RealMemory=2000 TmpDisk=64000
NodeName=DEFAULT State=UNKNOWN
NodeName=dev[0-25] NodeAddr=edev[0-25] Weight=16
# Update records for specific DOWN nodes
DownNodes=dev20 State=DOWN Reason="power,ETA=Dec25"
#
# Partition Configurations
#
PartitionName=DEFAULT MaxTime=30 MaxNodes=10 State=UP
PartitionName=debug Nodes=dev[0-8,18-25] Default=YES
PartitionName=batch Nodes=dev[9-17] MinNodes=4
PartitionName=long Nodes=dev[9-17] MaxTime=120 AllowGroups=admin

INCLUDE MODIFIERS

%c

Cluster name specified in the slurm.conf will be used.

EXAMPLE

ClusterName=linux include /home/slurm/etc/%c_config # Above line interpreted as # "include /home/slurm/etc/linux_config"

FILE AND DIRECTORY PERMISSIONS

AccountingStorageLoc

If this specifies a file, it must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines. It is recommended that the file be readable by all users from login and compute nodes.

Epilog

Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute node.

EpilogSlurmctld

Must be executable by user SlurmUser. It is recommended that the file be readable by all users. The file must be accessible by the primary and backup control machines.

HealthCheckProgram

Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute node.

JobCheckpointDir

Must be writable by user SlurmUser and no other users. The file must be accessible by the primary and backup control machines.

JobCompLoc

If this specifies a file, it must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.

JobCredentialPrivateKey

Must be readable only by user SlurmUser and writable by no other users. The file must be accessible by the primary and backup control machines.

JobCredentialPublicCertificate

Readable to all users on all nodes. Must not be writable by regular users.

MailProg

Must be executable by user SlurmUser. Must not be writable by regular users. The file must be accessible by the primary and backup control machines.

Prolog

Must be executable by user root. It is recommended that the file be readable by all users. The file must exist on every compute node.

PrologSlurmctld

Must be executable by user SlurmUser. It is recommended that the file be readable by all users. The file must be accessible by the primary and backup control machines.

ResumeProgram

Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.

SallocDefaultCommand

Must be executable by all users. The file must exist on every login and compute node.

slurm.conf

Readable to all users on all nodes. Must not be writable by regular users.

SlurmctldLogFile

Must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.

SlurmctldPidFile

Must be writable by user root. Preferably writable and removable by SlurmUser. The file must be accessible by the primary and backup control machines.

SlurmdLogFile

Must be writable by user root. A distinct file must exist on each compute node.

SlurmdPidFile

Must be writable by user root. A distinct file must exist on each compute node.

SlurmdSpoolDir

Must be writable by user root. A distinct file must exist on each compute node.

SrunEpilog

Must be executable by all users. The file must exist on every login and compute node.

SrunProlog

Must be executable by all users. The file must exist on every login and compute node.

StateSaveLocation

Must be writable by user SlurmUser. The file must be accessible by the primary and backup control machines.

SuspendProgram

Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.

TaskEpilog

Must be executable by all users. The file must exist on every compute node.

TaskProlog

Must be executable by all users. The file must exist on every compute node.

UnkillableStepProgram

Must be executable by user SlurmUser. The file must be accessible by the primary and backup control machines.

LOGGING

Note that while Slurm daemons create log files and other files as needed, it treats the lack of parent directories as a fatal error. This prevents the daemons from running if critical file systems are not mounted and will minimize the risk of cold-starting (starting without preserving jobs).

Log files and job accounting files, may need to be created/owned by the "SlurmUser" uid to be successfully accessed. Use the "chown" and "chmod" commands to set the ownership and permissions appropriately. See the section FILE AND DIRECTORY PERMISSIONS for information about the various files and directories used by Slurm.

It is recommended that the logrotate utility be used to insure that various log files do not become too large. This also applies to text files used for accounting, process tracking, and the slurmdbd log if they are used.

Here is a sample logrotate configuration. Make appropriate site modifications and save as /etc/logrotate.d/slurm on all nodes. See the logrotate man page for more details.

##
# Slurm Logrotate Configuration
##
/var/log/slurm/*log {

    compress

    missingok

    nocopytruncate

    nocreate

    nodelaycompress

    nomail

    notifempty

    noolddir

    rotate 5

    sharedscripts

    size=5M

    create 640 slurm root

    postrotate
       /etc/init.d/slurm reconfig


    endscript
}

COPYING

This file is part of Slurm, a resource management program. For details, see <http://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

FILES