esorex(1) ESO Recipe Execution Tool

SYNOPSIS

esorex [,esorex-options/] ,recipe /[,recipe-options/] ,sof/

DESCRIPTION

EsoRex is the ESO Recipe Execution Tool. It can list, configure and execute CPL-based recipes from the command line.

One of the features provided by the CPL is the ability to create data-reduction algorithms that run as plugins (dynamic libraries). These are called recipes and are one of the main aspects of the CPL data-reduction development environment.

As these recipes are dynamic libraries, it is not possible to run them directly from the command line. However, ESO provides several tools to do this, thus saving recipe developers the need to write such an application themselves. One of these is GASGANO (a GUI-based tool) and the other is EsoRex (which runs from the command line) and is described here.

OPTIONS

--help
Display this help and exit. If a recipe name is also given, then help will be given for it as well. [TRUE]
--version
Display version information and exit. [FALSE]
--check-sof-exist
When TRUE, all the input files must exist and be readable before calling the recipe. This option may also be set using the environment variable ESOREX_CHECK_SOF_EXIST. [FALSE]
--config
Configuration file to be used for EsoRex. This option may also be set using the environment variable ESOREX_CONFIG. []
--create-config
Creates a configuration file for Esorex. If set to TRUE a config file 'esorex.rc' is created in the '.esorex' directory in $HOME of the user.If a filename is specified, a config file will be created accordingly. If a recipe is specified in the command line, then the configuration file will be created for the recipe instead (called 'recipename.rc') Note that an existing file will be overwritten, but a backup file will be copied to 'filename.rc.bak' in the same directory. If the filename ends with extension .json then a machine-readable JSON format will be used. This option may also be set using the environment variable ESOREX_CREATE_CONFIG. [FALSE]
--link-dir
The directory in which a symbolic link to each of the product files should be written. The enable/disable switch to control whether the link is actually made is the '--suppress-link' option. This option may also be set using the environment variable ESOREX_LINK_DIR. [/tmp]
--log-dir
Directory where to place the logfile. This option may also be set using the environment variable ESOREX_LOG_DIR. [.]
--log-file
Filename of logfile. This option may also be set using the environment variable ESOREX_LOG_FILE. [esorex.log]
--log-level
Controls the severity level of messages that will be printed to the logfile. This option may also be set using the environment variable ESOREX_LOG_LEVEL. <debug | info | warning | error | off> [info]
--no-datamd5
Disables the computation of the MD5 data hash for FITS product files. This option may also be set using the environment variable ESOREX_NO_DATAMD5. [FALSE]
--no-checksum
Disables the computation of the standard FITS product checksums. This option may also be set using the environment variable ESOREX_NO_CHECKSUM. [FALSE]
--man-page
Display a manual page for the specified recipe, and then exit. Note that this option only applies to recipes, and that it does nothing for esorex by itself. See also the '--help' option. This option may also be set using the environment variable ESOREX_MAN_PAGE. [FALSE]
--mem-check
Report on memory status at completion of recipe execution. This option may also be set using the environment variable ESOREX_MEM_CHECK. [FALSE]
--msg-level
Controls the severity level of messages that will be printed to the terminal. This option may also be set using the environment variable ESOREX_MSG_LEVEL. <debug | info | warning | error | off> [info]
--output-dir
The directory where the product files should be finally moved to (all products are first created in the current dir). This option may also be set using the environment variable ESOREX_OUTPUT_DIR. [.]
--output-prefix
Prefix applied to any output file. For example, specifying 'pre' would translate 'filename.fits' to 'pre_0000.fits'. See also the '--suppress-prefix' option. This option may also be set using the environment variable ESOREX_OUTPUT_PREFIX. [out]
--output-readonly
When TRUE, any existing product files in the specified output directory will be set to read-only, for user, group and other. If FALSE, then EsoRex will use the default permissions for that account/directory. destroy any pre-existing files. This option exists for the Paranal operations environment. This option can additionally be used to prevent EsoRex from overwriting pre-existing files. This option may also be set using the environment variable ESOREX_OUTPUT_READONLY. [FALSE]
--paf-config
Configuration file for creation of PAF files. This option may also be set using the environment variable ESOREX_PAF_CONFIG. []
--params
List the input parameters and their current settings (whether from the command line or a configuration file) for the esorex application. Parameters are labelled using the parameter's alias. If a recipe is also specified, then the list of its parameters will also be generated in the same way. This option may also be set using the environment variable ESOREX_PARAMS. [FALSE]
--products-sof
Output file which contains the FITS files created by the recipe. If the filename ends with extension .json then a machine-readable JSON formatwill be used. This option may also be set using the environment variable ESOREX_PRODUCTS_SOF. []
--recipes
Display a list of all available recipes (that are available in the directory tree specified with '--recipe-dir'). This option may also be set using the environment variable ESOREX_RECIPES. [FALSE]
--recipe-config
Configuration file for any selected recipe. This option may also be set using the environment variable ESOREX_RECIPE_CONFIG. []
--recipe-dir
Directory containing recipe libraries. Note that esorex will recursively search not only the specified directory, but all sub-directories below it as well. Multiple directory heads may be specified, by separating the starting paths with colons (:). This option may also be set using the environment variable ESOREX_PLUGIN_DIR. [.]
--suppress-link
When TRUE, no symbolic link is created to the output product. However, if FALSE, then a symbolic link is created in the directory specified with the option '--link-dir' for each product that is created by the recipe. This option may also be set using the environment variable ESOREX_SUPPRESS_LINK. [TRUE]
--suppress-prefix
When TRUE, the original name of the output product, as produced by the recipe, is maintained. If FALSE, then the name of the output file is changed to the "prefix_number" format. The prefix can be altered using the '--output-prefix' option. This option may also be set using the environment variable ESOREX_SUPPRESS_PREFIX. [FALSE]
--time
Measure and show the recipe's execution time. This option may also be set using the environment variable ESOREX_TIME. [FALSE]
--unload-plugin
When TRUE, the plugin is unloaded after execution. If FALSE, the plugin is not unloaded after processing, so that a software like, e.g. valgrind, can be used for debugging the executed recipe. This option may also be set using the environment variable ESOREX_UNLOAD_PLUGIN. [TRUE]

ENVIRONMENT

All options can be set as environment parameters as well. See the previous paragraph for details.

FILES

/etc/esorex.rc $HOME/.esorex/esorex.rc

Default configuration files

The configuration file contains the EsoRex options, less the `--' switch, but prefixed with `esorex.caller.'. Blank lines are ignored and lines beginning with `#' are treated as comments.

Here is an example configuration file.


 # Example EsoRex configuration file
 #
 esorex.caller.recipe-dir=/home/username/EsoRex/Plugins
 esorex.caller.log-dir=.
 esorex.caller.log-file=esorex.log
 esorex.caller.log-file=esorex.log
 esorex.caller.output-dir=.
 esorex.caller.output-prefix=out_

filename.sof

A sof file contains a list of the input data. This data is specified in an sof file (which is just a text file), where each input file is specified with its associated classification and category. The format of each line in the sof file is as follows:


 full-path-to-file  classification

Optionally, a third column may be provided. Permitted values are either RAW or CALIB. This is for when a recipe does not identify the type of input file, but as all ESO recipes are required to do so, this column is typically not needed.

An example sof file, for the mythological "ZIMOS" instrument, might look like this:


 /data/mos/ZIMOS.03-12-26T01:05:06.fits  MOS_SCIENCE
 /data/mos/ZIMOS.03-12-26T01:26:00.fits  MOS_SCIENCE
 /data/mos/ZIMOS.03-12-26T01:47:04.fits  MOS_SCIENCE
 /data/cal/master_bias4.fits             MASTER_BIAS
 /data/cal/grs_LR_red.3.tfits            GRISM_TABLE
 /data/gasgano/extract_table2.fits       EXTRACT_TABLE
 /data/cal/badpixel.3.tfits              CCD_TABLE

For an concrete example for a specific instrument, check the documentation for that instrument.

HINTS

File permissions

When a recipe is used with the --suppress-prefix option, and the --output-dir is set to the current working directory, then the first execution of a recipe will work correctly, but subsequent executions may fail. This is due to output products being given `read-only' permission (to avoid the potential inadvertant loss of products). The recipe itself is unable to modify the permissions, and thus it fails when attempting to create the file. The solution (other than using a different output directory or prefixes) is to change the permission of these output files or delete them prior to any subsequent execution of that recipe.

This problem is less likely to occur in EsoRex v2+, due to the replacement of the --output-overwrite option with the --output-readonly (which is disabled by default). However, a determined user can still reach this situation, in which case the non-readable products must have their permissions changed, as described above.

Configuration files

When creating configuration files, if the the recipe is provided on the command line, then EsoRex will generate the configuration file for this recipe. If no recipe name is given, then EsoRex will generate a configuration file for EsoRex itself. All configuration files are written in the $HOME/.esorex/ directory.

Memory checking

It is possible to get EsoRex to check for memory leaks in the recipe that it is running, by enabling the --mem-check option. Then, at the conclusion of the recipe execution, and after memory deallocation, a list of all remaining allocated memory will be printed to screen. If there are no memory leaks, then no addition output is displayed.

AUTHOR

Written by the CPL group (Pipeline System Department) European Southern Observatory / SDD - Garching

REPORTING BUGS

Report bugs to <[email protected]>.

COPYRIGHT

Copyright 2003-2012, European Southern Observatory.