autowisp.database.processing module

Class Inheritance Diagram

Define base class for processing images or lightcurves.

exception autowisp.database.processing.ProcessingInProgress(processing)

Bases: Exception

Raised when a particular step is running in a different process/host.

class autowisp.database.processing.ProcessingManager(version=None, dummy=False)

Bases: ABC

Utilities for automated processing of images or lightcurves.

Attrs:

configuration(dict): Indexed by parameter name with values further

dictionaries with keys:

version: the actual version used including fallback

value: dict indexed by frozenset of expression IDs that an image must satisfy for the parameter to have a given value.

condition_expressions({int: str}): Dictionary of condition

expressions that must be evaluated against the header of each input images to determine the exact values of the configuration parameters applicable to a given image. Keys are the condition expression IDs from the database and values are the actual expressions.

step_version(dict): Indexed by step name of the largest value of the

actual version used for any parameter required by that step.

current_step(Step): The currently active step.

_current_processing(ImageProcessingProgress): The currently active

step (the processing progress initiated the last time start_step() was called).

_processed_ids(dict): The keys are the filenames of the required

inputs (DR or FITS) for the current step and the values are dictionaries with keys 'image_id' and 'channel' identifying what was processed.

_evaluated_expressions(dict): Indexed by image ID and then channel,

dictionary containing dictionary with keys:

• values: the values of the condition expressions for the given image and channel indexed by their expression IDs.

• matched: A set of the expression IDs for which the corresponding expression converts to boolean True.

• calibrated: the filename of the calibrated image

• dr: the filename of the data reduction file

• masters: A dictionary indexed by master type name of the best master of the given type to apply to the image

An additional entry with channel=None is included which contains just the common (intersection) set of expressions satisfied for all channels.

_master_expressions(dict): Indexed by master type, then tuple of

expression values ordered by expression ID of the masters of the given type that match the given expression values.

pending(dict): Information about what images or lightcurves still

need processing by the various steps. The format is different for image vs lightcurve processing managers.

abstractmethod __call__(limit_to_steps=None)

Perform all the processing for the given steps (all if None).

__init__(version=None, dummy=False)

Set the public class attributes per the given configuartion version.

Parameters:

• version (int) – The version of the parameters to get. If a parameter value is not specified for this exact version use the value with the largest version not exceeding version. By default us the latest configuration version in the database.

• dummy (bool) – If set to true, all logging is suppressed and no processing can be performed. Useful for reviewing the results of past processing.

Returns:

None

_check_running_processing(running_processing, this_host, db_session)

Check if any unfinished processing progresses are still running.

abstractmethod _cleanup_interrupted(db_session)

Cleanup previously interrupted processing for the current step.

_create_current_processing(step, target, db_session)

Add a new ProcessingProgress at start of given step.

_get_best_master(candidate_masters, image_eval)

Find the best master from given list for given image/channel.

_get_evaluated_entry(evaluate, image_type_id, calib_config, db_session)

Return entry to add to self._evaluated_expressions.

_get_master(master_type, image_values, image_eval, db_session)

Return the master that should be used for the given image.

_get_split_channels(image)

Return the split_channels option for the given image.

_write_config_file(matched_expressions, outf, db_session, *, db_steps=None, step_names=None)

Write to given file configuration for given matched expressions.

Returns:

Set of tuples of parameters and values as set in the file. Used for comparing configurations.

add_masters(new_masters, step_name=None, image_type_name=None)

Add new master files to the database.

Parameters:

• new_masters (dict or iterable of dicts) –

  Information about the new mbaster(s) to add. Each dictionary should include:

  • type: The type of master being added.

  • filename: The full path to the new master file.

  • preference_order: Expression to select among multiple possible masters. For each frame the expression for each candidate master is evaluateed using the frame header and the master with the smallest resulting value is used.

  • disable(bool): Optional. If set to True the masters are recorded in the database, but not flagged enabled.

• step_name (str) – The name of the step that generated the masters.

• image_type_name (str) – The name of the type of images whose processing created the masters.

create_config_file(example_header, outf, steps=None)

Save configuration for processing given header to given output file.

Parameters:

• example_header (str or dict-like) – The header to use to determine the values of the configuration parameters. Can be passed directly as a header instance or FITS or DR filename.

• outf (file or str) – The file to write the configuration to. Can be passed as something providing a write method or filename. Overwritten if exists.

• steps (list) – If specified, only configuration parameters required by these steps will be included.

• steps=None

Returns:

None

evaluate_expressions_image(image, db_session)

Return evaluator for header expressions for given image.

Parameters:

• image (Image) – Instance of database Image for which to evaluate the condition expressions. The image header is augmented by IMAGE_TYPE keyword set to the name of the image type of the given image.

• db_session – Used to select the best master.

• return_evaluator (bool) – Should an evaluator setup per the image header be returned for further use?

Returns:

Evaluator ready to evaluate additional expressions involving FITS headers. Only returned if return_evaluator is True.

Return type:

Evaluator or None

get_config(matched_expressions, db_session, *, db_step=None, step_name=None, image_id=None, channel=None)

Return the configuration for the given step for given expressions.

get_master_fname(image_id, channel, master_type_name)

Return the filename of best master for a given image/channel.

get_matched_expressions(evaluate)

Return set of matching expressions given an evaluator for image.

get_param_values(matched_expressions, parameters=None, db_session=None)

Return the values to use for the given parameters.

Parameters:

• matched_expressions (set) – Set of expression IDs that the image we are getting configuration for matches.

• parameters ([] or str) – List of parameter names, or a step, or its name to get configuration for. Defaults to current step if not specified.

• as_args (bool) – If True, return a list of arguments ready to pass directly to one of the command line parser of the processing steps.

• db_session – Session to use for DB queries. Only needed if specifying parameters by step name or using default.

Returns:

The values for the given parameters indexed by

parameter name.

Return type:

dict or list

get_product_fname(image_id, channel, product)

Return the dr or calibrated filename of specified image/channel.

self.evaluate_image_expressions() must already have been called for this image.

abstractmethod set_pending(db_session)

Set the unprocessed images and channels split by step and image type.

Parameters:

db_session (Session) – The database session to use.

Returns:

(Image, str)}:

The images and channels of the specified type for which the specified step has not applied with the current configuration.

Return type:

{(step.id, image_type.id)