autowisp.database.image_processing module

Class Inheritance Diagram

Handle data processing DB interactions.

class autowisp.database.image_processing.ExpressionMatcher(evaluated_expressions, ref_image_id, ref_channel, master_expression_ids, *, masters_only=False)

Bases: object

Compare condition expressions for an image/channel to a target.

Usually check if matched expressions and master expression values are identical, but also handles special case of calibrate step.

__call__(image_id, channel)

True iff the expressions for the given image/channel match.

__init__(evaluated_expressions, ref_image_id, ref_channel, master_expression_ids, *, masters_only=False)

Set up comparison to the given evaluated expressions.

_get_master_values(image_id, channel)

Return ready to compare masster expression values.

class autowisp.database.image_processing.ImageProcessingManager(*args, **kwargs)

Bases: ProcessingManager

Read configuration and record processing progress in the database.

Attrs:

See ProcessingManager.

pending(dict): Indexed by step ID, and image type ID list of

(Image, channel name, status) tuples listing all the images of the given type that have not been processed by the currently selected version of the step in the key and their status if previous processing by that step was interrupted or None if not.

_failed_dependencies(dict): Dictionary with keys (step, image_type)

that contains the list of images and channels that failed the given step.

__call__(limit_to_steps=None)

Perform all the processing for the given steps (all if None).

__init__(*args, **kwargs)

Initialize self._failed_dependencies in addition to normali init.

_check_ready(step, image_type, db_session)

Check if the given type of images is ready to process with given step.

Parameters:

• step (Step) – The step to check for readiness.

• image_type (ImageType) – The type of images to check for readiness.

• db_session (Session) – The database session to use.

Returns:

Whether all requirements for the specified processing are

satisfied.

Return type:

bool

_clean_pending_per_dependencies(db_session, from_step_id=None, from_image_type_id=None)

Remove pending images from steps if they failed a required step.

_cleanup_interrupted(db_session)

Cleanup previously interrupted processing for the current step.

_end_processing(input_fname, status=1, final=True)

Record that the current step has finished processing the given file.

Parameters:

input_fname – The filename of the input (DR or FITS) that was processed.

Returns:

None

_finalize_processing()

Update database and instance after processing.

_get_batch_config(batch, master_expression_values, step, db_session)

Split given batch of images by configuration for given step.

The batch must already be split by all relevant condition expressions. Only splits batches by the best master for each image.

Parameters:

• batch ([Image, channel, status]) – List of database image instances and for channels which to find the configuration(s). The channel should be None for the calibrate step

• master_expression_values (tuple) – The values the expressions required to select input masters or to guarantee a unique output master. Should be provided in consistent order for all batches processed by the same step.

• step (Step) – The database step instance to configure.

• db_session – Database session to use for queries.

Returns:

keys: guaranteed to match iff configuration, output master

conditions, and all best input master(s) match. In other words, if this function is called separately on multiple batches, it is safe to combine and process together those that end up with the same key.

values:

dict: The configuration to use for the given (sub-)batch.

[Image]: The (sub-)batch of images to process with given

configuration.

Return type:

dict

_get_config_batches(pending_images, step_input_type, db_session)

Return the batches of images to process with identical config.

_get_interrupted(need_cleanup, db_session)

Return list of interrupted files and configuration for cleanup.

_init_processed_ids(image, channels, step_input_type)

Prepare to record processing of the given image by current step.

_prepare_processing(step, image_type, limit_to_steps)

Prepare for processing images of given type by a calibration step.

_process_batch(batch, *, start_status, config, step_name, image_type_name)

Run the current step for a batch of images given configuration.

_set_calibration_config(config, first_image)

Retrun the specially formatted argument for the calibration step.

_split_by_master(batch, input_master_type)

Split the given list of images by the best master of given type.

_start_processing(input_fname, status=0)

Mark in the database that processing the given file has begun.

Parameters:

input_fname – The filename of the input (DR or FITS) that is about to begin processing.

Returns:

None

_start_step(step, image_type, db_session)

Record the start of a processing step and return the images to process.

Parameters:

• step (Step) – The database step to start.

• image_type (ImageType) – The database type of image to start processing.

• db_session – Active session for database queries.

Returns:

The list of images and channels to process.

str:

The type of input expected by the current step.

Return type:

[(Image, str)]

add_raw_images(image_collection)

Add the given RAW images to the database for processing.

find_processing_outputs(processing_progress, db_session=None)

Return all logging and output filenames for given processing ID.

get_step_input(image, channel_name, step_input_type)

Return the name of the file required by the current step.

group_pending_by_conditions(pending_images, db_session, *, match_observing_session=False, step_id=None, masters_only=False)

Group pendig_images by condition expression values.

Parameters:

• pending_images ([Image, str]) – A list of the images (instance of Image DB class) and channels to group.

• db_session – Database session to use for querries.

• match_observing_session – Whether each group of images needs to be from the same observing session.

• step_id (int) – The ID of the step for which to group the pending images. If not specified, defaults to the current step.

• masters_only – If True, grouping is done only by the values expressions required to determine the input or output masters for the current step.

Returns:

Each entry is contains a list of the image/channel combinations matching a unique set of conditions and the second entry is the master expression values for all images in the list.

Return type:

[([Image, str], tuple)]

set_pending(db_session, steps_imtypes=None, invert=False)

Set the unprocessed images and channels split by step and image type.

Set the self.pending attribute to a dictionary with format {(step.id, image_type.id): (Image, str)}, containing the images and channels of the specified type for which the specified step has not applied with the current configuration.

Parameters:

• db_session (Session) – The database session to use.

• steps_imtypes (Step, ImageType) – The step image type combinations to determine pending images for. If unspecified, the full processing sequence defined in the database is used.

• invert (bool) – If True, returns successfully completed (not failed) instead of pending.

Returns:

None

exception autowisp.database.image_processing.NoMasterError

Bases: ValueError

Raised when no suitable master can be found for a batch of frames.

autowisp.database.image_processing.get_master_expression_ids(step_id, image_type_id, db_session)

List all condition expression IDs determining input or output masters.

Parameters:

• step_id (int) – The ID of the step for which to return the master expression IDs.

• image_type_id (int) – The type of images being processed by the step for which to return the master expression IDs.

Returns:

The combined expression IDs reqired to determine which required masters can be used for the given step or which masters will be created by it.

Return type:

[int]

autowisp.database.image_processing.main(config)

Avoid global variables.

autowisp.database.image_processing.parse_command_line()

Return the command line configuration.

autowisp.database.image_processing.remove_failed_prerequisite(pending, pending_image_type_id, prereq_step_id, db_session)

Remove from pending any entries that failed the prerequisite step.