autowisp.light_curves.lc_data_io module

Class Inheritance Diagram

Define a class for collecting LC data from DR files.

class autowisp.light_curves.lc_data_io.LCDataIO

Bases: object

A callable class which gathers a slice of LC data from frames/DR files.

Suitable for multiprocessing.

Two instances of this class should never be created simultaneously!

(something)_config

Attributes containing the configurations for the various components for which configuration was defined in the processed frames and values which are in turn dictionaries indexed by the configurations found for this component as frozenset suitable for directly passing to LightCurve.add_configuration and values giving the number assigned to those configurations in the LCDataSlice structure that was filled.

lc_data_slice

An object which will be filled with data as it is being read from data reduction files.

Type:

LCDataSlice

source_destinations

Key: (field, source). Value: the offset of the given source in the LCDataSlice arrays that are per-source.

Type:

dict

_config

The configuration of how to perform the LC dumping.

_add_to_data_slice(*, data_reduction, frame_header, frame_index, lc_example)

Add the information from a single frame to the LCDataSlice.

Parameters:

• data_reduction (DataReductionFile) – An opened (at least for reading) data reduction file for the frame being processed.

• frame_header (dict-like) – A pyfits header or the equivalent dictionary for the frame being processed.

• frame_index (int) – The index at which to place this frame’s data in the LCDataSlice object being filled.

• lc_example (LightCurveFile) – See _classify_datasets().

Returns:

The of sources IDs (as tuples of integers) between (exclusive) self.last_source_below and self.first_source_above found in at least some of the frames which were not included in the LCDataSlice.

Return type:

[]

classmethod _classify_datasets(lc_example, ignore_splits)

Set dataset_dimensions, header_datasets, & config_components attributes.

Parameters:

• lc_example (LightCurveFile) – A fully configured instance of LightCurveFile to serve as an example of the structure of lightcurve files to expect.

• ignore_splits – See path_substitutions argument of create().

classmethod _config_to_lc_format(lc_quantity, lc_dtype, value)

Return value as it would be read from the LC.

_get_configurations(data_reduction, frame_header, get_lc_dtype)

Extract all configurations from the given data reduction file.

Parameters:

• data_reduction (DataReductionFile) – An opened (at least for reading) data reduction file for the frame being processed.

• frame_header (dict-like) – A pyfits header or the equivalent dictionary for the frame being processed.

• get_lc_dtype (callable) – Should return the data type of a dataset within light curves.

Returns:

The keys are the various components for which configuration is defined in the given data reduction file/frame. The values are lists of 2-tuples:

• a tuple of the coordinates along each dataset dimension for which the configuration applies.

• a frozen set of 2-tuples:

  • the pipeline key of the configuration quantity

  • The value of the configuration parameter.

Return type:

dict

classmethod _get_dimensions_iterator(dimensions, contract_multivalue=True, skip_dimensions=('frame', 'source'))

Return iterator over all possible values for a set of dimensions.

Parameters:

• dimensions (str or iterable) – Either the name of dataset for which to iterate over all dimensions other than ‘frame’ and ‘source’ or a tuple contaning all dimensions to iterate over.

• contract_multivalue (bool) – If true, multi-valued dimensions are treated as if they have a size of 1.

Returns:

Covering all possible combinatinos of values for each dimensions identified by dimensions.

Return type:

iterator

classmethod _get_field_index(quantity, dim_values, frame_index, source_index=None)

Return the index within the field for the specified entry.

classmethod _get_lc_data(quantity, dimension_values, source_index, defined_indices=None)

Return the data that should be added to an LC dataset.

Parameters:

• quantity (str) – The pipeline key of the LC quantity that will be extended.

• dimension_values (tuple(int)) – The values of the dimensions on which the dataset depends.

• source_index (int) – The index of the source for which the LC is being updated within the LCDataReader.lc_data_slice.

• defined_indices – See _add_configurations().

Returns:

The data to pass to LightCurveFile.extend_dataset.

Return type:

numpy.array

classmethod _get_num_entries(dimensions)

Return how many entries are in a single value for the given dims.

classmethod _get_slice_field(pipeline_key)

Return the field in the data slice containing the given quantity.

classmethod _get_substitutions(dimensions, dimension_values)

Return dict of path substitutions to fully resolve path to quantity.

Parameters:

• dimensions – See _get_dimensions_iterator().

• dimension_values (tuple(int)) – The values for the various dataset dimensions which to turn to substitutions.

Returns:

Ready to be passed to functions needing it to resolve path.

Return type:

dict

classmethod _set_field_entry(quantity, value, *, frame_index, dim_values, source_index=None)

Set the correct index within the field for the specified entry.

Parameters:

• quantity (str) – The pipeline key of the dataset being filled.

• value – The value to set to the field. Must have the correct type.

• frame_index – See _add_to_data_slice().

• dim_values (tuple(int,...)) – The value for each dimension of the dataset.

• source_index (int or None) – For source dependent datasets only, the index of the source in the data_slice.

Returns:

None

classmethod _write_configurations(light_curve, source_index, defined_indices, resolve_lc_size='confirmed')

Add all configurations to the LC and fix their config_ids in slice.

Parameters:

• light_curve (LightCurveFile) – The light curve to update.

• source_index (int) – The index of the source for which the lightcurve is being updated within the slice.

• defined_indices (numpy.array(dtype=bool)) – Array of flags indicating for each entry in the data slice should be included in this light curve.

Returns:

None

classmethod _write_slice_data(light_curve, source_index, defined_indices, resolve_lc_size)

Add all non-configuration datasets to the light curve.

cfg_index_id = 'cfg_index'

Added to the end of configuration dataset names to get the corresponding index dataset.

config_components = {}

Keys are all the configuration components that can be defined.

Values are - 2-tuple:

• a tuple of the keywords required to resolve the path to the configuration index dataset for this component for which all values found must be dumped.

• a set of the datasets belonging to this component

classmethod create(*, catalog_sources, config, source_id_parser, dr_fname_parser, source_list=None, optional_header=None, observatory=None, **path_substitutions)

Configure the class for use in multiprocessing LC collection.

Parameters:

• config –

  The configuratoin of how to generate the lightcurves, usually as parsed from the command line. In the latter case, the configuration should contain the following attributes:

  • max_apertures: The maximum number of photometriec apertures in any input frame.

  • num_magfit_iterations: The maximum number of magnitude fitting iterations in any input frame.

  • lcdump_catalogue: The filename of a catalogue file containing at least RA and Dec.

  • single_photref_dr_fname: The filename of the single photometric reference used for magfit of input images.

  • srcextract_psf_params: List of the parameters describing PSF shapes of the extracted sources.

  • srcproj_column_names: List of the projected source columns to transfer to the collected lightcurves.

  • memblocksize: The maximum amount of memory (in bytes) to allocate for temporaririly storing source information before dumping to LC.

• source_id_parser – A callable that can convert string source IDs to the corresponding tuple of integers source IDs.

• dr_fname_parser – A callable that parser the filename of DR files returning extra keywrds to add to the header.

• source_list – A list that includes all sources for which lightcurves will be generated. Sources not in this list are ignored. If None, the sources in the catalogue are used instead.

• optional_header –

  Indicate which header keywords could be missing from the FITS header. Missing keywors will be replaced by appropriate values indicating an unknown value. Should be in one of 3 formats:

  • None: all expected header keywords must be present in every frame.

  • ’all’: No header keywords are required.

  • iterable of strings: to be querried by the python in keyword.

• observatory (None or dict) – If specified, should be a dictionary with keys ‘SITELAT’, ‘SITELONG’, and ‘SITEALT’ which overwrite or add the header information for the observatory’s location on earth.

• path_substitutions – Path substitutions to be kept fixed during the entire lightcurve dumping process. Used to resolve paths both within the input data reduction files and with the generated lightcurves.

Returns:

None

dataset_dimensions = {}

Identifiers for the dimensions of datasets. The keys are the pipeline keys of lightcurve datasets and the values are tuples containing some of: ‘frame’, ‘source’, ‘srcproj_column_name’, ‘aperture_index’, ‘magfit_iteration’, ‘srcextract_psf_param’ identifying what the entries in the dataset depend on. It also contains a special dataset ‘source_in_frame’ which indicates which sources have observations in which frames.

header_datasets = {}

The set of datasets which contain FITS header keywords. The index is the pipeline key identifying the dataset, and the value is the corresponding header keyword.

max_dimension_size = {}

A dictionary with keys the various dimensions on which datasets can depend (see dataset_dimensions) and entries the maximum size for each dimension.

classmethod prepare_for_reading()

Must be called every time a new batch of frames is being read.

classmethod prepare_for_writing(configurations_collection)

Prepare for writing after a slice of data has been read.

Parameters:

configurations_collection – The list of configurations returned by LCDataReader for the data currently in the slice. It is assumed that the configurations follow the same order as the data slice entries.

Returns:

None

classmethod print_organized_configurations()

Print the result of organize_configurations() nicely formatted.

read(frame)

Add single frame’s information to configurations and the LCDataSlice.

Parameters:

frame (-) –

A 2-tulpe containing the following:

• The filename of the DR file to read.

• The index at which to place this frame’s data in the LCDataSlice object being filled.

Returns:

See return value of _get_configurations().

[]:

See return value of _add_to_data_slice().

Return type:

dict

write(source_and_light_curve)

Write the data of the given source to the given light curve.