Example Usage: Processing the test data

The following will walk you through processing a very small collection of images to lightcurves.

We use a jupyter notebook format for this description to allow this to be closer to a living document that updates along with AutoWISP, however it is meant to demonstrate the usage of AutoWISP from the command line. All commands starting with ! or % are shell commands, meaning you can execute them in your operating system’s terminal (command prompt in Windows).

Create a temporary directory to work in (remember to delete it at the end), enter it, and get its path

%mkdir autowisp_test
%cd autowisp_test

mkdir: autowisp_test: File exists
/Users/kpenev/projects/git/AutoWISP/autowisp/tests/test_data/autowisp_test

Download the test data

The raw images processed in this example as well as the expected files that processing will produce can be downloaded from: (https://zenodo.org/records/15786531/files/test_data.zip?download=1)

Download the zip file from zenodo and unpack it. This will produce several directories called: CAL, DR, Gaia, LC, MASTERS, and RAW. It should also produce several files, one of which is called test.cfg.

Move RAW and test.cfg to the directory we just created.

The remaining directories contain the files that processing should produce. You can compare what you get to those files.

Perform the calibration for the zero frames

!wisp-calibrate --config-file test.cfg RAW/zero

This should produce a directory called CAL under which there will be another directory called zero which contains the calibrated bias frames.

Create a master bias by stacking the calibrated zero frames

!wisp-stack-to-master -c test.cfg CAL/zero/

This should produce a directory called MASTERS containing a single FITS file, which is the master bias.

Perform the calibration for the dark frames

Note that when calibrating the dark frames we wish to subtract the master bias generated above. In order to do this, we specify that the master that was generated above should be used as a master bias by adding --master-bias "R:MASTERS/zero_R.fits.fz" to our command line

!wisp-calibrate --config-file test.cfg RAW/dark/*.fits.fz --master-bias "R:MASTERS/zero_R.fits.fz"

There should now be another directory under CAL, called dark which contains the calibrated dark frames.

Create a master dark by stacking the calibrated dark images

!wisp-stack-to-master -c test.cfg CAL/dark/

Perform calibration for flat frames

Those should be corrected both with the master bias and the master dark generated in the previous two steps:

!wisp-calibrate --config-file test.cfg RAW/flat/*.fits.fz --master-bias "R:MASTERS/zero_R.fits.fz" --master-dark "R:MASTERS/dark_R.fits.fz"

Build a master flat out of the individual calibrated flat frames

Note that creating a master flat uses a different command than creating master bias and dark!

!wisp-stack-to-master-flat -c test.cfg CAL/flat/

Perform calibration for object frames

Now we use all three masters

!wisp-calibrate --config-file test.cfg RAW/object/*.fits.fz --master-bias "R:MASTERS/zero_R.fits.fz" --master-dark "R:MASTERS/dark_R.fits.fz" --master-flat "R:MASTERS/flat_R.fits.fz"

If you wish to see how your calibrated images look, we recommend using SAO DS9.

Find stars in the calibrated object frames

This will create another directory called DR which contains what we call data reduction files. These files are where the pipeline saves the results of processing the calibrated images. So fare, these files will only contain the stars that we find in the images. If you wish to see the contents of HDF5 files, the best tool we are aware of is: HDFView

!wisp-find-stars --config-file test.cfg CAL/object/

Perform astrometry on DR files

This matches the extracted sources to a catalog of stars from the Gaia space mission and uses that information to figure out how to convert from coordinates on the sky to coordinates within each image. The conversion will also be stored in the DR files.

Note that this may take a while, since we use an online service to find an initial guess which is often over-subscribed.

!wisp-solve-astrometry --config-file test.cfg DR/

Perform PSF/PRF fitting

!wisp-fit-star-shape --config test.cfg CAL/object/

Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients
Reading model
Reading grid
Reading coefficients

Perform aperture photometry

!wisp-measure-aperture-photometry --config test.cfg CAL/object/

Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Reading backgroundReading background

Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Reading background
Setting apertures
Reading PSF map
Reading model
Reading gridSetting apertures

Reading PSF map
Reading model
Reading grid
Reading coefficients
Reading coefficientsCreating flux measuring object

Creating flux measuring object
Measuring flux for image 0Measuring flux for image
0
tree suffix = tree suffix = .0.0

Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Setting apertures
Reading PSF map
Reading model
Reading grid
Reading coefficients
Creating flux measuring object
Measuring flux for image 0
tree suffix = .0
Reading background
Reading background
Reading background
Reading background
Reading background
Reading background

Run fit source extracted PSF map

!wisp-fit-source-extracted-psf-map --config test.cfg DR/

Perform magnitude fitting

!wisp-fit-magnitudes --config test.cfg DR/

One of the products of this step is a series of magnitude fitting statistics files, which contain store information about most of the stars that are in the images. These are human readable (white-space separated files) that you can view with a text editor or open in your favorite application for manipulating data organized in columns. The inforamtion in this file can be a good guide on how good the photometry is.

For each star, the first half of the columns in the statistics files are as follows:

• For the first aperture:

  • The number of brightness measurements of the star

  • The number of brightness measurements that are not outliers (unusually high or low)

  • The median of all non-outlier brightness measurements

  • A first measure of the uncentainty in the brightness measurements, estimated as the root-mean-square deviation from the median

  • A second measure of the uncertainty in these measurements, estimated as the median of the absolute deviation around the median

• Repeat for the second aperture

• …

The second half of the columns are meant for internal use, and their definition may change in the future.

Create light curves

!wisp-create-lightcurves --config test.cfg DR/

Perform EPD on light curves

!wisp-epd --config test.cfg LC/

Parsing: 'sphotref = apphot.magfit.cfg.single_photref : aperture_index = 0'.
Parsing: "x = srcproj.columns : srcproj_column_name = 'x' & srcproj_version = 0".
Parsing: "y = srcproj.columns : srcproj_column_name = 'y' & srcproj_version = 0".
Parsing: 'bg = bg.value'.
Parsing: 'z = skypos.zenith_distance'.
Parsing: "S = srcextract.psf_map.eval: srcextract_psf_param = 's'".

Generate EPD statistics file for light curves

!wisp-generate-epd-statistics --config test.cfg LC/

Parsing: 'sphotref = apphot.magfit.cfg.single_photref : aperture_index = 0'.

Perform TFA on light curves

!wisp-tfa --config test.cfg LC/

Parsing: 'sphotref = apphot.magfit.cfg.single_photref : aperture_index = 0'.

Generate TFA statistics file for light curves

!wisp-generate-tfa-statistics --config test.cfg LC/

Parsing: 'sphotref = apphot.magfit.cfg.single_photref : aperture_index = 0'.

Extract brightness vs time from lightcurves

The lightcurves generated above are in HDF5 format. You can open the directly using HDFView as before. We have attempted to organze the content in a logical way. However, these files contain my be somewhat cumbersome to work with, and likely contain a lot more entries than typical user would care about. As an example, let us collect all aperture photometry measurements from one of the newly generated lightcurves in a CSV file

!wisp-get-from-lc LC/GDR3_809281106831668736.h5 --expression bjd=skypos.BJD --expression 'mag{aperture_index}=apphot.magfit.magnitude' -o GDR3_809281106831668736.csv --find-best

/opt/anaconda3/lib/python3.12/site-packages/autowisp/database/hdf5_file_structure.py:74: SAWarning: relationship 'Step.prerequisites' will copy column step.id to column step_dependencies.blocked_step_id, which conflicts with relationship(s): 'StepDependencies.blocked_step' (copies step.id to step_dependencies.blocked_step_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards.   To silence this warning, add the parameter 'overlaps="blocked_step"' to the 'Step.prerequisites' relationship. (Background on this warning at: https://sqlalche.me/e/20/qzyx) (This warning originated from the `configure_mappers()` process, which was invoked automatically in response to a user-initiated operation.)
  db_session.query(HDF5Product)
Single photref: b'DR/10-465248_2_center.h5'
Aperture indices:  None
Evaluating 'skypos.BJD' for aperture 0
Evaluating 'apphot.magfit.magnitude' for aperture 0
Evaluating 'skypos.BJD' for aperture 1
Evaluating 'apphot.magfit.magnitude' for aperture 1
Evaluating 'skypos.BJD' for aperture 2
Evaluating 'apphot.magfit.magnitude' for aperture 2
Evaluating 'skypos.BJD' for aperture 3
Evaluating 'apphot.magfit.magnitude' for aperture 3
Evaluating 'skypos.BJD' for aperture 4
Evaluating 'apphot.magfit.magnitude' for aperture 4

You can open this file with your favorite text editor or load it in your favorite software for manipulating column-organized data. The contents of the file should look like:

with open('GDR3_809281106831668736.csv', 'r') as f:
    print(f.read())

bjd,mag0,mag1,mag2,mag3
2457819.837050383,4.006945615264255,3.17406685414204,3.0312522103547104,2.981423791605317
2457819.837690473,3.743435615264255,3.0646768541420397,2.9453222103547105,2.886863791605317
2457819.838957673,4.121655615264255,3.17709685414204,3.0010422103547105,2.962123791605317
2457819.839599043,4.031455615264255,3.18722685414204,3.0456322103547104,3.044273791605317
2457819.840274193,3.945705615264255,3.15156685414204,2.9139522103547106,2.827073791605317
2457819.840906103,4.124025615264255,3.26076685414204,3.0048822103547104,2.940343791605317
2457819.841551873,4.051135615264255,3.19773685414204,2.9911622103547106,2.912093791605317
2457819.842187163,3.990045615264255,3.12918685414204,2.9628322103547107,2.919363791605317
2457819.842825823,3.761285615264255,3.04416685414204,2.8789522103547105,2.860253791605317

There are nine rows in the file corresponding to the nine successfully processed science images in this dataset.

• the first column (bjd) is the Barycentric Julian Date. This is a commonly used way of representing time in astronomy in units of days. It compensates for different light travel time from different stars to the Earth as the Earth revolves around the Sun, compensating for the fact that a given astronomical event will be observed to occur earlier if the earth is closer to the source than the sun and later if it is further.

• The remaining four columns show the measured and detrended brightness of the star in each of the four apertures used in this example

When you are done reviewing your results, remember to delete the directory you created at the beginning