A complete description of the data format can be found here and exemple code to read it is provided below.
The format of the DIVA high-level science products (HLSP) follows the specification of the format for high-contrast imaging that was proposed by the ALICE project. The goal of this format is to simplify HLSP exchange in the HCI community by adopting a standard output format for all the usual data products. The format is based on a single multi-extension FITS file (Pence et al.2010) that gathers all the high-level information about the datasets, which we call HCI-FITS file for High-Contrast Imaging FITS file. In the following we introduce the main specifications of Version 1 of HCI-FITS. This approach is largely inspired from the OI-FITS format, which is the standard for calibrated data exchange from optical interferometers (Pauls et al. 2004, 2005).
A single FITS file — To enable high-level science analyses of high-contrast imaging data, several products are mandatory (e.g. the reduced image, detection limits, source detections). In order to prevent information loss when exchanging data, all the information must be gathered in a single file. The FITS file format is ideal for this task since it enables the encapsulation of heterogeneous data types in independent extensions. This includes images (IMAGE extension type), which is appropriate for the reduced images, sensitivity and SNR maps, and multi-dimension tables (BINTABLE extension type), which is appropriate for radial detection limits, characteristics of potentially detected sources, or general characteristics of the observations and reduced products.
A flexible standard — The HCI-FITS format is developed to be compatible with any type of dataset, regardless of the instrument, or observing mode, or processing method used to obtain the final reduced products. It can be used for both ground-based or space observations, for coronagraphic and saturated imaging, for broad-band imaging, integral-field spectroscopy, and polarimetric imaging. In addition, a HCI-FITS file may contain products for either a single image or an image cube, and keep track of each specific images characteristics.
Structure of the HCI-FITS format — There are 5 different types of data products that are necessary for a high-level astrophysical exploitation of HCI results: the reduced images, the SNR maps, the sensitivity maps (sometimes refered to as "noise" maps), the radial detection limits (or "contrast curves"), and the properties of detected point sources. A complete HCI-FITS file is therefore composed of 6 extensions, corresponding to the above products plus an extension containing the characteristics of each image in the file. The extensions may appear in any order, but they must implement the EXTNAME keyword that enables their identification. The primary header of any HCI-FITS file also implements a list of the available extensions and their type, with keywords in the form of EXT?NAME and EXT?TYPE, where ? represents the corresponding extension number. The extensions, their format and dimensions are detailed in Table 1. The SOURCE_DETECTION is optional but must follow the specified format if present. Other optional extensions may be included, e.g. for intermediate products like unocculted/unsaturated PSF images.
Label | Type | Dimensions | Description |
---|---|---|---|
DATA_INFORMATION | BINTABLE | Nim x 12 | Characteristics of each image |
REDUCED_DATA | IMAGE | Nim x Nx x Ny | Reduced data product |
SNR_MAP | IMAGE | Nim x Nx x Ny | SNR maps |
SENSITIVITY_MAP | IMAGE | Nim x Nx x Ny | 2D detection limits |
DETECTION_LIMIT | BINTABLE | Nim x 2 x Nr | Radial detection limits |
SOURCE_DETECTION | BINTABLE | Nsource x 20 x Nim | Detection point sources properties |
DATA_INFORMATION: this extension of type BINTABLE identifies the characteristics of each reduced image in the HCI-FITS file. The structure of this extension is flexible enough to allow any collection of images. It is composed of 12 fields that track the orientation and all the main characteristics of the images contained in the file. These fields are described in Table 2.
Label | Type | Description |
---|---|---|
Image_Number | I | Index of the image in the cube |
Orientation | D | Sky-orientation of the image vertical axis |
Combined_rotation_angle | D | Parallactic angle combined in the image |
Number_of_Exposures | I | Number of exposures combined in the image |
Exposure_Time | D | Total exposure time combined in the image |
Observation_Start | D | Mod. Julian Date at the start of the exposure |
Observation_End | D | Mod. Julian Date at the end of the exposure |
UT_Midpoint_Date_of_Observation | A | UT date at the image mid-point |
UT_Midpoint_Time_of_Observation | A | UT time at the image mid-point |
Wavelength | D | Effective wavelength of the image |
Bandwidth | D | Effective bandwidth of the image |
Polarization | A | Polarization state of the image |
REDUCED_DATA: this extension of type IMAGE contains the reduced images of the HCI-FITS file. If several images are provided, the order must be the same as in the data provided in the DATA_INFORMATION table.
SNR_MAP: this extension of type IMAGE contains the SNR maps corresponding to the reduced images.
SENSITIVITY_MAP: this extension of type IMAGE contains the 2D sensitivity maps of the observations. The header of this extension must contain the
DETECTION_LIMIT: this extension of type BINTABLE contains the radial point source detection limits for each image present in the file. The header of this extension must also contain the
Label | Type | Description |
---|---|---|
Radius | D (Nim) | Radial separation from the star |
Detection_Limit | D (Nim) | Point source detection limit |
SOURCE_DETECTION: this extension of type BINTABLE contains the properties of all the point sources detected in each of the images. This table has 20 fields, which are described in Table 4.
Label | Type | Description |
---|---|---|
Candidate | I | Index of the detected source |
SNR | D (Nim) | SNR of the source |
dRA | D (Nim) | Relative R.A. from the star |
err_dRA | D (Nim) | Uncertainty on the dRA |
dDEC | D (Nim) | Relative Declination from the star |
err_dDEC | D (Nim) | Uncertainty on the dDEC |
Sep | D (Nim) | Separation from the star |
err_Sep | D (Nim) | Uncertainty on the separation |
PA | D (Nim) | Position Angle (east of north) |
err_PA | D (Nim) | Uncertainty on the PA |
Flux_cs | D (Nim) | Photometry in count/s |
err_Flux_cs | D (Nim) | Uncertainty on Flux_cs |
Flux_mag | D (Nim) | Photometry in magnitude |
err_Flux_mag | D (Nim) | Uncertainty on Flux_mag |
Flux_Jy | D (Nim) | Photometry in Jy |
err_Flux_Jy | D (Nim) | Uncertainty on Flux_Jy |
Flux_erg | D (Nim) | Photometry in erg/cm2/s/A |
err_Flux_erg | D (Nim) | Uncertainty on Flux_erg |
Contrast | D (Nim) | Contrast from the star |
err_Contrast | D (Nim) | Uncertainty on the Contrast |
The following sections contains simple code exemples in Python and IDL to read and display some of the content of the DIVA HLSP. You can download the sample codes with a DIVA test file.
import matplotlib.pyplot as plt import numpy as np from astropy.io import fits # # read HLSP # hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits' hdu = fits.open(hlsp_file) hdr = hdu[0].header # # list extensions # nextend = hdr['NEXTEND'] print() print('File has {0} extensions:'.format(nextend)) extensions = {} for i in range(1, nextend+1): extname = hdr['EXT{0}NAME'.format(i)] exttype = hdr['EXT{0}TYPE'.format(i)] print(' * {0}; {1}'.format(extname, exttype)) # save for later extensions[extname] = i # retrieve basic information surv = hdr['SURVNAME'] survpi = hdr['SURVPI'] instru = hdr['INSTRUME'] filt = hdr['FILTER'] target = hdr['TARGNAME'] # # data information # print() print('Data information:') idx = extensions.get('DATA_INFORMATION') data_info = hdu[idx].columns # list columns print(data_info) # retrieve obs information date = data_info['UT_Midpoint_Date_of_Observation'].array[0] Texp = data_info['Exposure_Time'].array[0] FoV_rot = data_info['Combined_rotation_angle'].array[0] # # reduced image # idx = extensions.get('REDUCED_DATA') if (idx is not None): img = hdu[idx].data # display fig = plt.figure(0, figsize=(10, 10)) plt.clf() plt.imshow(img, vmin=-0.005*np.nanmax(img), vmax=0.005*np.nanmax(img), aspect='equal') plt.figtext(0.07, 0.9, 'Texp = {0:.1f} min \nFoV rot = {1:.1f} deg'.format(Texp/60, FoV_rot), fontsize=15) plt.title('{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi)) plt.tight_layout() # # detection limit # idx = extensions.get('DETECTION_LIMIT') if (idx is not None): dat = hdu[idx].data.columns # threshold nsigma = hdu[idx].header['NSIGMA'] # values sep = dat['Radius'].array dlim = dat['Detection_limit'].array # units sep_u = dat['Radius'].unit dlim_u = dat['Detection_limit'].unit # plot fig = plt.figure(1, figsize=(10, 7)) plt.clf() plt.plot(sep, dlim, linestyle='-', color='b') plt.gca().invert_yaxis() plt.xlabel(r'Angular separation [{0}]'.format(sep_u)) plt.ylabel(r'Sensitivity limit @ {0:.0f}$\sigma$ [{1}]'.format(nsigma, dlim_u)) plt.title(r'{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi)) plt.tight_layout() # # candidates # print() print('Detected candidates:') candnum = hdr['CANDNUM'] if (candnum == -1): print(' * this survey has no information on detected candidates') elif (candnum > 0): idx = extensions.get('SOURCE_DETECTION') dat = hdu[idx].data.columns # list properties print('idx dRA err_dRA dDEC err_dDEC dmag err_dmag') print('---------------------------------------------------------------') for c in range(candnum): print('{:2d}{:10.4f}{:10.4f}{:10.4f}{:10.4f}{:10.2f}{:7.2f}'. format(c, dat['dRA'].array[c], dat['err_dRA'].array[c], dat['dDEC'].array[c], dat['err_dDEC'].array[c], dat['Contrast'].array[c], dat['err_Contrast'].array[c]))
;; ;; read HLSP ;; hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits' hdu = mrdfits(hlsp_file, 0, hdr, /silent) ;; ;; list extensions ;; nextend = sxpar(hdr, 'NEXTEND') print print,'File has '+string(nextend, format='(I0)')+' extensions:' extensions = [''] for i=1,nextend do begin extname = sxpar(hdr, string(i, format='("EXT",I0,"NAME")')) exttype = sxpar(hdr, string(i, format='("EXT",I0,"TYPE")')) print,' * '+extname+'; '+exttype ;; save for later extensions = [extensions, extname] endfor ;; retrieve basic information surv = strtrim(sxpar(hdr, 'SURVNAME')) survpi = strtrim(sxpar(hdr, 'SURVPI')) instru = strtrim(sxpar(hdr, 'INSTRUME')) filt = strtrim(sxpar(hdr, 'FILTER')) target = strtrim(sxpar(hdr, 'TARGNAME')) ;; ;; data information ;; print print,'Data information:' idx = where(extensions eq 'DATA_INFORMATION') data_info = mrdfits(hlsp_file, idx[0], /silent) ;; list colmuns help,/str,data_info ;; retrieve obs information date = data_info.UT_Midpoint_Date_of_Observation[0] Texp = data_info.Exposure_Time[0] FoV_rot = data_info.Combined_rotation_angle[0] ;; ;; reduced image ;; idx = where(extensions eq 'REDUCED_DATA') if (idx ne -1) then begin img = mrdfits(hlsp_file, idx[0], /silent) img[where(finite(img) eq 0)] = 0 img <= 0.005*max(img) img >= -0.005*max(img) window, 0, xs=600, ys=600 tvscl, congrid(img, 600, 600) xyouts, 10, 10, target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')', $ color=255, /device, size=1.8 endif ;; ;; detection limit ;; idx = where(extensions eq 'DETECTION_LIMIT') if (idx ne -1) then begin dat = mrdfits(hlsp_file, idx[0], chdr, /silent) ;; threshold nsigma = sxpar(chdr, 'NSIGMA') ;; values sep = dat.Radius dlim = dat.Detection_limit ;; plot window, 1, xs=640, ys=480 plot,sep,dlim,yr=[max(dlim, /nan), min(dlim, /nan)], xtitle='Angular separation', $ ytitle='Sensitivity limite @ '+string(nsigma, format='(I0)')+'-sigma', $ title=target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')' endif ;; ;; candidates ;; print print,'Detected candidates:' candnum = sxpar(hdr, 'CANDNUM') if (candnum eq -1) then begin print,' * this survey has no information on detected candidates' endif else if (candnum gt 0) then begin idx = where(extensions eq 'SOURCE_DETECTION') dat = mrdfits(hlsp_file, idx[0], /silent) ;; list properties print,'idx dRA err_dRA dDEC err_dDEC dmag err_dmag' print,'---------------------------------------------------------------' for c=0,candnum-1 do begin print, c, dat[c].dRA, dat[c].err_dRA, dat[c].dDEC, dat[c].err_dDEC, $ dat[c].Contrast, dat[c].err_Contrast,format='(I2,F10.4,F10.4,F10.4,F10.4,F10.2,F7.2)' endfor endif end