Introduction to HCI-FITS

A complete description of the data format can be found here and exemple code to read it is provided below.

The format of the DIVA high-level science products (HLSP) follows the specification of the format for high-contrast imaging that was proposed by the ALICE project. The goal of this format is to simplify HLSP exchange in the HCI community by adopting a standard output format for all the usual data products. The format is based on a single multi-extension FITS file (Pence et al.2010) that gathers all the high-level information about the datasets, which we call HCI-FITS file for High-Contrast Imaging FITS file. In the following we introduce the main specifications of Version 1 of HCI-FITS. This approach is largely inspired from the OI-FITS format, which is the standard for calibrated data exchange from optical interferometers (Pauls et al. 2004, 2005).

A single FITS file — To enable high-level science analyses of high-contrast imaging data, several products are mandatory (e.g. the reduced image, detection limits, source detections). In order to prevent information loss when exchanging data, all the information must be gathered in a single file. The FITS file format is ideal for this task since it enables the encapsulation of heterogeneous data types in independent extensions. This includes images (IMAGE extension type), which is appropriate for the reduced images, sensitivity and SNR maps, and multi-dimension tables (BINTABLE extension type), which is appropriate for radial detection limits, characteristics of potentially detected sources, or general characteristics of the observations and reduced products.

A flexible standard — The HCI-FITS format is developed to be compatible with any type of dataset, regardless of the instrument, or observing mode, or processing method used to obtain the final reduced products. It can be used for both ground-based or space observations, for coronagraphic and saturated imaging, for broad-band imaging, integral-field spectroscopy, and polarimetric imaging. In addition, a HCI-FITS file may contain products for either a single image or an image cube, and keep track of each specific images characteristics.

Structure of the HCI-FITS format — There are 5 different types of data products that are necessary for a high-level astrophysical exploitation of HCI results: the reduced images, the SNR maps, the sensitivity maps (sometimes refered to as "noise" maps), the radial detection limits (or "contrast curves"), and the properties of detected point sources. A complete HCI-FITS file is therefore composed of 6 extensions, corresponding to the above products plus an extension containing the characteristics of each image in the file. The extensions may appear in any order, but they must implement the EXTNAME keyword that enables their identification. The primary header of any HCI-FITS file also implements a list of the available extensions and their type, with keywords in the form of EXT?NAME and EXT?TYPE, where ? represents the corresponding extension number. The extensions, their format and dimensions are detailed in Table 1. The SOURCE_DETECTION is optional but must follow the specified format if present. Other optional extensions may be included, e.g. for intermediate products like unocculted/unsaturated PSF images.

Table 1: Structure of the HCI-FITS files
Label Type Dimensions Description
DATA_INFORMATION BINTABLE Nim x 12 Characteristics of each image
REDUCED_DATA IMAGE Nim x Nx x Ny Reduced data product
SNR_MAP IMAGE Nim x Nx x Ny SNR maps
SENSITIVITY_MAP IMAGE Nim x Nx x Ny 2D detection limits
DETECTION_LIMIT BINTABLE Nim x 2 x Nr Radial detection limits
SOURCE_DETECTION BINTABLE Nsource x 20 x Nim Detection point sources properties

Description of the extensions

• DATA_INFORMATION: this extension of type BINTABLE identifies the characteristics of each reduced image in the HCI-FITS file. The structure of this extension is flexible enough to allow any collection of images. It is composed of 12 fields that track the orientation and all the main characteristics of the images contained in the file. These fields are described in Table 2.

Table 2: DATA_INFORMATION extension
Label Type Description
Image_Number I Index of the image in the cube
Orientation D Sky-orientation of the image vertical axis
Combined_rotation_angle D Parallactic angle combined in the image
Number_of_Exposures I Number of exposures combined in the image
Exposure_Time D Total exposure time combined in the image
Observation_Start D Mod. Julian Date at the start of the exposure
Observation_End D Mod. Julian Date at the end of the exposure
UT_Midpoint_Date_of_Observation A UT date at the image mid-point
UT_Midpoint_Time_of_Observation A UT time at the image mid-point
Wavelength D Effective wavelength of the image
Bandwidth D Effective bandwidth of the image
Polarization A Polarization state of the image

• REDUCED_DATA: this extension of type IMAGE contains the reduced images of the HCI-FITS file. If several images are provided, the order must be the same as in the data provided in the DATA_INFORMATION table.

• SNR_MAP: this extension of type IMAGE contains the SNR maps corresponding to the reduced images.

• SENSITIVITY_MAP: this extension of type IMAGE contains the 2D sensitivity maps of the observations. The header of this extension must contain the NSIGMA keyword to indicate the confidence level of the sensitivity map.

• DETECTION_LIMIT: this extension of type BINTABLE contains the radial point source detection limits for each image present in the file. The header of this extension must also contain the NSIGMA keyword to indicate the confidence level of the detection limit. The format of this table is described in Table 3.

Table 3: DETECTION_LIMIT extension
Label Type Description
Detection_Limit D (Nim) Point source detection limit

• SOURCE_DETECTION: this extension of type BINTABLE contains the properties of all the point sources detected in each of the images. This table has 20 fields, which are described in Table 4.

Table 4: SOURCE_DETECTION extension
Label Type Description
Candidate I Index of the detected source
SNR D (Nim) SNR of the source
dRA D (Nim) Relative R.A. from the star
err_dRA D (Nim) Uncertainty on the dRA
dDEC D (Nim) Relative Declination from the star
err_dDEC D (Nim) Uncertainty on the dDEC
Sep D (Nim) Separation from the star
err_Sep D (Nim) Uncertainty on the separation
PA D (Nim) Position Angle (east of north)
err_PA D (Nim) Uncertainty on the PA
Flux_cs D (Nim) Photometry in count/s
err_Flux_cs D (Nim) Uncertainty on Flux_cs
Flux_mag D (Nim) Photometry in magnitude
err_Flux_mag D (Nim) Uncertainty on Flux_mag
Flux_Jy D (Nim) Photometry in Jy
err_Flux_Jy D (Nim) Uncertainty on Flux_Jy
Flux_erg D (Nim) Photometry in erg/cm2/s/A
err_Flux_erg D (Nim) Uncertainty on Flux_erg
Contrast D (Nim) Contrast from the star
err_Contrast D (Nim) Uncertainty on the Contrast

Code examples

The following sections contains simple code exemples in Python and IDL to read and display some of the content of the DIVA HLSP. You can download the sample codes with a DIVA test file.

Python

import matplotlib.pyplot as plt
import numpy as np

from astropy.io import fits

#
#
hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits'

hdu = fits.open(hlsp_file)

#
# list extensions
#
nextend = hdr['NEXTEND']

print()
print('File has {0} extensions:'.format(nextend))
extensions = {}
for i in range(1, nextend+1):
extname = hdr['EXT{0}NAME'.format(i)]
exttype = hdr['EXT{0}TYPE'.format(i)]

print(' * {0}; {1}'.format(extname, exttype))

# save for later
extensions[extname] = i

# retrieve basic information
surv   = hdr['SURVNAME']
survpi = hdr['SURVPI']
instru = hdr['INSTRUME']
filt   = hdr['FILTER']
target = hdr['TARGNAME']

#
# data information
#
print()
print('Data information:')

idx = extensions.get('DATA_INFORMATION')
data_info = hdu[idx].columns

# list columns
print(data_info)

# retrieve obs information
date = data_info['UT_Midpoint_Date_of_Observation'].array[0]
Texp = data_info['Exposure_Time'].array[0]
FoV_rot = data_info['Combined_rotation_angle'].array[0]

#
# reduced image
#
idx = extensions.get('REDUCED_DATA')
if (idx is not None):
img = hdu[idx].data

# display
fig = plt.figure(0, figsize=(10, 10))
plt.clf()
plt.imshow(img, vmin=-0.005*np.nanmax(img), vmax=0.005*np.nanmax(img), aspect='equal')
plt.figtext(0.07, 0.9, 'Texp = {0:.1f} min \nFoV rot = {1:.1f} deg'.format(Texp/60, FoV_rot), fontsize=15)
plt.title('{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi))
plt.tight_layout()

#
# detection limit
#
idx = extensions.get('DETECTION_LIMIT')
if (idx is not None):
dat = hdu[idx].data.columns

# threshold

# values
dlim = dat['Detection_limit'].array

# units
dlim_u = dat['Detection_limit'].unit

# plot
fig = plt.figure(1, figsize=(10, 7))
plt.clf()
plt.plot(sep, dlim, linestyle='-', color='b')
plt.gca().invert_yaxis()
plt.xlabel(r'Angular separation [{0}]'.format(sep_u))
plt.ylabel(r'Sensitivity limit @ {0:.0f}$\sigma$ [{1}]'.format(nsigma, dlim_u))
plt.title(r'{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi))
plt.tight_layout()

#
# candidates
#
print()
print('Detected candidates:')

candnum = hdr['CANDNUM']
if (candnum == -1):
print(' * this survey has no information on detected candidates')
elif (candnum > 0):
idx = extensions.get('SOURCE_DETECTION')
dat = hdu[idx].data.columns

# list properties
print('idx  dRA        err_dRA   dDEC      err_dDEC   dmag    err_dmag')
print('---------------------------------------------------------------')
for c in range(candnum):
print('{:2d}{:10.4f}{:10.4f}{:10.4f}{:10.4f}{:10.2f}{:7.2f}'.
format(c, dat['dRA'].array[c], dat['err_dRA'].array[c],
dat['dDEC'].array[c], dat['err_dDEC'].array[c],
dat['Contrast'].array[c], dat['err_Contrast'].array[c]))


IDL

;;
;;
hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits'

hdu = mrdfits(hlsp_file, 0, hdr, /silent)

;;
;; list extensions
;;
nextend = sxpar(hdr, 'NEXTEND')

print
print,'File has '+string(nextend, format='(I0)')+' extensions:'
extensions = ['']
for i=1,nextend do begin
extname = sxpar(hdr, string(i, format='("EXT",I0,"NAME")'))
exttype = sxpar(hdr, string(i, format='("EXT",I0,"TYPE")'))

print,' * '+extname+'; '+exttype

;; save for later
extensions = [extensions, extname]
endfor

;; retrieve basic information
surv   = strtrim(sxpar(hdr, 'SURVNAME'))
survpi = strtrim(sxpar(hdr, 'SURVPI'))
instru = strtrim(sxpar(hdr, 'INSTRUME'))
filt   = strtrim(sxpar(hdr, 'FILTER'))
target = strtrim(sxpar(hdr, 'TARGNAME'))

;;
;; data information
;;
print
print,'Data information:'

idx = where(extensions eq 'DATA_INFORMATION')
data_info = mrdfits(hlsp_file, idx[0], /silent)

;; list colmuns
help,/str,data_info

;; retrieve obs information
date = data_info.UT_Midpoint_Date_of_Observation[0]
Texp = data_info.Exposure_Time[0]
FoV_rot = data_info.Combined_rotation_angle[0]

;;
;; reduced image
;;
idx = where(extensions eq 'REDUCED_DATA')
if (idx ne -1) then begin
img = mrdfits(hlsp_file, idx[0], /silent)

img[where(finite(img) eq 0)] = 0
img <=  0.005*max(img)
img >= -0.005*max(img)

window, 0, xs=600, ys=600
tvscl, congrid(img, 600, 600)
xyouts, 10, 10, target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')', $color=255, /device, size=1.8 endif ;; ;; detection limit ;; idx = where(extensions eq 'DETECTION_LIMIT') if (idx ne -1) then begin dat = mrdfits(hlsp_file, idx[0], chdr, /silent) ;; threshold nsigma = sxpar(chdr, 'NSIGMA') ;; values sep = dat.Radius dlim = dat.Detection_limit ;; plot window, 1, xs=640, ys=480 plot,sep,dlim,yr=[max(dlim, /nan), min(dlim, /nan)], xtitle='Angular separation',$
ytitle='Sensitivity limite @ '+string(nsigma, format='(I0)')+'-sigma', $title=target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')' endif ;; ;; candidates ;; print print,'Detected candidates:' candnum = sxpar(hdr, 'CANDNUM') if (candnum eq -1) then begin print,' * this survey has no information on detected candidates' endif else if (candnum gt 0) then begin idx = where(extensions eq 'SOURCE_DETECTION') dat = mrdfits(hlsp_file, idx[0], /silent) ;; list properties print,'idx dRA err_dRA dDEC err_dDEC dmag err_dmag' print,'---------------------------------------------------------------' for c=0,candnum-1 do begin print, c, dat[c].dRA, dat[c].err_dRA, dat[c].dDEC, dat[c].err_dDEC,$
dat[c].Contrast, dat[c].err_Contrast,format='(I2,F10.4,F10.4,F10.4,F10.4,F10.2,F7.2)'
endfor

endif

end