Introduction to HCI-FITS

A complete description of the data format can be found here and exemple code to read it is provided below.

The format of the DIVA high-level science products (HLSP) follows the specification of the format for high-contrast imaging that was proposed by the ALICE project. The goal of this format is to simplify HLSP exchange in the HCI community by adopting a standard output format for all the usual data products. The format is based on a single multi-extension FITS file (Pence et al.2010) that gathers all the high-level information about the datasets, which we call HCI-FITS file for High-Contrast Imaging FITS file. In the following we introduce the main specifications of Version 1 of HCI-FITS. This approach is largely inspired from the OI-FITS format, which is the standard for calibrated data exchange from optical interferometers (Pauls et al. 2004, 2005).

A single FITS file — To enable high-level science analyses of high-contrast imaging data, several products are mandatory (e.g. the reduced image, detection limits, source detections). In order to prevent information loss when exchanging data, all the information must be gathered in a single file. The FITS file format is ideal for this task since it enables the encapsulation of heterogeneous data types in independent extensions. This includes images (IMAGE extension type), which is appropriate for the reduced images, sensitivity and SNR maps, and multi-dimension tables (BINTABLE extension type), which is appropriate for radial detection limits, characteristics of potentially detected sources, or general characteristics of the observations and reduced products.

A flexible standard — The HCI-FITS format is developed to be compatible with any type of dataset, regardless of the instrument, or observing mode, or processing method used to obtain the final reduced products. It can be used for both ground-based or space observations, for coronagraphic and saturated imaging, for broad-band imaging, integral-field spectroscopy, and polarimetric imaging. In addition, a HCI-FITS file may contain products for either a single image or an image cube, and keep track of each specific images characteristics.

Structure of the HCI-FITS format — There are 5 different types of data products that are necessary for a high-level astrophysical exploitation of HCI results: the reduced images, the SNR maps, the sensitivity maps (sometimes refered to as "noise" maps), the radial detection limits (or "contrast curves"), and the properties of detected point sources. A complete HCI-FITS file is therefore composed of 6 extensions, corresponding to the above products plus an extension containing the characteristics of each image in the file. The extensions may appear in any order, but they must implement the EXTNAME keyword that enables their identification. The primary header of any HCI-FITS file also implements a list of the available extensions and their type, with keywords in the form of EXT?NAME and EXT?TYPE, where ? represents the corresponding extension number. The extensions, their format and dimensions are detailed in Table 1. The SOURCE_DETECTION is optional but must follow the specified format if present. Other optional extensions may be included, e.g. for intermediate products like unocculted/unsaturated PSF images.

Table 1: Structure of the HCI-FITS files
Label Type Dimensions Description
DATA_INFORMATION BINTABLE Nim x 12 Characteristics of each image
REDUCED_DATA IMAGE Nim x Nx x Ny Reduced data product
SNR_MAP IMAGE Nim x Nx x Ny SNR maps
SENSITIVITY_MAP IMAGE Nim x Nx x Ny 2D detection limits
DETECTION_LIMIT BINTABLE Nim x 2 x Nr Radial detection limits
SOURCE_DETECTION BINTABLE Nsource x 20 x Nim Detection point sources properties

Description of the extensions

  • DATA_INFORMATION: this extension of type BINTABLE identifies the characteristics of each reduced image in the HCI-FITS file. The structure of this extension is flexible enough to allow any collection of images. It is composed of 12 fields that track the orientation and all the main characteristics of the images contained in the file. These fields are described in Table 2.

Table 2: DATA_INFORMATION extension
Label Type Description
Image_Number I Index of the image in the cube
Orientation D Sky-orientation of the image vertical axis
Combined_rotation_angle D Parallactic angle combined in the image
Number_of_Exposures I Number of exposures combined in the image
Exposure_Time D Total exposure time combined in the image
Observation_Start D Mod. Julian Date at the start of the exposure
Observation_End D Mod. Julian Date at the end of the exposure
UT_Midpoint_Date_of_Observation A UT date at the image mid-point
UT_Midpoint_Time_of_Observation A UT time at the image mid-point
Wavelength D Effective wavelength of the image
Bandwidth D Effective bandwidth of the image
Polarization A Polarization state of the image

  • REDUCED_DATA: this extension of type IMAGE contains the reduced images of the HCI-FITS file. If several images are provided, the order must be the same as in the data provided in the DATA_INFORMATION table.

  • SNR_MAP: this extension of type IMAGE contains the SNR maps corresponding to the reduced images.

  • SENSITIVITY_MAP: this extension of type IMAGE contains the 2D sensitivity maps of the observations. The header of this extension must contain the NSIGMA keyword to indicate the confidence level of the sensitivity map.

  • DETECTION_LIMIT: this extension of type BINTABLE contains the radial point source detection limits for each image present in the file. The header of this extension must also contain the NSIGMA keyword to indicate the confidence level of the detection limit. The format of this table is described in Table 3.

Table 3: DETECTION_LIMIT extension
Label Type Description
Radius D (Nim) Radial separation from the star
Detection_Limit D (Nim) Point source detection limit

  • SOURCE_DETECTION: this extension of type BINTABLE contains the properties of all the point sources detected in each of the images. This table has 20 fields, which are described in Table 4.

Table 4: SOURCE_DETECTION extension
Label Type Description
Candidate I Index of the detected source
SNR D (Nim) SNR of the source
dRA D (Nim) Relative R.A. from the star
err_dRA D (Nim) Uncertainty on the dRA
dDEC D (Nim) Relative Declination from the star
err_dDEC D (Nim) Uncertainty on the dDEC
Sep D (Nim) Separation from the star
err_Sep D (Nim) Uncertainty on the separation
PA D (Nim) Position Angle (east of north)
err_PA D (Nim) Uncertainty on the PA
Flux_cs D (Nim) Photometry in count/s
err_Flux_cs D (Nim) Uncertainty on Flux_cs
Flux_mag D (Nim) Photometry in magnitude
err_Flux_mag D (Nim) Uncertainty on Flux_mag
Flux_Jy D (Nim) Photometry in Jy
err_Flux_Jy D (Nim) Uncertainty on Flux_Jy
Flux_erg D (Nim) Photometry in erg/cm2/s/A
err_Flux_erg D (Nim) Uncertainty on Flux_erg
Contrast D (Nim) Contrast from the star
err_Contrast D (Nim) Uncertainty on the Contrast

Code examples

The following sections contains simple code exemples in Python and IDL to read and display some of the content of the DIVA HLSP. You can download the sample codes with a DIVA test file.


import matplotlib.pyplot as plt
import numpy as np

from import fits

# read HLSP
hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits'

hdu =
hdr = hdu[0].header

# list extensions
nextend = hdr['NEXTEND']

print('File has {0} extensions:'.format(nextend))
extensions = {}
for i in range(1, nextend+1):
    extname = hdr['EXT{0}NAME'.format(i)]
    exttype = hdr['EXT{0}TYPE'.format(i)]

    print(' * {0}; {1}'.format(extname, exttype))

    # save for later
    extensions[extname] = i

# retrieve basic information
surv   = hdr['SURVNAME']
survpi = hdr['SURVPI']
instru = hdr['INSTRUME']
filt   = hdr['FILTER']
target = hdr['TARGNAME']

# data information
print('Data information:')

idx = extensions.get('DATA_INFORMATION')
data_info = hdu[idx].columns

# list columns

# retrieve obs information
date = data_info['UT_Midpoint_Date_of_Observation'].array[0]
Texp = data_info['Exposure_Time'].array[0]
FoV_rot = data_info['Combined_rotation_angle'].array[0]

# reduced image
idx = extensions.get('REDUCED_DATA')
if (idx is not None):
    img = hdu[idx].data

    # display
    fig = plt.figure(0, figsize=(10, 10))
    plt.imshow(img, vmin=-0.005*np.nanmax(img), vmax=0.005*np.nanmax(img), aspect='equal')
    plt.figtext(0.07, 0.9, 'Texp = {0:.1f} min \nFoV rot = {1:.1f} deg'.format(Texp/60, FoV_rot), fontsize=15)
    plt.title('{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi))

# detection limit
idx = extensions.get('DETECTION_LIMIT')
if (idx is not None):
    dat = hdu[idx].data.columns

    # threshold
    nsigma = hdu[idx].header['NSIGMA']

    # values
    sep  = dat['Radius'].array
    dlim = dat['Detection_limit'].array

    # units
    sep_u  = dat['Radius'].unit
    dlim_u = dat['Detection_limit'].unit

    # plot
    fig = plt.figure(1, figsize=(10, 7))
    plt.plot(sep, dlim, linestyle='-', color='b')
    plt.xlabel(r'Angular separation [{0}]'.format(sep_u))
    plt.ylabel(r'Sensitivity limit @ {0:.0f}$\sigma$ [{1}]'.format(nsigma, dlim_u))
    plt.title(r'{0} / {1} {2} / {3} / {4} ({5})'.format(target, instru, filt, date, surv, survpi))

# candidates
print('Detected candidates:')

candnum = hdr['CANDNUM']
if (candnum == -1):
    print(' * this survey has no information on detected candidates')
elif (candnum > 0):
    idx = extensions.get('SOURCE_DETECTION')
    dat = hdu[idx].data.columns

    # list properties
    print('idx  dRA        err_dRA   dDEC      err_dDEC   dmag    err_dmag')
    for c in range(candnum):
              format(c, dat['dRA'].array[c], dat['err_dRA'].array[c],
                        dat['dDEC'].array[c], dat['err_dDEC'].array[c],
                        dat['Contrast'].array[c], dat['err_Contrast'].array[c]))


;; read HLSP
hlsp_file = 'TWA_21_2010-02-18_NaCo_H_diva_hlsp.fits'

hdu = mrdfits(hlsp_file, 0, hdr, /silent)

;; list extensions
nextend = sxpar(hdr, 'NEXTEND')

print,'File has '+string(nextend, format='(I0)')+' extensions:'
extensions = ['']
for i=1,nextend do begin
   extname = sxpar(hdr, string(i, format='("EXT",I0,"NAME")'))
   exttype = sxpar(hdr, string(i, format='("EXT",I0,"TYPE")'))

   print,' * '+extname+'; '+exttype

   ;; save for later
   extensions = [extensions, extname]

;; retrieve basic information
surv   = strtrim(sxpar(hdr, 'SURVNAME'))
survpi = strtrim(sxpar(hdr, 'SURVPI'))
instru = strtrim(sxpar(hdr, 'INSTRUME'))
filt   = strtrim(sxpar(hdr, 'FILTER'))
target = strtrim(sxpar(hdr, 'TARGNAME'))

;; data information
print,'Data information:'

idx = where(extensions eq 'DATA_INFORMATION')
data_info = mrdfits(hlsp_file, idx[0], /silent)

;; list colmuns

;; retrieve obs information
date = data_info.UT_Midpoint_Date_of_Observation[0]
Texp = data_info.Exposure_Time[0]
FoV_rot = data_info.Combined_rotation_angle[0]

;; reduced image
idx = where(extensions eq 'REDUCED_DATA')
if (idx ne -1) then begin
   img = mrdfits(hlsp_file, idx[0], /silent)

   img[where(finite(img) eq 0)] = 0
   img <=  0.005*max(img)
   img >= -0.005*max(img)

   window, 0, xs=600, ys=600
   tvscl, congrid(img, 600, 600)
   xyouts, 10, 10, target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')', $
           color=255, /device, size=1.8

;; detection limit
idx = where(extensions eq 'DETECTION_LIMIT')
if (idx ne -1) then begin
   dat = mrdfits(hlsp_file, idx[0], chdr, /silent)

   ;; threshold
   nsigma = sxpar(chdr, 'NSIGMA')

   ;; values
   sep  = dat.Radius
   dlim = dat.Detection_limit

   ;; plot
   window, 1, xs=640, ys=480
   plot,sep,dlim,yr=[max(dlim, /nan), min(dlim, /nan)], xtitle='Angular separation', $
        ytitle='Sensitivity limite @ '+string(nsigma, format='(I0)')+'-sigma', $
        title=target+' / '+instru+' '+filt+' / '+date+' / '+surv+' ('+survpi+')'

;; candidates
print,'Detected candidates:'

candnum = sxpar(hdr, 'CANDNUM')
if (candnum eq -1) then begin
   print,' * this survey has no information on detected candidates'
endif else if (candnum gt 0) then begin
   idx = where(extensions eq 'SOURCE_DETECTION')
   dat = mrdfits(hlsp_file, idx[0], /silent)

   ;; list properties
   print,'idx  dRA        err_dRA   dDEC      err_dDEC   dmag    err_dmag'
   for c=0,candnum-1 do begin
      print, c, dat[c].dRA, dat[c].err_dRA, dat[c].dDEC, dat[c].err_dDEC, $
             dat[c].Contrast, dat[c].err_Contrast,format='(I2,F10.4,F10.4,F10.4,F10.4,F10.2,F7.2)'