ESA JWST Archive (astroquery.esa.jwst)

The James Webb Space Telescope (JWST) is a collaborative project between NASA, ESA, and the Canadian Space Agency (CSA). Although radically different in design, and emphasizing the infrared part of the electromagnetic spectrum, JWST is widely seen as the successor to the Hubble Space Telescope (HST). The JWST observatory consist of a deployable 6.6 meter passively cooled telescope optimized for infrared wavelengths, and is operated in deep space at the anti-Sun Earth-Sun Lagrangian point (L2). It carries four scientific instruments: a near-infrared camera (NIRCam), a near-infrared multi-object spectrograph (NIRSpec) covering the 0.6 - 5 μm spectral region, a near-infrared slit-less spectrograph (NIRISS), and a combined mid-infrared camera/spectrograph (MIRI) covering 5 - 28 μm. The JWST focal plane (see image to the right) contains apertures for the science instruments and the Fine Guidance Sensor (FGS).

The scientific goals of the JWST mission can be sorted into four broad themes: The birth of stars and proto-planetary systems Planetary systems and the origins of life

  • The end of the dark ages: first light and re-ionization.

  • The assembly of galaxies.

  • The birth of stars and proto-planetary systems.

  • Planetary systems and the origins of life.

This package provides access to the metadata and datasets provided by the European Space Agency JWST Archive using a TAP+ REST service. TAP+ is an extension of Table Access Protocol (TAP: http://www.ivoa.net/documents/TAP/) specified by the International Virtual Observatory Alliance (IVOA: http://www.ivoa.net).

The TAP query language is Astronomical Data Query Language (ADQL: http://www.ivoa.net/documents/ADQL/2.0), which is similar to Structured Query Language (SQL), widely used to query databases.

TAP provides two operation modes: Synchronous and Asynchronous:

  • Synchronous: the response to the request will be generated as soon as the request received by the server. (Do not use this method for queries that generate a big amount of results.)

  • Asynchronous: the server starts a job that will execute the request. The first response to the request is the required information (a link) to obtain the job status. Once the job is finished, the results can be retrieved.

This module can use these two modes, usinc the ‘async_job=False/True’ tag in different functions.

ESA JWST TAP+ server provides two access mode: public and authenticated:

  • Public: this is the standard TAP access. A user can execute ADQL queries and upload tables to be used in a query ‘on-the-fly’ (these tables will be removed once the query is executed). The results are available to any other user and they will remain in the server for a limited space of time.

  • Authenticated: some functionalities are restricted to authenticated users only. The results are saved in a private user space and they will remain in the server for ever (they can be removed by the user).

    • ADQL queries and results are saved in a user private area.

    • Cross-match operations: a catalog cross-match operation can be executed. Cross-match operations results are saved in a user private area.

    • Persistence of uploaded tables: a user can upload a table in a private space. These tables can be used in queries as well as in cross-matches operations.

This python module provides an Astroquery API access.

Examples

It is highly recommended checking the status of JWST TAP before executing this module. To do this:

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.get_status_messages()

This method will retrieve the same warning messages shown in JWST Science Archive with information about service degradation.

1. Non authenticated access

1.1. Query region

>>> import astropy.units as u
>>> from astropy.coordinates import SkyCoord
>>> from astroquery.esa.jwst import Jwst
>>>
>>> coord = SkyCoord(ra=53, dec=-27, unit=(u.degree, u.degree), frame='icrs')
>>> width = u.Quantity(5, u.deg)
>>> height = u.Quantity(5, u.deg)
>>> result = Jwst.query_region(coordinate=coord, width=width, height=height)
>>> result  
         dist                observationid          ...
  ------------------ ------------------------------ ...
  0.5520678701664351 jw02516010001_xx107_00007_miri ...
  0.5520678701664351 jw02516010001_xx104_00004_miri ...
  0.5520678701664351 jw02516010001_xx102_00002_miri ...
  0.5520678701664351 jw02516010001_xx10b_00011_miri ...
  0.5520678701664351 jw02516010001_xx109_00009_miri ...
  0.5520678701664351 jw02516010001_xx105_00005_miri ...
  0.5520678701664351 jw02516010001_xx103_00003_miri ...
  0.5520678701664351 jw02516010001_xx10a_00010_miri ...
  0.5520678701664351 jw02516010001_xx101_00001_miri ...
  0.5520678701664351 jw02516010001_xx106_00006_miri ...
  ...                ...                            ...

1.3. Query by target name

To provide the target coordinates based on its name and execute the query region method. It uses three different catalogs to resolve the coordinates: SIMBAD, NED and VIZIER. An additional target resolver is provider, ALL (which is also the default value), using all the aforementioned catalogues in the defined order to obtain the required coordinates (using the following element in the list if the target name cannot be resolved).

>>> from astroquery.esa.jwst import Jwst
>>> import astropy.units as u
>>>
>>> target_name = 'M1'
>>> target_resolver = 'ALL'
>>> radius = u.Quantity(5, u.deg)
>>> result = Jwst.query_target(target_name=target_name, target_resolver=target_resolver, radius=radius)
>>> result  
          dist                  observationid           ...
  -------------------- -------------------------------- ...
  0.003349189664076155   jw01714001004_xx106_00002_miri ...
  0.003349189664076155   jw01714001003_xx10q_00002_miri ...
  0.003349189664076155   jw01714001006_xx107_00003_miri ...
  0.003349189664076155   jw01714001002_xx105_00001_miri ...
  0.003349189664076155 jw01714005001_xx106_00006_nircam ...
  0.003349189664076155   jw01714001003_xx10w_00004_miri ...
  0.003349189664076155   jw01714001002_xx103_00003_miri ...
  0.003349189664076155   jw01714001004_xx104_00004_miri ...
  0.003349189664076155   jw01714001006_xx10n_00003_miri ...
  0.003349189664076155   jw01714001005_xx10p_00001_miri ...
  ...                    ...                            ...

This method uses the same parameters as query region, but also includes the target name and the catalogue (target resolver) to retrieve the coordinates.

>>> from astroquery.esa.jwst import Jwst
>>> import astropy.units as u
>>>
>>> target_name = 'LMC'
>>> target_resolver = 'NED'
>>> width = u.Quantity(5, u.deg)
>>> height = u.Quantity(5, u.deg)
>>> result = Jwst.query_target(target_name=target_name, target_resolver=target_resolver,
...                            width=width, height=height, async_job=True)
INFO: Query finished. [astroquery.utils.tap.core]
>>> result  
          dist                  observationid            ...
  ------------------- ---------------------------------- ...
  0.25984680687093176 jw01043010001_02101_00013_mirimage ...
  0.25984680687093176 jw01043010001_02101_00004_mirimage ...
  0.25984680687093176 jw01043010001_02101_00014_mirimage ...
  0.25984680687093176 jw01043007001_02101_00006_mirimage ...
  0.25984680687093176 jw01043008001_02101_00016_mirimage ...
  0.25984680687093176 jw01043007001_02101_00008_mirimage ...
  0.25984680687093176 jw01043009001_02101_00013_mirimage ...
  0.25984680687093176 jw01043007001_02101_00007_mirimage ...
  0.25984680687093176 jw01043007001_02101_00009_mirimage ...
  0.25984680687093176 jw01043009001_02101_00004_mirimage ...
  ...                 ...                                ...

1.4 Getting data products

To query the data products associated with a certain Observation ID

>>> from astroquery.esa.jwst import Jwst
>>> product_list = Jwst.get_product_list(observation_id='jw01063107001_02101_00013_nrca3')
>>> print(product_list['filename'])  
                      filename
  ------------------------------------------------
  jw01063-o107_20220328t013707_image2_144_asn.json
          jw01063107001_02101_00013_nrca3_cal.fits
           jw01063107001_02101_00013_nrca3_cal.jpg
     jw01063107001_02101_00013_nrca3_cal_thumb.jpg
          jw01063107001_02101_00013_nrca3_i2d.fits
  ...

You can filter by product type and calibration level (using a numerical value or the option ‘ALL’ -set by default- that will download all the products associated to this observation_id with the same and lower levels).

>>> from astroquery.esa.jwst import Jwst
>>> product_list = Jwst.get_product_list(observation_id='jw01023029001_02101_00004_mirimage', product_type='science')
>>> print(product_list['filename'])
                       filename
  -------------------------------------------------
  jw01023029001_02101_00004_mirimage_c1006_crf.fits
        jw01023029001_02101_00004_mirimage_cal.fits
        jw01023029001_02101_00004_mirimage_i2d.fits
   jw01023029001_02101_00004_mirimage_o029_crf.fits
       jw01023029001_02101_00004_mirimage_rate.fits
  ...

To download a data product

>>> from astroquery.esa.jwst import Jwst
>>> query = """select o.observationid, a.artifactid, a.filename
... from jwst.observation o join jwst.artifact a on a.obsid = o.obsid
... where o.proposal_id = '01166' and o.intent = 'science'"""
>>> job = Jwst.launch_job(query, async_job=True)
INFO: Query finished. [astroquery.utils.tap.core]
>>> results = job.get_results()
>>> results  
           observationid                       artifactid                                    filename
  ------------------------------- ------------------------------------ ----------------------------------------------------
  jw01166091001_02102_00002_nrca3 6ab73824-6587-4bca-84a8-eb48ac7251be jw01166-o091_20220505t044658_wfs-image2_058_asn.json
  jw01166091001_02102_00002_nrca3 c999249f-c554-4a5f-bd6a-099f1fe48864             jw01166091001_02102_00002_nrca3_cal.fits
  jw01166091001_02102_00002_nrca3 004433de-a4b3-4dc6-9177-0f9d78a97bda              jw01166091001_02102_00002_nrca3_cal.jpg
  jw01166091001_02102_00002_nrca3 5b89920d-532d-43be-b70b-973c4bfdfdcc        jw01166091001_02102_00002_nrca3_cal_thumb.jpg
  jw01166091001_02102_00002_nrca3 5a86e351-3066-4c46-8698-c49aade6ec98            jw01166091001_02102_00002_nrca3_rate.fits
  ...                             ...                                  ...
>>> output_file = Jwst.get_product(artifact_id='6ab73824-6587-4bca-84a8-eb48ac7251be')  
>>> output_file = Jwst.get_product(file_name='jw01166091001_02102_00002_nrca3_cal.fits')  

To download products by observation identifier, it is possible to use the get_obs_products function, with the same parameters than get_product_list.

>>> from astroquery.esa.jwst import Jwst
>>> observation_id = 'jw01122001001_0210r_00001_nrs2'
>>> results = Jwst.get_obs_products(observation_id=observation_id, cal_level=2, product_type='science')  

A temporary directory is created with the files and a list of the them is provided.

When more than one product is found, a tar file is retrieved. This method extracts the products.

This method is only intended to download the products with the same calibration level or below. If an upper level is requested:

ValueError: Requesting upper levels is not allowed

If proprietary data is requested and the user has not logged in:

403 Error 403:
Private file(s) requested: MAST token required for authentication.

It is also possible to extract the products associated to an observation with upper calibration levels with get_related_observations. Using the observation ID as input parameter, this function will retrieve the observations (IDs) that use it to create a composite observation.

>>> from astroquery.esa.jwst import Jwst
>>> observation_id = 'jw02739-o001_t001_nircam_clear-f444w'
>>> results = Jwst.get_related_observations(observation_id=observation_id)
>>> results[0:5]
['jw02739001001_02105_00001_nrcalong',
 'jw02739001001_02105_00001_nrcblong',
 'jw02739001001_02105_00002_nrcalong',
 'jw02739001001_02105_00002_nrcblong',
 'jw02739001001_02105_00003_nrcalong']

1.5 Getting public tables

To load only table names (TAP+ capability)

>>> from astroquery.esa.jwst import Jwst
>>> tables = Jwst.load_tables(only_names=True)
INFO: Retrieving tables... [astroquery.utils.tap.core]
INFO: Parsing tables... [astroquery.utils.tap.core]
INFO: Done. [astroquery.utils.tap.core]
>>> print(*(table.name for table in tables), sep="\n")
  ivoa.obscore
  jwst.archive
  jwst.artifact
  jwst.chunk
  jwst.harvestskipuri
  jwst.main
  jwst.moc
  jwst.observation
  ...

To load table names (TAP compatible)

>>> from astroquery.esa.jwst import Jwst
>>> tables = Jwst.load_tables()
INFO: Retrieving tables... [astroquery.utils.tap.core]
INFO: Parsing tables... [astroquery.utils.tap.core]
INFO: Done. [astroquery.utils.tap.core]
>>> print(*(table.name for table in tables), sep="\n")
  ivoa.obscore
  jwst.archive
  jwst.artifact
  jwst.chunk
  jwst.harvestskipuri
  jwst.main
  ...

To load only a table (TAP+ capability)

>>> from astroquery.esa.jwst import Jwst
>>> table = Jwst.load_table('jwst.main')
>>> print(table) 
TAP Table name: jwst.jwst.main
Description: None
Num. columns: 109

Once a table is loaded, columns can be inspected

>>> from astroquery.esa.jwst import Jwst
>>> table = Jwst.load_table('jwst.main')
>>> print(*(column.name for column in table.columns), sep="\n") 
"public"
algorithm_name
calibrationlevel
collection
creatorid
dataproducttype
energy_bandpassname
...

1.6 Synchronous query

A synchronous query will not store the results at server side. These queries must be used when the amount of data to be retrieve is ‘small’.

There is a limit of 2000 rows. If you need more than that, you must use asynchronous queries.

The results can be saved in memory (default) or in a file.

Query without saving results in a file:

>>> from astroquery.esa.jwst import Jwst
>>>
>>> job = Jwst.launch_job("SELECT TOP 100 "
...       "instrument_name, proposal_id, calibrationlevel, "
...       "dataproducttype "
...       "FROM jwst.main ORDER BY instrument_name, observationuri")
>>>
>>> print(job)  
<Table length=100>
      name       dtype
---------------- ------
 instrument_name object
     proposal_id object
calibrationlevel  int32
 dataproducttype object
Jobid: None
Phase: COMPLETED
Owner: None
Output file: 1661441953031O-result.vot.gz
Results: None
>>> result = job.get_results()
>>> result  
<Table length=100>
instrument_name proposal_id calibrationlevel dataproducttype
     object        object        int32            object
--------------- ----------- ---------------- ---------------
            FGS        1014                3           image
            FGS        1014                3           image
            FGS        1014                3           image
            FGS        1014                3           image
            FGS        1014                3           image
            FGS        1014                2           image
            FGS        1014                1           image
            FGS        1014                1           image
            FGS        1014                2           image
            FGS        1014                2           image
            ...         ...              ...             ...
            FGS        1017                2           image
            FGS        1017                1           image
            FGS        1017                2           image
            FGS        1017                1           image
            FGS        1017                2           image
            FGS        1017                1           image
            FGS        1017                2           image
            FGS        1017                1           image
            FGS        1017                2           image
            FGS        1017                1           image
            FGS        1017                2           image

Query saving results in a file:

>>> from astroquery.esa.jwst import Jwst
>>> job = Jwst.launch_job("SELECT TOP 100 "
...                       "instrument_name, proposal_id, calibrationlevel, "
...                       "dataproducttype "
...                       "FROM jwst.main ORDER BY instrument_name, observationuri",
...                       dump_to_file=True)
>>>
>>> print(job)  
Jobid: None
Phase: COMPLETED
Owner: None
Output file: 1655978085454O-result.vot.gz
Results: None
>>> result = job.get_results()
>>> result  
  <Table length=100>
  instrument_name proposal_id calibrationlevel dataproducttype
       object        object        int32            object
  --------------- ----------- ---------------- ---------------
              FGS       01014                3           image
              FGS       01014                3           image
              FGS       01014                3           image
              FGS       01014                3           image
              FGS       01014                3           image
              FGS       01014                3           image
              FGS       01014                2           image
  ...             ...         ...              ...

1.7 Synchronous query on an ‘on-the-fly’ uploaded table

A table can be uploaded to the server in order to be used in a query.

>>> from astroquery.esa.jwst import Jwst
>>> upload_resource = 'mytable.xml.gz'
>>> j = Jwst.launch_job(query="SELECT * from tap_upload.table_test",
...                     upload_resource=upload_resource,
...                     upload_table_name="table_test", verbose=True)
>>> result = j.get_results()
>>> result.pprint()
source_id alpha delta
--------- ----- -----
        a   1.0   2.0
        b   3.0   4.0
        c   5.0   6.0

1.8 Asynchronous query

Asynchronous queries save results at server side. These queries can be accessed at any time. For anonymous users, results are kept for three days.

The results can be saved in memory (default) or in a file.

Query without saving results in a file:

>>> from astroquery.esa.jwst import Jwst
>>> job = Jwst.launch_job("select top 100 * from jwst.main", async_job=True)
INFO: Query finished. [astroquery.utils.tap.core]
>>> print(job)  
Jobid: 1542383562372I
Phase: COMPLETED
Owner: None
Output file: async_20181116165244.vot
Results: None
>>> r = job.get_results()
>>> r['observationid']  
<MaskedColumn name='observationid' dtype='object' length=100>
     jw01070001002_04101_00003_nrca1
     jw01070001002_04101_00003_nrca1
   jw01286007001_xx10k_00001_nirspec
      jw01290023001_xx10b_00001_miri
    jw01568006004_xx101_00001_nircam
                                 ...

Query saving results in a file:

>>> from astroquery.esa.jwst import Jwst
>>> job = Jwst.launch_job("select top 100 * from jwst.main", dump_to_file=True)
>>> print(job)  
Jobid: None
Phase: COMPLETED
Owner: None
Output file: 1635853688471D-result.vot.gz
Results: None
>>> r = job.get_results()
>>> r['instrument_name']  
<MaskedColumn name='instrument_name' dtype='object' length=100>
        NIRCAM
        NIRCAM
    NIRISS/MSA
      MIRI/IFU
  NIRCAM/IMAGE
          MIRI
           ...

1.9 Asynchronous job removal

To remove asynchronous

>>> from astroquery.esa.jwst import Jwst
>>> job = Jwst.remove_jobs(["job_id_1", "job_id_2", ...])

2. Authenticated access

Authenticated users are able to access to TAP+ capabilities (shared tables, persistent jobs, etc.) In order to authenticate a user, login method must be called. After a successful authentication, the user will be authenticated until logout method is called.

All previous methods (query_object, cone_search, load_table, load_tables, launch_job) explained for non authenticated users are applicable for authenticated ones.

The main differences are:

  • Asynchronous results are kept at server side for ever (until the user decides to remove one of them).

  • Users can access to shared tables.

  • It is also possible to set a token after logging using set_token function.

2.1. Login/Logout

Using the command line:

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.login(user='userName', password='userPassword')

It is possible to use a file where the credentials are stored:

The file must containing user and password in two different lines.

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.login(credentials_file='my_credentials_file')

MAST tokens can also be used in command line functions:

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.login(user='userName', password='userPassword', token='mastToken')

If the user is logged in and a MAST token has not been included or must be changed, it can be specified using the set_token function.

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.login(user='userName', password='userPassword')
>>> Jwst.set_token(token='mastToken')

To perform a logout:

>>> from astroquery.esa.jwst import Jwst
>>> Jwst.logout()

2.2. Listing shared tables

>>> from astroquery.esa.jwst import Jwst
>>> tables = Jwst.load_tables(only_names=True, include_shared_tables=True)
INFO: Retrieving tables... [astroquery.utils.tap.core]
INFO: Parsing tables... [astroquery.utils.tap.core]
INFO: Done. [astroquery.utils.tap.core]
>>> print(*(table.name for table in tables), sep="\n")  
public.dual
tap_schema.columns
tap_schema.key_columns
tap_schema.keys
tap_schema.schemas
tap_schema.tables
jwst.artifact
jwst.chunk
jwst.main
jwst.observation
jwst.observationmember
jwst.part
jwst.plane
jwst.plane_inputs
...
user_schema_1.table1
user_schema_2.table1
...
Reference/API
astroquery.esa.jwst Package

European Space Astronomy Centre (ESAC) European Space Agency (ESA)

Classes

JwstClass(*[, tap_plus_handler, ...])

Proxy class to default TapPlus object (pointing to JWST Archive)

JwstDataHandler([base_url])

Conf()

Configuration parameters for astroquery.esa.jwst.