Fully automated cloud based science data processing for Emirates Mars Mission

Paper number

IAC-21,B6,2,1,x63687

Author

Mr. Omran Al Hammadi, United Arab Emirates, Mohammed Bin Rashid Space Centre (MBRSC)

Coauthor

Mr. Bryan Harter, United States, Laboratory for Atmospheric and Space Physics (LASP) at University of Colorado

Year

2021

Abstract

The science data processing system of the Emirates Mars Mission (EMM) is responsible for generating Quicklook, Level 1, and Level 2 science data products using the processing pipeline software developed by the mission Instruments Team Facility (ITF) and distributing the products to the entire EMM science team and science community. It receives Level 0 science data and ancillary data from the Mission Operation Center. The science data processing environment is well-architected and hosted at Amazon Web Service (AWS) and make use of different advanced services provided by AWS such as AWS Lambda, AWS Elastic Container Service (ECS) AWS Step Function and AWS Simple Storage Service (S3).
The development of the processing pipeline software takes place on the ITFs’ institution. Once the software is of sufficient maturity, it is promoted to the ITF staging area within the science data processing environment. The cloud staging checkout ideally consists of; testing in the cloud with cloud resources (ECS/S3/etc) and packaging the processing software to be tested as a running container in the ITF staging area. After the software validation process completed at the cloud staging environment, the ITF promotes the updated version to the science data processing production platform.  
Upon the reception of new Level 0 science and housekeeping files as well as the spice kernels at the specific S3 bucket, the science data processing uses an AWS Lambda function to automatically check the files integrity and trigger the AWS processing pipeline step function. AWS Step Function is utilized to organize the workflow of science processing and production for all instruments scientific data, and reflects the update of newly received SPICE kernel from the MOC. A Lambda function is automatically triggered after the generation of each level of science products to index generated products into the science data processing database. The science data processing environment provides different data access mechanisms to the available science products and ancillary files for the mission science team as well as the science community. A well-developed web application provides an easy graphical user interface to the scientists in order to retrieve the scientific products. The back-end of data access and retrieval environment uses Serverless model. The access or retrieval request is handled by AWS API Gateway in which it triggers an AWS Lambda function to retrieve the metadata of science products from the science processing database or send back the requested products as objects to the scientist.
The presentation will cover details about the mission data processing platform and how different AWS managed services utilized and integrated to automate the data processing pipeline.

Abstract document

IAC-21,B6,2,1,x63687.brief.pdf

Manuscript document

IAC-21,B6,2,1,x63687.pdf (🔒 authorized access only).

To get the manuscript, please contact IAF Secretariat.