analysis of mapper citizen science data quality
- Paper number
IAC-14,E1,8,9,x23217
- Author
Mr. Alexander Konyha, NASA, United States
- Year
2014
- Abstract
In 2008, NASA's Pavilion Lake Research Project (PLRP) began a study to develop a highly detailed map of the morphology, depth, and location of microbialites - underwater formations that provide insight into some of the earliest forms of life on Earth – in Pavilion and Kelly lakes, Canada. Data for this project was collected between 2008 and 2012 in three phases, the first two of which were limited to the PLRP science team. The third phase, called the Morphology Analysis Project for Participatory Exploration and Research (MAPPER), was an online Citizen Science activity open to the general public, and was made available in the run-up to the PLRP 2011 field season. MAPPER was built from the ground up with Citizen Science in mind, and included reference material, tutorials, and quizzes to provide a consistent training experience for new users. Using MAPPER, more than 4,600 people from around the world classified over 1.1 million images and submitted over 1.4 million observations, representing 2,200+ person-hours of effort. Quality of the expert and volunteer data are the prime focus of our study. Classification fields for a given image by a user were divided into three primary “labs”: ID, microbialites, algae, each with subfields pertaining to details of that field. Images were classified by multiple users. An “expert group consensus” was formed for each expert classification by determining the mode among all the experts that rated that particular field and image as a basis to compare the expert ratings to the volunteers. For example, for the primary fields identifying the presence of microbialites and algae in a given image, experts unanimously agreed on 46.5\%% and 36.2\%% respectively of the total image classifications with mean consistencies of 83.2\%% and 78.5\%% respectively for a sample size of 147,276 images and 49 experts. These results were used to compare against each volunteer classification. For the primary fields identifying the presence of microbialites and algae in a given image, volunteers were consistent with the expert group consensus by 82.8\%% and 85.0\%% respectively for a sample size of 441,696 ratings among 1958 unique volunteers. Overall, MAPPER volunteers appear to have aptitude at identifying the presence of algae and microbialites with respect to the experts, and key lessons were learnt through this process on how to better enable Citizen Scientists to be adequately trained and able to provide valid scientific results. These findings will be discussed during our presentation.
- Abstract document
- Manuscript document
(absent)