Relevance assessment of crowdsourced data (CSD) using semantics and geographic information retrieval (GIR) techniques

Relevance assessment of crowdsourced data (CSD) using semantics and geographic information retrieval (GIR) techniques

Crowdsourced data (CSD) generated by citizens is becoming more popular as its potential utilization in many applications increases due to its currency and availability. However, the quality of CSD, including its relevance, is often questioned as the data is not generated by professionals nor follows standard data-collection procedures. The quality of CSD can be assessed according to a range of characteristics including its relevance. In this paper, information relevance has been explored through using geographic information retrieval (GIR) techniques to identify the most highly relevant information from a set of crowdsourced data. This research tested a relevance assessment approach for CSD by adapting relevance assessment techniques available in the GIR domain. Thematic and geographic relevance were assessed by analyzing the frequency of selected terms which appeared in CSD reports using natural language processing techniques. The study analyzed crowdsourced reports from the 2011 Australian flood’s Crowdmap to examine a proof of concept on relevance assessment using a subset of this dataset based on a defined set of queries. The results determined that the thematic and geographic specificities of the queries were 0.44 and 0.67, respectively, which indicated the queries used were more geographically specific than thematically specific. The Spearman’s rho value of 0.62 indicated that the final ranked relevance lists showed reasonable agreement with a manually classified list and confirmed the potential of the approach for CSD relevance assessment. In particular, this research has contributed to the field of CSD relevance assessment through an integrated thematic and geographic relevance ranking process by using a user-query specificity approach to improve the final ranking.