Truth Discovery in Crowdsourcing Systems


Crowdsourced data is noisy in most cases, and thus repetitive labeling is usually utilized to ensure the quality of the obtained labels. Repetitive labeling requires multiple workers to provide labels for each item. By aggregating the multiple labels, one single result, which is called consensus result, can be used as estimated true label for each item.
The AC2 data set was originally used in the work of Ipeirotis et al. “Quality management on Amazon mechanical Turk”. It includes the AMT judgments for websites for the presence of adult content on the page. The original TREC data set is used in the work of Buckley et al. “Overview of the trec 2010 relevance feedback track (notebook)”. It has AMT ordinal graded relevance judgments for pairs of search queries and URLs.

Datasets and framework for induction of feature drifts

Datasets for concept drift detection

The repository presents datasets used in the paper:

Sethi, Tegjyot Singh, and Mehmed Kantardzic. “On the Reliable Detection of Concept Drift from Streaming Unlabeled Data.” Expert Systems with Applications (2017).