Crowdsourcing Quality Evaluation Strategies Based on Voting Consistency 〓
YUE Dejun1， YU Ge1， SHEN Derong1， YU Xiaocong2
1 School of Information Science & Engineering， Northeastern University， Shenyang 110819， China; 2 Department of Computer Crime Detection， China Criminal Police College， Shenyang 110035， China.
[1] Perona P，Welinder P.Online crowdsourcing:rating annotators and obtaining costeffective labels［C］//Computer Vision and Pattern Recognition Workshop.San Francisco:IEEE Computer Society，2010:25/32. [2] Manas J，Hector G M，Aditya P.Evaluating the crowd with confidence［C］// Proceedings of ACM SIGKDD.New York:ACM Press，2013:686/694. [3] Snow R，Connor B O，Jurafsky D，et al.Cheap and fast—but is it good? evaluating nonexpert annotations for natural language tasks［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing.New York:ACM Press，2008:254/263. [4] Liu X，Lu M Y，Ooi B C，et al.CDAS:a crowdsourcing data analytics system［J］.Proceedings of the VLDB Endowment，2012，5(10):1040/1051. [5] Cao C C，She J Y，Tong Y X，et al.Whom to ask? Jury selection for decision making tasks on microblog［J］.Proceedings of the VLDB Endowment，2012，5(11):1495/1506. [6] 张志强，逄居升，谢晓芹，等.众包质量控制策略及评估算法研究［J］.计算机学报，2013，36(8):1636/1649.(Zhang Zhiqiang，Pang Jusheng，Xie Xiaoqin，et al.Research on crowdsourcing quality control strategies and evaluation algorithm［J］.Chinese Journal of Computers，2013，36(8):1636/1649.) [7] Nitzan S，Paroush J.Optimal decision rules in uncertain dichotomous choice situation［J］.International Economic Review，1982，23:289/297. [8] Owen G B，Grofman B，Feld S L.Proving a distributionfree generation of the Condorcet Jury theorem［J］.Mathematical Social Science，1989，17:1/16. [9] Yashar R B，Paroush J.A nonasymptotic Condorcet Jury theorem［J］.Social Choice and Welfare，2000，17:189/199. [10] Tian Y D，Zhu J.Learning from crowds in the presence of schools of thought［C］// Proceedings of the ACM SIGKDD.New York:ACM Press，2012:226/234.
