Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
This paper proposes two algorithms for recovering ground truth labels in crowd sourcing tasks for binary classisification. The problem is formulated as an online Bayesian version of the Dawid & Skene model (with beta priors) which is quite natural. The algorithms are based on variational approximations of the posterior (i.e. they try to find the best approximation that is product distribution). From this approach two algorithms are derived. One is fast (i.e. O(1) per label) and less accurate. The other one is more accurate and but slower (still polynomial time). Both algorithms are analyzed under two policies that choose the next instance/task to be labeled. (Round robin picks the instance with least labels. The least informative picks instance with greatest spread of the posterior.) The paper has theoretical analysis of the convergence of algorithms under both policies. It also contains results of experiments on the online problem in which the two algorithms are compared with other existing algorithms. The paper is clearly written. The results in the paper are a nice contribution to the literature of crowd sourcing.