NOTE: Please take a look at the more recent, more complete, cleaner and faster Julia implementations of these methods (and others) for learning from multiple annotators and crowds available here.

MA-LR is a Python implementation of the multiple-annotator logistic regression model proposed in:

[1] Rodrigues, F. and Pereira, F.C. and Ribeiro, B. , Learning from Multiple Annotators: Distinguishing Good from Random Labelers, Pattern Recognition Letters, 2013.

Furthermore, it provides an implementation of the multi-class extension of the model proposed in:

[2] Raykar, V., Yu, S., Zhao, L., Jerebko, A., Florin, C., Valadez, G., Bogoni, L., Moy, L., Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proc. of the 26th Int. Conf. on Machine Learning, pp. 889–896, 2009.

[3] Raykar, V., Yu, S., Zhao, L., Valadez, G., Florin, C., Bogoni, L., Moy, L., Learning from crowds. Journal of Machine Learning Research, 1297– 1322, 2010.

The tar.gz with the source code can be obtained here.

The Amazon’s Mechanical Turk data used in [1] is also available here for download.

DOWNLOAD:

Source code // AMT data

CONTACT:

Please send questions and comments to fmpr [at] dei.uc.pt