A neural network layer that enables training of deep neural networks directly from crowdsourced labels (e.g. from Amazon Mechanical Turk) or, more generally, labels from multiple annotators with different biases and levels of expertise, as proposed in the paper:

• Rodrigues, F. and Pereira, F. Deep Learning from Crowds. In Proc. of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18).

This implementation is based on Keras and Tensorflow.

Check out the GitHub project for further details, usage instructions and examples.

DOWNLOAD:

• CrowdLayer

DATASETS:

• LabelMe (multi-class classification; annotations from Amazon Mechanical Turk)
• MovieReviews (regression; annotations from Amazon Mechanical Turk)
• 2003 CONLL NER task (sequence labelling; annotations from Amazon Mechanical Turk)

CONTACT:

Please send questions and comments to rodr [at] dtu.dk