CRF-MA: Sequence labeling with multiple annotators
CRF-MA is an extension of the Java implementation of Conditional Random Fields (CRFs) available in the Mallet toolbox in order to handle multiple annotators. CRF-MA uses the Expectation-Maximization algorithm to jointly learn the CRF model parameters, the relia- bility of the annotators and the estimated ground truth. When it comes to performance, the proposed method (CRF-MA) significantly outperforms typical approaches such as majority voting. See the original paper for further details:
The tar.gz with the source code can be obtained here.
The datasets (including the Amazon’s Mechanical Turk data) used are also available here for download.
Please send questions and comments to fmpr [at] dei.uc.pt