CRF-MA is an extension of the Java implementation of Conditional Random Fields (CRFs) available in the Mallet toolbox in order to handle multiple annotators. CRF-MA uses the Expectation-Maximization algorithm to jointly learn the CRF model parameters, the relia- bility of the annotators and the estimated ground truth. When it comes to performance, the proposed method (CRF-MA) significantly outperforms typical approaches such as majority voting. See the original paper for further details:

Rodrigues, F. and Pereira, F.C. and Ribeiro, B., Sequence labeling with multiple annotators, Machine Learning, Springer, 2013.

Download here.