Generating corrupted data sources for the evaluation of matching systems

One of the most difficult aspects of developing matching systems – whether for matching ontologies or for other types of mismatched data – is evaluation. The accuracy of ontology matchers are usually evaluated by measuring the results produced by the systems against reference ontologies, but reference ontologies are expensive and difficult to create.In this paper we discuss the use of crptr, a system that corrupts data of different sorts in order to mimic the kind of differences one might expect to find between different data sources on related topics. This automatically creates a map between the original and the corrupted data source, and matching systems can be evaluated by comparing their output to this map. We describe the extension of crptr to ontology-based data and query mismatch, and then discuss how it could be extended to other kinds of matching including ontology matching.