A methodology for underwater video classifier design and comparison on limited datasets

This paper presents an experimental protocol developed for the design, performance estimation and comparison of underwater video classifer systems. Such systems have to be designed using application data that is small, sparse and extremely variable. The proposed protocol uses outlier rejection, data pairing, Bootstrap performance estimation and hypothesis testing to achieve a robust performance estimate and comparison between classifer designs. The protocol is demonstrated and assessed on an application experiment. The application involves the design of a classification system for the automated detection of trawling marks from mission video. Two systems are proposed using selective and geometric feature types and an ensemble classifier. The protocol robustly identifies differences between the two proposed system designs using error and discrimination rates. Overall the geometric feature system is chosen as the final system. The protocol was also compared with other performance estimates and found to have the closest match to actual test data performance. © 2005 IEEE.