Deep multi-model fusion network based real object tactile understanding from haptic data

The tactile information of an object is one of the crucial features which define the impression of that object. This paper presents a novel multi-model fusion network for real object’s tactile understanding from haptic data. Furthermore, a low-cost 3D printed artificial finger-based tactile sensing system is designed for capturing haptic information in the form of acceleration profile, angular velocity, and normal force. Our proposed multi-model fusion network includes three different networks. First, we introduce a novel ensemble 2D convolutional neural network, namely SpectroNet, which captures the spatial features from the spectrogram of acceleration profile. Second, we design a 1-D convolutional neural network (CNN) with residual connection for extracting detailed spatial information from each segment of collected data. Third, we design bi-directional gated recurrent unit networks (BiGRU) to capture temporal dynamics. Moreover, the attention mechanism is utilized in all three proposed networks to assign weights to the features according to their contributions, which enhance the performance further. Finally, extensive experimental analysis is conducted on our dataset (i.e., 60 real objects, which cover both planner and non-planner surfaces) as well as the TUM surface material database. Empirical evaluations demonstrate that the proposed method significantly outperformed state-of-the-art methods in terms of accuracy, precision, recall and F1-score. Furthermore, we also found that the proposed multi-model fusion network substantially improves the performance compared to the single network.