Deep multi-model fusion network based real object tactile understanding from haptic data

Joolekha Bibi Joolee, Md Azher Uddin, Seokhee Jeon*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

6 Citations (Scopus)


The tactile information of an object is one of the crucial features which define the impression of that object. This paper presents a novel multi-model fusion network for real object’s tactile understanding from haptic data. Furthermore, a low-cost 3D printed artificial finger-based tactile sensing system is designed for capturing haptic information in the form of acceleration profile, angular velocity, and normal force. Our proposed multi-model fusion network includes three different networks. First, we introduce a novel ensemble 2D convolutional neural network, namely SpectroNet, which captures the spatial features from the spectrogram of acceleration profile. Second, we design a 1-D convolutional neural network (CNN) with residual connection for extracting detailed spatial information from each segment of collected data. Third, we design bi-directional gated recurrent unit networks (BiGRU) to capture temporal dynamics. Moreover, the attention mechanism is utilized in all three proposed networks to assign weights to the features according to their contributions, which enhance the performance further. Finally, extensive experimental analysis is conducted on our dataset (i.e., 60 real objects, which cover both planner and non-planner surfaces) as well as the TUM surface material database. Empirical evaluations demonstrate that the proposed method significantly outperformed state-of-the-art methods in terms of accuracy, precision, recall and F1-score. Furthermore, we also found that the proposed multi-model fusion network substantially improves the performance compared to the single network.

Original languageEnglish
Pages (from-to)16605-16620
Number of pages16
JournalApplied Intelligence
Issue number14
Early online date24 Mar 2022
Publication statusPublished - Nov 2022


  • 1-D convolutional neural network
  • Bi-directional gated recurrent unit networks
  • Multi-model fusion network
  • SpectroNet
  • Tactile understanding

ASJC Scopus subject areas

  • Artificial Intelligence


Dive into the research topics of 'Deep multi-model fusion network based real object tactile understanding from haptic data'. Together they form a unique fingerprint.

Cite this