The aim of this study is to design an adaptive controller for the hard contact interaction problem of underwater vehicle-manipulator systems to realize asset inspection through physical interaction. The proposed approach consists of a force and position controller in the operational space of the end-effector of the robot manipulator mounted on an underwater vehicle. The force tracking algorithm keeps the end-effector perpendicular to the unknown surface of the asset and the position tracking algorithm makes it follow a desired trajectory on the surface. The challenging problem in such a system is to maintain the end-effector of the manipulator in a continuous and stable contact with the unknown surface in the presence of disturbances and reaction forces that constantly move the floating robot base in an unexpected manner. The main contribution of the proposed controller is the development of the adaptive force tracking control algorithm based on switching actions between contact and non-contact states. When the end-effector loses contact with the surface, a velocity feed-forward augmented impedance controller is activated to rapidly regain contact interaction by generating a desired position profile whose speed is adjusted depending on the time and the point where the contact was lost. Once the contact interaction is reestablished, a dynamic adaptive damping-based admittance controller is operated for fast adaptation and continuous stable force tracking. To validate the proposed controller, we conducted experiments with a land robotic setup composed of a 6 DOF Stewart Platform imitating an underwater vehicle and a 7 DOF KUKA IIWA robotic arm imitating the underwater robot manipulator attached to the vehicle. The proposed scheme significantly increases the contact time under realistic disturbances, in comparison to our former controllers without and adaptive control scheme. We have demonstrated the superior performance of the current controller with experiments and quantified measures.