The visual properties of an object provide many cues as to the tensile strength, compliance, and density of the material from which it is made. However, it is not well understood how these implicit associations affect our perceptions of these properties and how they determine the initial forces that are applied when an object is picked up. Here we examine the effects of these cues on such forces by using the classic "material-weight illusion" (MWI). Grip and load forces were measured in three experiments as participants lifted cubes made from metal, wood, and expanded polystyrene. These cubes were adjusted to have a different mass than would be expected for a particular material. For the initial lifts, the forces were scaled to the expected weight of each object, such that the metal block was gripped and lifted with more force than the polystyrene one. After a few lifts, however, participants scaled their forces to the actual weight of the blocks, implicitly disregarding the misleading visual cues to each block's composition (experiments 1 and 2). Despite this rapid rescaling, participants experienced a robust MWI throughout the duration of the experiments. In fact, the grip and load forces never matched the perception of weight until the differences in the visual surface properties between the blocks were removed (experiment 3). These findings are discussed in relation to recent debates about the underlying causes of weight-based illusions and the effect of top-down visual cues on perception and action.