Improving existing buildings' energy performance requires energy models that accurately represent the building. Computer vision methods, particularly image-based 3D reconstruction, can effectively support the creation of 3D building models. In this paper, we present an image-based 3D reconstruction pipeline that supports the semi-automated modeling of existing buildings. We developed two methods for the robust estimation of the building planes from a 3D point cloud that (i) independently estimate each plane and (ii) impose a perpendicularity constraint to plane estimation. We also estimate external walls' thermal transmittance values using an infrared thermography-based method, with the surface temperatures measured by a thermal camera. We validate our approach (i) by testing the pipeline's ability in constructing accurate surface models subject to different image sets with varying sizes and levels of image quality, and (ii) through a comparative analysis between the calculated energy performance metrics of a ground truth and calculated energy simulation model.