In this work, we present a machine learning method to guide an ultrasound operator towards a selected area of interest. Unlike other automatic medical imaging methods, ultrasound imaging is one of the few imaging modalities where the operator’s skill and training are critical in obtaining high quality images. Additionally, due to recent advances in affordability and portability of ultrasound technology, its utilization by non-experts has increased. Thus, there is a growing need for intelligent systems that have the ability to assist ultrasound operators in both clinical and non-clinical scenarios. We propose a system that leverages machine learning to map real time ultrasound scans to transformation vectors that can guide a user to a target organ or anatomical structure. We present a unique training system that passively collects supervised training data from an expert sonographer and uses this data to train a deep regression network. Our results show that we are able to recognize anatomical structure through the use of ultrasound imaging and give the user guidance toward obtaining an ideal image.