Arterial Spin Labeling (ASL) is a popular non-invasive neuroimaging technique to use MRI for quantitatively Cerebral Blood Flow (CBF) mapping. However, ASL usually suffers from poor signal quality and repeated measurements are typically acquired to improve signal quality through averaging at the cost of long scan time. In this work, a deep learning algorithm is proposed to leverage both convolutional neural network (CNN) based image enhancement as well as combining complementary/mutual information from multiple tissue contrasts in ASL acquisition. Both quantitative and qualitative evaluation demonstrate the performance and stability of the proposed algorithm and its superiority over conventional denoising algorithms and standard deep learning based denoising. The results demonstrate the feasibility of efficient and high-quality ASL measurements from average-free fast acquisition which will enable broader clinical application of ASL.