Abstract: Existing knowledge-enhanced methods have achieved remarkable results in certain Q\&A tasks via obtaining diverse knowledge from different knowledge bases. However, limited by the properties of retrieved knowledge, they still have trouble benefiting from both the knowledge relevance and distinguishment simultaneously. To address the challenge, we propose \textbf{CPACE}, a \textbf{C}oncept-centric \textbf{P}rompt-b\textbf{A}sed \textbf{C}ontrastive \textbf{E}xplanation Generation model, which aims to convert obtained symbolic knowledge into the contrastive explanation for better distinguishing the differences among given candidates. Firstly, following previous works, we retrieve different types of symbolic knowledge with a concept-centric knowledge extraction module. After that, we generate corresponding contrastive explanation using acquired symbolic knowledge and prompt as guidance for better modeling the knowledge distinguishment and interpretability. Finally, we regard the generated contrastive explanation as external knowledge for downstream task enhancement. We conduct a series of experiments on three widely-used question-answering datasets: CSQA, QASC, and OBQA. Experimental results demonstrate that with the help of generated contrastive explanation, our CPACE model achieves new SOTA on CSQA (89.8\% on the testing set, 0.9\% higher than human performance), and gains impressive improvement on QASC and OBQA (4.2\% and 3.5\%, respectively).