We investigate response generation for multi-turn dialogue in generative chatbots. Existing generative models
based on RNNs (Recurrent Neural Networks) usually employ the last hidden state to summarize the history, which makes
models unable to capture the subtle variability observed in different dialogues and cannot distinguish the differences
between dialogues that are similar in composition. In this paper, we propose Pseudo-Variational Gated Recurrent Unit (PVGRU). The key novelty of PVGRU is a recurrent summarizing variable that
aggregates the accumulated distribution variations of subsequences. We train PVGRU without relying on posterior knowledge, thus avoiding the training-inference inconsistency problem. PVGRU can perceive subtle semantic variability through summarizing variables that are optimized by two objectives we employ for training: distribution consistency and reconstruction. In addition, we build a Pseudo-Variational Hierarchical Dialogue
(PVHD) model based on PVGRU. Experimental results demonstrate that PVGRU can broadly improve the diversity and
relevance of responses on two benchmark datasets.