Learning word dependencies in text by means of a deep recurrent belief network

Chaturvedi, Iti, Ong, Yew-Soon, Tsang, Ivor W., Welsch, Roy E., and Cambria, Erik (2016) Learning word dependencies in text by means of a deep recurrent belief network. Knowledge Based Systems, 108. pp. 144-154.

[img] PDF (Published Version) - Published Version
Restricted to Repository staff only

View at Publisher Website: https://doi.org/10.1016/j.knosys.2016.07...


We propose a deep recurrent belief network with distributed time delays for learning multivariate Gaussians. Learning long time delays in deep belief networks is difficult due to the problem of vanishing or exploding gradients with increase in delay. To mitigate this problem and improve the transparency of learning time-delays, we introduce the use of Gaussian networks with time-delays to initialize the weights of each hidden neuron. From our knowledge of time delays, it is possible to learn the long delays from short delays in a hierarchical manner. In contrast to previous works, here dynamic Gaussian Bayesian networks over training samples are evolved using Markov Chain Monte Carlo to determine the initial weights of each hidden layer of neurons. In this way, the time-delayed network motifs of increasing Markov order across layers can be modeled hierarchically using a deep model. To validate the proposed Variable-order Belief Network (VBN) framework, it is applied for modeling word dependencies in text. To explore the generality of VBN, it is further considered for a real-world scenario where the dynamic movements of basketball players are modeled. Experimental results obtained showed that the proposed VBN could achieve over 30% improvement in accuracy on real-world scenarios compared to the state-of-the-art baselines.

Item ID: 62594
Item Type: Article (Research - C1)
ISSN: 1872-7409
Keywords: deep belief networks, time-delays, variable-order, Gaussian networks, Markov Chain Monte Carlo
Copyright Information: © 2016 Elsevier B.V. All rights reserved.
Funders: ASTAR Thematic Strategic Research Programme, Nanyang Technological University, Singapore
Projects and Grants: ASTAR Grant no. 1121720013
Date Deposited: 23 Mar 2020 23:33
FoR Codes: 46 INFORMATION AND COMPUTING SCIENCES > 4602 Artificial intelligence > 460208 Natural language processing @ 100%
SEO Codes: 97 EXPANDING KNOWLEDGE > 970108 Expanding Knowledge in the Information and Computing Sciences @ 100%
Downloads: Total: 2
More Statistics

Actions (Repository Staff Only)

Item Control Page Item Control Page