Skip to main content
Kent Academic Repository

Improving Language Modelling with Noise Contrastive Estimation

Liza, Farhana Ferdousi, Grzes, Marek (2018) Improving Language Modelling with Noise Contrastive Estimation. In: The Thirty-Second AAAI Conference on Artificial Intelligence. . pp. 5277-5284. Association for the Advancement of Artificial Intelligence, Palo Alto, California, USA ISBN 978-1-57735-800-8. (KAR id:65147)

Abstract

Neural language models do not scale well when the vocabulary is large. Noise contrastive estimation (NCE) is a sampling-based method that allows for fast learning with large vocabularies. Although NCE has shown promising performance in neural machine translation, its full potential has not been demonstrated in the language modelling literature. A sufficient investigation of the hyperparameters in the NCE-based neural language models was clearly missing. In this paper, we showed that NCE can be a very successful approach in neural language modelling when the hyperparameters of a neural network are tuned appropriately. We introduced the `search-then-converge' learning rate schedule for NCE and designed a heuristic that specifies how to use this schedule. The impact of the other important hyperparameters, such as the dropout rate and the weight initialisation range, was also demonstrated. Using a popular benchmark, we showed that appropriate tuning of NCE in neural language models outperforms the state-of-the-art single-model methods based on the standard LSTM recurrent neural networks.

Item Type: Conference or workshop item (Proceeding)
Subjects: Q Science > QA Mathematics (inc Computing science) > QA276 Mathematical statistics
Q Science > QA Mathematics (inc Computing science) > QA 76 Software, computer programming, > QA76.87 Neural computers, neural networks
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
Depositing User: Marek Grzes
Date Deposited: 08 Dec 2017 16:00 UTC
Last Modified: 10 Oct 2023 13:00 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/65147 (The current URI for this page, for reference purposes)

University of Kent Author Information

Liza, Farhana Ferdousi.

Creator's ORCID:
CReDIT Contributor Roles:

Grzes, Marek.

Creator's ORCID: https://orcid.org/0000-0003-4901-1539
CReDIT Contributor Roles:
  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.