Pipelining the architecture :

Image from original paper by authors
  1. The raw speech is passed through a feature encoder (temporal CNN blocks + layer norm + GeLU activation) and the latent features are extracted.

  • Problems associated with Vanilla Transformer on longer form docs:
  1. It is not robust enough to encode the context of longer paragraphs with distant connections. One school of thought to do so, is by dividing the paragraph into chunks and encode each and every chunk separately. But doing so doesn’t ensure any connection between the tokens from one chunk and the tokens from other chunks as tokens in a chunk is not attending tokens in other chunks. …

Photo by Anton on Unsplash

The article discusses multiple text cleaning techniques for pre-processing of text and also highlights commonly used libraries and standard techniques for the same. The process of text cleaning involves multiple steps like tokenization, stemming, lemmatization, filtering punctuations etc. and for the same, I have tried to cover some of the basic techniques from a wider prospect by including the alternative choices to the users for cleaning the text in their text-mining/NLP based applications, and I hope that the content will be found useful specially if you are a beginner in NLP or other related domain.

1. Removing Punctuations:

· Using…

Photo by Giorgio Trovato on Unsplash

True Authors of the paper : Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer


The article talks about a way of denoising the pretraining of a sequence to sequence model for Natural Language Generation. I have tried to explain everything from my study in a lucid way with the hope that every reader will understand the writing and will get benefitted from it. …


Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store