NLP techniques

Text data vectorization

The process of converting text to numbers is called vectorization. Now, after preprocessing the text, we need to represent the text in numerical form, that is, encode the text data in the form of numbers, which can later be used in algorithms.

Lots of words

This is one of the easiest ways to vectorize text. In BOW logic, two sentences can be said to be the same if they contain the same set of words.

BOW creates a dictionary of unique words in a corpus (collecting all tokens in the data). For example, the corpus in the image above consists of all the words of the sentences S1 and S2.

Now we can create a table in which the columns correspond to d unique words included in the corpus, and the rows correspond to sentences (documents). We set the value to 1 if the word is in the sentence, and 0 if it is not.

Word frequency

Term Frequency calculates the probability of finding a word in a document. Well, for example, we want to know what is the probability of finding the word wi in the document dj.

Frequency of reverse documents

In IDF logic, if a word occurs in all documents, it is not very useful. This determines the uniqueness of the word in the entire corpus.

According to Research and Market, the global natural language processing market will grow from $ 10.2 billion in 2019 to $ 26.4 billion by 2024, at a CAGR of 21.0% over the period. Forecast: doctranslator.

The main drivers of growth in the NLP market: increased use of smart devices, as well as cloud solutions and applications based on NLP that improve customer service, increased investment in technology in the healthcare sector.