Natural Language Processing (NLP)

So far I learnt how to apply Machine Learning for specific problems. I have primarily taken Classifier examples to understand the ML and how it is applied. In all these examples, I worked on "clean data". That means, data was already processed and it was available for ML algorithms. In reality, it will never be the case. We have to extract data and do a lot of processing before that can be applied to the ML models. This pre-processing turns out to be a major chunk of work in a ML project.

So as a next step, I need to start learning these techniques which involves data extraction, cleansing etc... There are specific terminology for these processes which I will start using as I go through the learning. For now, I'll keep it in a layman's language.

When I was discussing this with my colleague who is a ML engineer, he suggested me to get into NLP. He said 'if you get into NLP, you will end up working on end-to-end project from data extraction to prediction/classification'. It was a sound piece of advice! With that I started learning NLP!

What is NLP or Natural Language Processing? Till now the input data which was fed to ML models were discrete and numeric in nature. In contrast, what if you want to take human speech and natural texts as input data? Obviously the ML models do not understand these as it is and on top of that, human speech and natural texts are most complex in nature. So how do we deal with such data so that machine can understand. That's exactly what NLP is for! Of course, I have tried to put in a layman's language. But I hope you understand (may be at 10,000 ft level) what NLP is all about!

Example of natural language texts
Reviews in Google Map - an example of natural language texts

In next series of posts, you will see articles related to NLP. I'm excited to get my hands dirty in the world of NLP!

Comments

Post a Comment

Popular posts from this blog

Ordinal Encoder, OneHotEncoder and LabelBinarizer in Python

Usually in the data set we work on, we will have both numerical data as well as categorical data (non-numeric). There are algorithms which do not work on categorical data. These algorithms are designed that way to enhance efficiency. So in such cases how do we train the model with categorical data? This is where we need to apply a technique called Encoding. Encoding transforms categorical data into numeric. There are various Encoders available in sci-kit learn library. In this post, I'll share my learning on three Encoders. Ordinal Encoder This performs ordinal (integer) encoding of categorical data. Let's look at an example. I have an array of categorical data. This is a 2 dimensional array. multiArray = np.array([[ 'Karnataka' ,  'KA' ], [ 'Maharastra' ,  'MH' ], [ 'Gujarat' ,  'GJ' ]]) Then I use OrdinalEncoder to transform this data. ordinalEncoder = OrdinalEncoder() ordinalEncoderArray = ordinalEncoder.fit_transform(mult
Read more

Natural Language Toolkit (NLTK)

To work on Natural Language Processing (NLP), we need a library. One popular Python library is Natural Language Toolkit or NLTK. As mentioned in the NLTK website , NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning. Some of the terms like stemming, semantic reasoning are new to me. I'll explore them in future. For now I wanted to test this library so I took the example provided in that website. First I imported nltk library import  nltk Then, formed a sample sentence sentence =  "I'm learning Natural Language Processing. Natural Language Processing is also called as NLP" Then, tokenize the text into words tokens = nltk.word_tokenize(sentence) When I ran this code, I got this error. LookupError: ********
Read more

Failed to create Anaconda menus

Image
After my first attempt at executing a  ML program in Weka  platform, I wanted to switch to Python. I did take up an online course on Python a few months back. So Python was my go-to language/library in my ML journey. As explained in  this article , I had to install Anaconda to set up the environment in my laptop. I downloaded the  Anaconda 3 2019.10 Windows 64 bit version , and when I installed (on Windows 7), it threw an error “ Failed to create menus ” as shown below. After clicking Retry a couple of times, ignored this error and went ahead with the installation. After the installation, I  could not find Anaconda program  in  Windows Start Menu . So the above error might have caused a problem in the installation itself. I tried multiple times by uninstalling and installing again but without any success. Then I came across this post in  Stack Overflow , where it was suggested to use an older version of Anaconda. So I downloaded   Anaconda 2 2019.3 Windows 64 bit version , and the inst
Read more