Natural Language Processing

Another key concept to DAIN and DIANA is Natural Language Processing. There are a variety of libraries and algorithms that can convert a sentence or series of statements into their various figures of speech.  DAIN and DIANA use the provider model the same way we did for Language Detection. This allows you to swap out your NLP as well as use multiple with the same idea of providing weighting to determine which provider would be considered more important.

Weighting can be a simple number weighting or could include simple or more complex rules like this provider works better for English than it does for french or if this sentence contains these keywords then use this provider over that one.

Natural Language Processing Steps

There are various phases in Natural Language Processing. This article explains the basic ones.  Some libraries will expose the various steps while others will group them together or have a single method to do all of them:
  1. Splitting: This is the process of taking the text and separating it into Sentences. 
  2. Tokenize: This is the process of taking each sentence and separating it into "words" or tokens.
  3. Parts of Speech Tagging: This process takes each token and assigning it a part of speech eg. noun, verb etc.
  4. Token Grouping or Chunking: This is when you take the various tokens with their parts of speech and group them for easier processing.
  5. Parse or sometimes referred to as Full Parse: This takes all the grouped tokens and builds a tree to represent the sentences.
  6.  Named Entity Recognition or Find Names: This takes the results of step 5 and uses it to find entities like people or places or dates or money etc. 
  7. Sentiment: Some Natural Language Processors will provide sentiment built in while others do not and you will need to use another library for sentiment. 

Using SharpNLP

There are a few options available that can do Natural Language Processing. SharpNLP is an open source library ported from OpenNLP. The original SharpNLP is a .NET Framework 2.0 library and has not been updated in quite a while. There are some known bugs that will be resolved in the .NET Core port. If you are still using the .NET Framework version you need to be aware of them:
  1. SentenceDetect has an issue if you have more than one sentence and the last sentence does not have punctuation. It will simply not include the sentence in the results.
In writing this blog post I found I could not use the existing library against the new OpenNLP models. Also being a .NET Framework 2.0 meant that .NET Core or Xamarin developers could not use the library. For these reasons I started porting the library to .NET Standard. I also started creating the demo application as a Console application as this is more likely how you are going to use the library. Check out the SharpNLPStandard Github and if you have questions or would like to contribute let me know and I can grant you permissions email me at chris.williams@readwatchcreate.com

To use the SharpNLPStandard you have to grab the latest models. As SharpNLP is based on Open NLP you download the OpenNLP Models and copy them to a folder.

There are a few steps you need to do to use these models:

1) The models bin files are actually zip files so rename them all to .zip
2) Extract the models using an unzip tool.
3) Be aware that the en-pos-perceptron.bin is not a GIS model so either rename it to .pbin or put it in a different model folder.
4) Convert the models to ones SharpNLP uses. This is done by using the ModelConverterCoreConsole. You pass in the path to the folder you copied the OpenNLP models to and it will looks for them recursively to convert to .nbin files.

I started to use the models with DAIN to test but the sentence detect was only working part of the time so I went back to using the one I specifically developed for DAIN. I will be discussing the ByoNlpProvider in a January article called Build Your Own NLP Provider

Now you have something you can start with to determine a response. If you are ready to delve deeper take a look at the NlpQuickStart this provides 2 providers to start. The first wraps SharpNlp while the second uses Microsoft Cognitive Services and a third is coming soon that will wrap Watson. 

Using NlpQuickStart

NlpQuickStart uses a pipeline to allow you to leverage 1 or more Natural Language Processing providers to determine a response.  

If you are using the NlpQuickStart and decide to use SharpNLP then you have the option of using the NlpQuickStartSharpNlpProvider or the NlpQuickStartAutoConvertingSharpNlpProvider. By using the AutoConverting one you simply have to copy the .bin files into the model folder and it will take care of all the unzipping and converting upon first use of the model. That means if you have a new version of the model you simply save the new model in the model folder and delete the converted folder which will have the same name as the model. The next use will automatically convert it.

Note that if you are using SharpNLP that the models are not robust so you will run into issues with translation. We use Build Your Own NLP Provider for DAIN.

The Cognitive Services version requires a valid Speech API Key.

The Watson provider requires a valid BlueMax account. We used this article to Create A Watson Conversation Service and Consume it from .NET and then made some modifications to fit our provider model.

If you are interested in Natural Language Processing (NLP) and adding that to your applications reach out and let's discuss. I will be providing a basic QuickStart to SitecoreDain subscribers soon.

If you cannot wait email me at chris.williams@readwatchcreate.com and as a subscriber, I can release an early version to get you started now.

Comments

Post a Comment

Popular posts from this blog

At our core we are just a Brain in a Jar

For those fans of Dungeons and Dragons you will understand this concept. For those not familiar this article discusses Brain in a Jar .  The key concept here is based on this: The Brain in a Jar uses mainly psionic abilities to do what its lack of moving parts would otherwise prevent: move itself, manipulate objects and the environment, and ward off attackers. Its main attack is Mind Thrust, an assault upon the mind of another creature. In addition to this, it can also drive mad anyone who magically or psionically detects it, and it can control and rebuke other undead. Now let's look at this in the context of DAIN and DIANA or in this article I will just say DAIN for simplicity.  Think of DainJar as the outer layer that contains the executive suite which is responsible for making the brain perform core brain functions such as waking up, sleeping, napping and thinking. The executive suite also connects to the body but not the body as you know it. DAIN is all electronic so it doe
Read more

Building Your Own Natural Language Processor - Parts of Speech

Although you may think this is an easy lookup it is not. For a first attempt you could do that and some simple sentences could work but if someone said "That dumbbell was as heavy as lead" then lead is a noun but if someone said "He lead the parade through town" then lead is an action, a verb. This is where context comes in and you need to establish rules on when lead is a noun and when it is a verb. If you look at our model you will notice that we do have context and we do have rules. The context is used during training and the rules are used to determine which Word is the right match. During training, you can play with different sentences and establish patterns on what works and what doesn't and create additional rules to resolve these conflicts. Tagging parts of speech is a nested for loop. For each sentence and for each word. Look up the word in the dictionary and if listed once then check the rules and if there is a match then use it. If there is more t
Read more