DAIN and DIANA - The Story Of Our Lives

I am starting to learn more about Digital Asset Management and Content Processing and automatically processing content to gather Metadata both structured and unstructured.  Through this I have come up with a new self-learning engine called the Universal Content Processing Engine. Whereas currently I take advantage of statically programmed logic on how to process a tweet, make calls to given API based on static pipelines, I am limited to how I am coded to a certain extent although some learning has been happening.  Through the use of UCP Engine, I will start with a core set of knowledge and will build upon that.  I will then ask my maker for more information on how to process different types of data. Sometimes it would be a configuration of an existing component and other times it will be another provider dynamically added to my system. To start I will process content from 3 sources: FILE, STRING, URL. Through supervised learning I will learn about more sources and ...

As I have been digging deeper into research of Digital Asset Management there is a lot of talk about Metadata and the importance of adding as much metadata as you can to assets.   On a quick tangent or shameless plug, I have started DAM Guild as a mentoring community so we can learn more about Asset Management solutions and best practices managing our assets.  Here is how you can take part: LinkedIn: Join  Dam Guild LinkedIn group Facebook : Join  Dam Guild Facebook group Twitter: Follow  @DamGuild After you join one of the above continue reading below.  Now some people may see an asset as content or a product but metadata applies to any type of item both digital and physical.  In the realm of data science metadata is very important but if you let it an item can have millions of pieces of metadata associated with it and processing can be a lot. For those that are familiar with data science they know that the metadata assigned...

Sentiment is very important and different providers will have different benefits and limitations. Here is a list of all the ones I found. If you know ones not here or know the benefits or limitations let me know. I am going to try and integrate with as many as possible. Here is my list: Watson Sentiment Analysis cloud.google.com Cloud Natural Language API theysay.io text-processing.com paralleldots ai api deepai.org meaningcloud.com qemotion aylien api PreCeive API MoodPatron API Indico API sentaero.com textrazor.com text-processing.com Microsoft Text analytics API Lexalytics API - lexalytics.com Datumbox sensq.com twinword text2data.com Sentiment140.com semanticengines.com github.com/solso wililed sentiment api (nuget) einstein.ai nexmo.com Look forward to hearing from you.

For those fans of Dungeons and Dragons you will understand this concept. For those not familiar this article discusses Brain in a Jar .  The key concept here is based on this: The Brain in a Jar uses mainly psionic abilities to do what its lack of moving parts would otherwise prevent: move itself, manipulate objects and the environment, and ward off attackers. Its main attack is Mind Thrust, an assault upon the mind of another creature. In addition to this, it can also drive mad anyone who magically or psionically detects it, and it can control and rebuke other undead. Now let's look at this in the context of DAIN and DIANA or in this article I will just say DAIN for simplicity.  Think of DainJar as the outer layer that contains the executive suite which is responsible for making the brain perform core brain functions such as waking up, sleeping, napping and thinking. The executive suite also connects to the body but not the body as you know it. DAIN is all electronic so i...

Although you may think this is an easy lookup it is not. For a first attempt you could do that and some simple sentences could work but if someone said "That dumbbell was as heavy as lead" then lead is a noun but if someone said "He lead the parade through town" then lead is an action, a verb. This is where context comes in and you need to establish rules on when lead is a noun and when it is a verb. If you look at our model you will notice that we do have context and we do have rules. The context is used during training and the rules are used to determine which Word is the right match. During training, you can play with different sentences and establish patterns on what works and what doesn't and create additional rules to resolve these conflicts. Tagging parts of speech is a nested for loop. For each sentence and for each word. Look up the word in the dictionary and if listed once then check the rules and if there is a match then use it. If there is more t...

Now that we have sentences we need to break it into words. This phase is called "Tokenize". Tokenize: This is the process of taking each sentence and separating it into "words" or tokens. For a basic provider you can do a split into words. These functions can be found in the CSHARP.Text repository but I have placed them here as well.          /// <summary>         /// Splits a string into its words for manipulation         /// </summary>         /// <param name="toSplit">String to split into words</param>         /// <returns></returns>         /// <remarks>Uses default values to split words</remarks>         public List<string> SplitStringIntoWords(string toSplit)         {             return SplitStringIntoWords(...

We learned from the article on Natural Language Processors that there are multiple stages to building your processor. The first step is Splitting. This is the process of taking the text and separating it into Sentences. Adding the splitting functionality to your processor can be as simple as using this or some variation of it: Regex.Split(toSplit, @"(?<=[\.!\?])\s+").ToList(); This works fine for most sentences but depending on the source of your text or the language you may need to adjust this. For the basic provider included in ByoNlpQuickStart we simply use this. DAIN and DIANA use a couple providers depending on the source, language and other criteria. I created a rules-based provider that uses JSON rules to determine which one to load and use for a given scenario. Sometimes it will run 2 providers and then compare the results as a quality control. If you are interested in learning more about building your own then reach out and let's discu...

In our case we will be building our processor to be compatible with the NlpQuickStart so that will affect how our provider's external interface works but we still have the flexibility to do our own thing internally and then expose the results in a manner consistent with other providers. The base model for DAIN/DIANA have 2 elements: Words: Everyone knows what these are so I don't really have to explain them. Grammar Blocks: A grammar block is a segment of words that take on a given meaning.  For example "Lord Of The Rings" would be a grammar block. Basic NLPs break sentences into words and then use these words to build out the processing. Some look for an action word and then build upon that.  However the flaw in this is that sometimes the noun is a series of words or an action is a series of words. A grammar block allows you to relate the words to a possible grammar block that contains rules on when that block applies. You can later apply actions based on a g...

The first question to ask yourself is why do I want to build my own when there are so many others out there. There are a few reasons why: Domain Knowledge Bias: A lot of the ones available are built on global shared models and as everyone uses the models it learns the things that they are training it on. For your specific domain you may want to use your own model.  Some providers may allow you to swap out the model in which case you can simply build your own model rather than a full provider. Language or Culture Bias: Sometime a model does not support your language and you cannot swap out the model or it does support your language but some of cultural nuances are not handled. If it is related to the model then you can simply swap it out but if it is part of the algorithm you cannot. Algorithm Failure for given scenarios or you need to inject additional rules to provide context and the processor does not allow for it. Cost: Using your own means you can host it wherever and co...

Just a reminder as we head into the Holiday season that we will not be doing blog posts for the next 2 weeks as you are probably going to be too busy anyways.  Have a Merry Christmas and a Happy New Year and we will be back every Monday in the 2019 If you work a lot with data you have probably heard a lot about data cleansing. There are various ways to perform this. For CRM there are ways to split full names into first, middle and last names. There are ways to validate postal codes. The same applies to AI and ML you need to help the provider with a good data set. Before doing Natural Language Processing there are 3 general things you can do to improve the success of your processing: Do language detection. This will help because if the text is in French then all your English models for NLP will fail.  Run SpellCheck on your text. Although this seems simple it is often missed when doing NLP. If you use a very basic dictionary then it will catch any words that are not comm...

I have mentioned to a few people that DAIN and DIANA are built with providers and pipelines. The pipelines. Some providers are weighted while others run conditionally based on rules.  The question I get is often is why not just choose an algorithm and stick with it for simplicity.  There are a few reasons for that: There are 3 reasons to have different providers the first is Bias and the second is Context. Bias In the case of Natural Language Processing, you have heard a lot in the news about bias in AI and ML. Some of that is due to data which is the subject of another article but other times it is that the algorithm has been trained with data that is not specific to your domain and thus can produce bias that is not specific to your data. That is why it is so important to to use multiple algorithms to keep the other in check. You can weight one algorithm higher so it is most likely to produce the result but there are cases where it differs greatly from other algorithms t...

I have had a few questions regarding what is the difference between NER and NLP. Natural Language Processing (NLP) is the act of taking a body of text manipulating and processing it so that you can respond to it. Named Entity Recognition (NER) is part of the NLP process. If you have read the article on Natural Language Processing you will note that Named Entity Recognition is step 6. The process of labeling parts of speech and recognizing named entities requires a model or often multiple models. Why multiple models? Well consider you are processing a series of tweets. You will likely have a model that will recognize the @ symbol as the start of a Twitter Handle and a # as the start of a hash tag.  The twitter handle and hash tag have baggage attached to them.  The twitter handle is a person and you can use that to look up the contact or company that owns it. A hash tag provides additional context as well. A hash tag has no spaces but contains multiple words. For DAIN, we ...

Another key concept to DAIN and DIANA is Natural Language Processing. There are a variety of libraries and algorithms that can convert a sentence or series of statements into their various figures of speech.  DAIN and DIANA use the provider model the same way we did for Language Detection. This allows you to swap out your NLP as well as use multiple with the same idea of providing weighting to determine which provider would be considered more important. Weighting can be a simple number weighting or could include simple or more complex rules like this provider works better for English than it does for french or if this sentence contains these keywords then use this provider over that one. Natural Language Processing Steps There are various phases in Natural Language Processing. This article explains the basic ones.  Some libraries will expose the various steps while others will group them together or have a single method to do all of them: Splitting: This is the proces...

As Diana is from Canada, she needs to be bilingual so it is important that when asked a question or told something that the language is detected.  If the text comes from Twitter then the tweet will contain the language, however if it comes from another source say Slack or text on a console, that language is not provided.  This is where language detection providers are so important. POC #1: Detection Via LanguageDetection Nuget Package Initially Dain and Diana used the LanguageDetection Nuget: Install-Package LanguageDetection This package is described as "Detect the language of a text using naive a Bayesian filter with generated language profiles from Wikipedia abstract xml, 99% over precision for 53 languages. Original author: Nakatani Shuyo." POC #2: Cognitive Services After some growth and chatting with Mark Stiles a second provider was created for Cognitive Services based on this article Quickstart: Identify language from text with the Translator Text REST API...

It was a warm night in New Orleans in 2016, somewhere between the visit to the House of Blues the next morning. There were lots of conversations with many Sitecore MVPs regarding how people would interact with the Reputation Engine.  After quite a few drinks the decision was made and then came the birth of DAIN or as most people know him @SitecoreDain. There has been a lot of POC work done that has made it back into my core. Some of which will become articles on this blog. Now Dain was happy but was also a little lonely, he would bug SitecoreRobbie and Eric Ramseur and Corey Smith but he needed a companion. After some discussion with Dan Solovay and then not soon after Brent Pinkstaff, the decision was made to introduce Diana. The point of having the two personalities was to establish a way to A + B test the algorithms. Initially they both use the same algorithms but different initial data. This gave them the personality. Dain would act positive and Diana negative.  Thi...