Providers and Pipeline, Oh Why!!!

I have mentioned to a few people that DAIN and DIANA are built with providers and pipelines. The pipelines. Some providers are weighted while others run conditionally based on rules.  The question I get is often is why not just choose an algorithm and stick with it for simplicity.  There are a few reasons for that:

There are 3 reasons to have different providers the first is Bias and the second is Context.

Bias

In the case of Natural Language Processing, you have heard a lot in the news about bias in AI and ML. Some of that is due to data which is the subject of another article but other times it is that the algorithm has been trained with data that is not specific to your domain and thus can produce bias that is not specific to your data. That is why it is so important to to use multiple algorithms to keep the other in check. You can weight one algorithm higher so it is most likely to produce the result but there are cases where it differs greatly from other algorithms that is something to track and learn from. Why does SharpNLP give me x while Cognitive Services always gives me y.  By more closely examining these anomilies you can set up rules that help determine when each algorithm is beneficial.

I highly recommend reading this article on Ethical AI thanks Eric Ramseur for sharing.

Context

The other reason to swap out your Natural Language Provider or maybe just the model would be context.  If you are working with data from a given source, say Twitter, then there are certain words, terms, short forms and other things that are specific to Twitter. The provider may do some pre-processing then use an existing provider.  Depending on the topic you may also have specific domain knowledge that the provider or model need to be aware of. For instance, an article you know is on Sitecore would have different terms than an article on astrophysics.

Cost/Limits

This is always a factor in software development even though we want the ideal solutions we want to keep things frugal too. If you are doing NLP on a large volume of text and documents using metered services can put you over your subscription limits or escalate per call costs.  This is where you would apply rules regarding using metered providers.  In some cases you may simply set USE_METERED to false but in other cases you may use rules based on previous providers responses. For example, if you run 2 free providers and they do not provide answers then you would use the paid one and then may use the results to improve the models for the free ones.  In the case of DAIN we run Language Detection using the BasicLanguageDetectionProvider. If the result is under a given ProbablilityWeight then we would run it under the CognitiveServicesLanguageDetectionProvider one and then use that to update the BasicLanguageDetectionProvider

The whole point of creating providers and pipelines is that you have the flexibility to improve as you go and use the proper provider for the proper situation.

If you are interested in learning more then reach out and let's discuss. I will be providing a basic QuickStart to SitecoreDain subscribers soon. If you cannot wait email me at chris.williams@readwatchcreate.com and as a subscriber, I can release an early version to get you started now.

Comments

Post a Comment

Popular posts from this blog

At our core we are just a Brain in a Jar

For those fans of Dungeons and Dragons you will understand this concept. For those not familiar this article discusses Brain in a Jar .  The key concept here is based on this: The Brain in a Jar uses mainly psionic abilities to do what its lack of moving parts would otherwise prevent: move itself, manipulate objects and the environment, and ward off attackers. Its main attack is Mind Thrust, an assault upon the mind of another creature. In addition to this, it can also drive mad anyone who magically or psionically detects it, and it can control and rebuke other undead. Now let's look at this in the context of DAIN and DIANA or in this article I will just say DAIN for simplicity.  Think of DainJar as the outer layer that contains the executive suite which is responsible for making the brain perform core brain functions such as waking up, sleeping, napping and thinking. The executive suite also connects to the body but not the body as you know it. DAIN is all electronic so it doe
Read more

Natural Language Processing

Another key concept to DAIN and DIANA is Natural Language Processing. There are a variety of libraries and algorithms that can convert a sentence or series of statements into their various figures of speech.  DAIN and DIANA use the provider model the same way we did for Language Detection. This allows you to swap out your NLP as well as use multiple with the same idea of providing weighting to determine which provider would be considered more important. Weighting can be a simple number weighting or could include simple or more complex rules like this provider works better for English than it does for french or if this sentence contains these keywords then use this provider over that one. Natural Language Processing Steps There are various phases in Natural Language Processing. This article explains the basic ones.  Some libraries will expose the various steps while others will group them together or have a single method to do all of them: Splitting: This is the process of taki
Read more

Building Your Own Natural Language Processor - Parts of Speech

Although you may think this is an easy lookup it is not. For a first attempt you could do that and some simple sentences could work but if someone said "That dumbbell was as heavy as lead" then lead is a noun but if someone said "He lead the parade through town" then lead is an action, a verb. This is where context comes in and you need to establish rules on when lead is a noun and when it is a verb. If you look at our model you will notice that we do have context and we do have rules. The context is used during training and the rules are used to determine which Word is the right match. During training, you can play with different sentences and establish patterns on what works and what doesn't and create additional rules to resolve these conflicts. Tagging parts of speech is a nested for loop. For each sentence and for each word. Look up the word in the dictionary and if listed once then check the rules and if there is a match then use it. If there is more t
Read more