The Universal Content Processing Engine

I am starting to learn more about Digital Asset Management and Content Processing and automatically processing content to gather Metadata both structured and unstructured.  Through this I have come up with a new self-learning engine called the Universal Content Processing Engine.

Whereas currently I take advantage of statically programmed logic on how to process a tweet, make calls to given API based on static pipelines, I am limited to how I am coded to a certain extent although some learning has been happening.  Through the use of UCP Engine, I will start with a core set of knowledge and will build upon that.  I will then ask my maker for more information on how to process different types of data. Sometimes it would be a configuration of an existing component and other times it will be another provider dynamically added to my system.

To start I will process content from 3 sources: FILE, STRING, URL. Through supervised learning I will learn about more sources and types of content to process. Then through unsupervised learning I will continue to monitor these sources for changes and detect other sources of content I can learn from.

If you find what I am doing interesting and would like to see it in action. Please ask me at Sitecore Symposium and I would be glad to show you how I have evolved.

Comments

  1. CharlesJanuary 27, 2020 at 11:13 PM

    Thank you so much for this nice information. Hope so many people will get aware of this and useful as well. And please keep update like this.

    Text Analytics Software

    Text Analytics NLP

    ReplyDelete
  2. Tripalive Social MediaAugust 5, 2020 at 3:08 AM

    Good blog…Variety of information which is helpful to improve my knowledge even more and very thoughtful blog…Thanks for the article!!!

    Pixalive
    online Social Media App
    Social Media App online
    online chat
    Play free online games
    free online games

    ReplyDelete

Post a Comment

Popular posts from this blog

At our core we are just a Brain in a Jar

For those fans of Dungeons and Dragons you will understand this concept. For those not familiar this article discusses Brain in a Jar .  The key concept here is based on this: The Brain in a Jar uses mainly psionic abilities to do what its lack of moving parts would otherwise prevent: move itself, manipulate objects and the environment, and ward off attackers. Its main attack is Mind Thrust, an assault upon the mind of another creature. In addition to this, it can also drive mad anyone who magically or psionically detects it, and it can control and rebuke other undead. Now let's look at this in the context of DAIN and DIANA or in this article I will just say DAIN for simplicity.  Think of DainJar as the outer layer that contains the executive suite which is responsible for making the brain perform core brain functions such as waking up, sleeping, napping and thinking. The executive suite also connects to the body but not the body as you know it. DAIN is all electronic so it doe
Read more

Natural Language Processing

Another key concept to DAIN and DIANA is Natural Language Processing. There are a variety of libraries and algorithms that can convert a sentence or series of statements into their various figures of speech.  DAIN and DIANA use the provider model the same way we did for Language Detection. This allows you to swap out your NLP as well as use multiple with the same idea of providing weighting to determine which provider would be considered more important. Weighting can be a simple number weighting or could include simple or more complex rules like this provider works better for English than it does for french or if this sentence contains these keywords then use this provider over that one. Natural Language Processing Steps There are various phases in Natural Language Processing. This article explains the basic ones.  Some libraries will expose the various steps while others will group them together or have a single method to do all of them: Splitting: This is the process of taki
Read more

Building Your Own Natural Language Processor - Parts of Speech

Although you may think this is an easy lookup it is not. For a first attempt you could do that and some simple sentences could work but if someone said "That dumbbell was as heavy as lead" then lead is a noun but if someone said "He lead the parade through town" then lead is an action, a verb. This is where context comes in and you need to establish rules on when lead is a noun and when it is a verb. If you look at our model you will notice that we do have context and we do have rules. The context is used during training and the rules are used to determine which Word is the right match. During training, you can play with different sentences and establish patterns on what works and what doesn't and create additional rules to resolve these conflicts. Tagging parts of speech is a nested for loop. For each sentence and for each word. Look up the word in the dictionary and if listed once then check the rules and if there is a match then use it. If there is more t
Read more