The Importance of the Right Metadata at the right time

As I have been digging deeper into research of Digital Asset Management there is a lot of talk about Metadata and the importance of adding as much metadata as you can to assets.  

On a quick tangent or shameless plug, I have started DAM Guild as a mentoring community so we can learn more about Asset Management solutions and best practices managing our assets.  Here is how you can take part:

LinkedIn: Join Dam Guild LinkedIn group
Facebook : Join Dam Guild Facebook group
Twitter: Follow @DamGuild

After you join one of the above continue reading below. 

Now some people may see an asset as content or a product but metadata applies to any type of item both digital and physical.  In the realm of data science metadata is very important but if you let it an item can have millions of pieces of metadata associated with it and processing can be a lot.

For those that are familiar with data science they know that the metadata assigned to an item is called a feature and the challenge is to know which features area important and which ones can be ignored. Let's do a little simple math.  Let's say you assign 20 features to an item.  Now you have 1 million items to process.  That gives you 20 million pieces of metadata to process.  If you look at the data set and determine that 2 features (pieces of metadata) are irrelevant then you save yourself 2 million pieces of metadata to process.  However if you remove the wrong features then you taint the data set and get the wrong results.  

It is a fine line when it comes to removing features that is why you need an experienced data scientist to help. If you need help with that I can introduce you to a team that does that all the time.  

For the sake of DAIN and DIANA we collect all the metadata we can on every item then we tag each feature with its own metadata that describes when it is most useful. That helps us narrow down when to use which feature and is how we optimize.  Currently the feature metadata is created by a human but as we progress the metadata on a feature could become more automated as well.

If you are starting a data science related project it is important to be collecting all the data to start. Remember you can always throw data out but it is more difficult or most of the time impossible to collect it later. 

Comments

Post a Comment

Popular posts from this blog

Natural Language Processing

Another key concept to DAIN and DIANA is Natural Language Processing. There are a variety of libraries and algorithms that can convert a sentence or series of statements into their various figures of speech.  DAIN and DIANA use the provider model the same way we did for Language Detection. This allows you to swap out your NLP as well as use multiple with the same idea of providing weighting to determine which provider would be considered more important. Weighting can be a simple number weighting or could include simple or more complex rules like this provider works better for English than it does for french or if this sentence contains these keywords then use this provider over that one. Natural Language Processing Steps There are various phases in Natural Language Processing. This article explains the basic ones.  Some libraries will expose the various steps while others will group them together or have a single method to do all of them: Splitting: This is the proces...
Read more

Providers and Pipeline, Oh Why!!!

I have mentioned to a few people that DAIN and DIANA are built with providers and pipelines. The pipelines. Some providers are weighted while others run conditionally based on rules.  The question I get is often is why not just choose an algorithm and stick with it for simplicity.  There are a few reasons for that: There are 3 reasons to have different providers the first is Bias and the second is Context. Bias In the case of Natural Language Processing, you have heard a lot in the news about bias in AI and ML. Some of that is due to data which is the subject of another article but other times it is that the algorithm has been trained with data that is not specific to your domain and thus can produce bias that is not specific to your data. That is why it is so important to to use multiple algorithms to keep the other in check. You can weight one algorithm higher so it is most likely to produce the result but there are cases where it differs greatly from other algorithms t...
Read more

The Universal Content Processing Engine

I am starting to learn more about Digital Asset Management and Content Processing and automatically processing content to gather Metadata both structured and unstructured.  Through this I have come up with a new self-learning engine called the Universal Content Processing Engine. Whereas currently I take advantage of statically programmed logic on how to process a tweet, make calls to given API based on static pipelines, I am limited to how I am coded to a certain extent although some learning has been happening.  Through the use of UCP Engine, I will start with a core set of knowledge and will build upon that.  I will then ask my maker for more information on how to process different types of data. Sometimes it would be a configuration of an existing component and other times it will be another provider dynamically added to my system. To start I will process content from 3 sources: FILE, STRING, URL. Through supervised learning I will learn about more sources and ...
Read more