Impact of Unstructured Data

What are some of the things that unstructured data impacts? The list is continually growing but the data must be transformed from unstructured to structured data. There are tools on the market (either ones you will need to buy or might already have) that will support this requirement. In an earlier post, I talked about prototyping and this transformation would be a good candidate. Data integration needs will change in that unstructured data would need to be transformed into structured data and data discovery techniques would need to change. Is traditional ETL the best route or would extract, load and then transform work better? Would you only need evolutionary change that may put another letter in the traditional DI techniques be enough or would a revolutionary change be needed? Is there a better way that has not yet been implemented? How would this effect and change data capture you have implemented? How you do data modeling changes?. Currently most data arrives to the data warehouse in structured form already so most traditional methods work. Since you probably want the same level of professional management for your unstructured data as your structured data and the same level of service to the end-user, you probably want to be able to put the unstructured data into the data warehouse. How do you do that in a way that supports access as a part of your overall BI program and whole document access? Does this effect what your data warehouse team needs to know about handling unstructured data? Think what new training these people need.

Thoughts about Unstructured Data

Most BI/DW environments are supported by a robust technology stack for structured data however are not well suited for supporting semi-structured/unstructured data. Does this mean that existing investments will be replaced? I don't think so since modeling such data into more structured data formats can often be automated and the process is well-known. Many data warehouses are built on database systems that have XML in the database, capabilities to index unstructured data, built-ins for simple parsing and publicly available tools to support more complex requirements. All of these provide the ability for semi-structured and unstructured data to be stored in the current data warehouse and thus managed in the same way that the structured data is managed. There are several end-user tools that also meet many of these requirements either as dedicated BI tools or office tools that can save unstructured documents in semi-structured forms. This data can now be cleansed, moved, backed-up, searched and otherwise managed as structured data in the data warehouse.