Data, data, everywhere… The flood of data from sensors, controls, inspections, movements, etc. is overwhelming. There is gold in this data, but it is only valuable if mined. To do so requires work, intelligence, the proper tools and the right processes.
The first thing to do is to collect your data. This process starts by extracting the data from whatever source it originates from. Most data is accessible wirelessly these days which greatly reduces the effort to extract it. When you collect data, make sure that you are also keeping any metadata (data about the data – date and time stamps, device, process, etc.). Metadata often contains information that can help you find patterns that you might not otherwise uncover.
Data can come in a variety of formats – Five, 5, 5.0, .085 hrs., one work week, etc. To make data useful, it needs to be transformed into a metric that has meaning across the analysis platforms. A cursory look at downstream applications that might process or use the data will help you determine what format your data should be stored in. Some data is generated with high levels of precision but not all of that precision is valuable, taking up storage and display space without adding sufficient value. On the other hand, in some cases, the extra level of data precision can unlock valuable information. Experience provides wisdom so ask if you have any concerns.
The final step in the data preparation process is to load it into a data storage system. An operational data store is usually a database that can hold data from multiple sources and use that data for reporting, controlling and decision making. Once your data is stored in an operational data store, it can be further processed and used to populate a data warehouse or other database designed to allow for deeper analysis. The speed of access, the security of the location, the quantity of data and the capability of the storage system are all important factors in choosing where and in what system to store data. Once the data is stored it can be migrated to other systems, used by multiple applications and be the source for one version of the truth.