In October 2012, The Harvard Business Review announced that the “Data Scientist role is the Sexiest Role of the 21st century”(1). This bold, and probably intentionally provocative, statement got them lots of attention.
The title “data scientist” has only been around a few years and is used to describe a high-ranking professional with the training and curiosity to make discoveries in the world of big data. A data scientist has the skills of a mathematician, statistician, programmer, database analyst and MBA. Industry analysts observe this mix is so rare in the real world people now refer to these near-mythical data scientists as “Unicorns”(2,3,4).
Data is the New Oil
When Anne Winblad, respected Silicon Valley venture capital investor was asked, “What’s the next big thing?” She replied “Data is the new Oil”. (5)
She’s not the first person to use this phrase, but it was interesting to see the idea has gained acceptance with people who specialise in the next big thing.
The metaphor goes like this “Data is just like crude. It’s valuable, but if unrefined it cannot really be used. It has to be changed into gas, plastic, chemicals, etc., to create a valuable entity that drives profitable activity; so must data be broken down, analysed for it to have value.” (6)
Data Science vs Data Analysis
If scientists are the smart guys in the white coats running experiments in their lab and occasionally making discoveries, then data engineers are the professionals running the factories that turn raw data into valuable commodities, on a continuous basis, and at commercial scale.
At BizCubed, we do believe data is the oil of the 21st century, but we don’t believe that data scientists are necessary to unlock the value of information. In our experience, there are already large numbers of people in organisations who are already doing 90% of the work. Analysts are typically from business backgrounds, so they don’t have the detailed programming knowledge attributed the unicorn data scientists.
However, the data analytics software industry has recognised this gap, and produced a wide range of tools to help bridge that technical gap. Because of the perceived value in the results, these tools are also often prohibitively expensive to buy and deploy at scale. As a result, many organisations have hobbled their skilled analyst assets and are limiting the value that can be generated.
Refining Data Oil into Fuel for Growth
Extending the oil analogy further, data is not just the black gold with intrinsic value, but with refinement, it is the fuel driving growth to 21st century businesses. Refining data into insight that organisations can use to make good business decisions takes a broad mix of skills.
Organisations that are best able to convert the raw material into consumable products will have a dramatic competitive advantage.
Drilling for Oil
There are multiple sources of data oil in any business; typically in the finance systems, often in the CRM system and increasingly in the online marketing tools. Many of these tools are in different places, some in the corporate network, and others in the cloud. Where the data scientist may take a sample of data from each source, the data engineer builds the pipeline infrastructure to ensure there is a reliable flow from each well to the refinery.
In the big-data world, this is known as data integration, and tools like Pentaho Data Integrator (PDI) allows non-IT data analysts to use a graphical drag and drop interface to connect to a wide range of data sources blend this data together. PDI includes standard connectors for all the typical data sources such as Excel, CSV, MS Access, and big data sources like Hadoop and NoSQL, but more importantly also includes connectors for applications like Salesforce.com.
The reason that BizCubed works with products like Pentaho, is these tools enable all capable business analysts to demystify the data analytics process, and become superb data engineers.
If organisations invest their money in unicorns – they may get small quantities of the best fuel in the world, but will miss the opportunity to build an organisation that has the ability to constantly refine fuel at scale, cost effectively and faster than their competition.
Check out how simple Pentaho’s drag-and-drop graphical interface extracts information from Salesforce.com with our Salesforce Data Integration kit.