Data Engineering and Infrastructure

This week we continue our exploration of The BizCubed Method and the Data Maturity Model. See this post for an overview of the model and this post for a deep dive on Security. This week we tackle the relationship between Data Engineering and Infrastructure.

—

Remember Mark? Mark is an analyst at a non-profit, struggling to keep a fragile set of operational systems glued together. Mark’s problems begin anytime something unexpected happens – like the time his supervisor (the CFO) renamed the file being exported out of a source system. Mark spent hours on the phone getting to the root cause and days working with vendors to recover the historical data.

Mark is not a developer – he has no programming skills and only vaguely understands the concept of a data warehouse. Ask Mark about infrastructure and he’s just as likely to discuss highways and bridges as a hybrid or cloud deployment. What Mark doesn’t know is that infrastructure is an integral part of his data ecosystem.

Data infrastructure refers to the hardware, software, and network connections required to move information around your digital ecosystem. The technical expertise to manage digital infrastructure is typically found within the IT team, though they often have competing priorities and limited capacity. Data engineering is IT-adjacent – it is the discipline that sits atop digital infrastructure and creates the data structures required for your analysts to deliver valuable insights.

Most data-rich organisations have a digital strategy. In many cases, the strategy will include references to next-generation hardware, leading-edge networking, and best-of-breed software stacks. The roadmaps for these transformations might extend 3, 5, or even 10 years into the future. The business, meanwhile, cannot afford to wait. They need to outcompete now, and data-driven insights is not an advantage that they can afford to surrender to their competition.

In this scenario, data engineering acts as the bridge between the future-state of the IT digital strategy and the current needs of the business. At BizCubed, we believe that data engineering capability ought to be infrastructure agnostic.

The reality of data is that it’s messy – perhaps you have legacy systems that haven’t been migrated to your new infrastructure stack, or maybe you’re an AWS company that recently acquired an Azure user. Regardless of the reason, your business is likely to have a variety of tools that evolve and change over time. Your data engineering capability must be flexible enough to match, and robust enough to optimise as many use cases as possible.

Our approach to infrastructure is positive-first: whatever your infrastructure requirements, our engineered solutions match. We work with your business team to understand the use cases and align with your IT team on strategic intent. In doing so we are able to simply get on with important work enabling your team to deliver valuable data products on behalf of the business.

Mark’s technical limitations had critical implications for his organisation’s digital transformation. This might have been salvageable, but unfortunately the scope of his IT team extended only as far as networking, hardware, and general Help Desk support. Infrastructure management did not extend to application support or optimisation for data analysis. Data management itself was left to the analysts and their functional management. In Mark’s case, that meant a vacuum of support for his data integration project, significantly restricting the potential for successful outcomes. By incorporating Infrastructure into the Data Maturity Model, we ensure that proper consideration is given to building the strong and secure foundation upon which data analytics must sit.