Enterprise Engineering Insights at Scale

PISR: Problem, Impact, Solution, Result

Problem: Commonwealth Bank of Australia, an enterprise financial services organisation, was embarking on a large-scale DevSecOps transformation across 12 capability streams but lacked fundamental visibility into their engineering practices. They had no source of truth for basic questions like "how many repositories exist?", "who owns our digital products?", or "what's our actual engineering maturity?" Previous attempts by emerging technology teams had failed due to scale and complexity challenges.
Business Impact: This visibility gap severely hampered CBA's DevSecOps transformation and APRA regulatory reporting capabilities. Without baseline metrics, they couldn't measure programme effectiveness, identify improvement opportunities, or demonstrate compliance progress. The bank discovered that 90% of their digital assets had unknown ownership - a critical risk for a regulated financial institution.
Our Solution: Over 18 months, ClearRoute's team of 7 QCEs partnered with CBA to architect and build the Engineering Insights Platform (EIP) - a comprehensive data platform aggregating metrics across all engineering practices. We implemented a serverless data lake on AWS using Data Vault 2.0 modelling, integrated 10+ critical source systems including GitHub, JIRA, ServiceNow, and cloud platforms, and established self-service analytics capabilities.
Tangible Result: The transformation delivered CBA's first enterprise-wide engineering metrics platform, providing visibility into hundreds of thousands of digital assets and enabling data-driven decision making across their DevSecOps transformation. The platform now directly supports APRA regulatory reporting requirements and drives strategic decisions across all 12 capability streams, fundamentally changing how CBA approaches engineering improvement.

The Challenge

Business & Client Context

Primary Business Goal: Enable successful DevSecOps transformation across CBA through data-driven insights and measurement, whilst meeting increasing APRA regulatory reporting requirements for technology risk management.
Pressures: Large-scale organisational transformation timeline pressure, regulatory compliance deadlines, need to demonstrate measurable progress to executive stakeholders, and previous failed attempts creating scepticism about feasibility.
Technology Maturity: Highly fragmented data landscape with multiple siloed systems, no centralised engineering metrics, inconsistent tooling across teams, and limited visibility into actual engineering practices versus reported practices.

Current State Assessment: Key Pain Points

Visibility Gap: Fundamental questions about the engineering estate were unanswerable - "How many repositories exist at CBA?", "How many engineers work at CBA?", "What percentage of every squad is an engineer?", "Who owns our digital products?"
Measurement Absence: No baseline metrics for DevSecOps maturity, making it impossible to measure transformation progress or programme impact across 12 capability streams.
Data Quality Crisis: Discovery revealed 90% of digital assets in ServiceNow had unknown ownership, creating massive operational and compliance risks for a regulated bank.
Regulatory Pressure: APRA reporting requirements around technology changes and risk profiles couldn't be satisfied without fundamental engineering visibility.
Organisational Resistance: Teams worried that measurement would be used for performance evaluation rather than improvement, creating adoption challenges.

Baseline Metrics (Where Available)

Metric Category	Baseline	Notes
Engineering Visibility	Unknown	No source of truth existed
Asset Ownership	~10% known	90% of ServiceNow items had unknown owners
Data Sources	Siloed	Multiple disconnected systems
Transformation Measurement	None	No way to measure DevSecOps progress
Regulatory Reporting	Manual/Incomplete	Significant APRA compliance challenges

Solution Overview

Engagement Strategy & Phases

Phase 1: Discovery & Foundation (April-June 2024): Mapped existing data landscape, identified key stakeholder requirements across 12 capability streams, and established architectural principles for serverless, scalable platform.
Phase 2: Platform Architecture (June-September 2024): Designed data lake architecture using Data Vault 2.0, established AWS serverless infrastructure, and began integration with highest-impact source systems.
Time to First Value: Delivered first production beta in December 2024 with GitHub, JIRA, and initial ServiceNow data, enabling basic engineering visibility after 8 months.
Phase 3: Data Integration Scale (December 2024-March 2025): Integrated critical systems including cloud platforms, TeamForm organisational data, and additional ServiceNow modules.
Phase 4: Production & Adoption (March-July 2025): Public release with SSO access, comprehensive dashboards, and self-service capabilities for all CBA engineering teams.
Phase 5: Enhancement & Evolution (Ongoing): Continuous expansion of data sources, advanced analytics capabilities, and integration with transformation programmes.

Architectural Overview

Before State

After State

Technical Architecture Deep Dive

Data Pipeline Design Philosophy

Our pipeline follows a clear separation of concerns across three layers:

Acquisition Layer: Raw extraction with zero transformation. "Acquisition does no transformations. It doesn't do anything to the data other than pick it up from where it is and dump it in S3." This preserves source system fidelity and allows us to handle any file format (CSV, Parquet, JSON blobs) without loss.

Staging Layer: Standardisation and "data munging" - converting everything to Apache Iceberg format while making decisions about nested data depth. For JSON blobs with 100 nested fields, "we probably don't need to unnest all 100, so this is where we sort of make that decision at staging."

Vault Layer: Business entity modelling using Data Vault 2.0, providing stable core entities that can handle schema evolution without breaking existing data structures.

Service Domain Access Control Strategy

We implemented the BYON (Bring Your Own Network) model from banking industry standards, breaking the bank into immutable service domains (e.g., home lending, credit card transactions). This creates natural access boundaries where "you can't join data, clean data, touch data in between service domains. All the access is always at a service domain level." This approach aligns with banking's move away from reporting lines toward logical business divisions.

Metrics Discovery & Collaborative Process

A critical aspect of our approach was the collaborative process for determining what to measure. Rather than imposing metrics top-down, we worked with each capability stream to identify meaningful indicators. "The capabilities we go off and get the metrics that we think that they should be measuring or they think they should be measuring. So that's a collaborative process we sort of recommend you should measure this this what are the things that you want to drive and we'll start to measure that."

This collaborative approach was essential because teams had legitimate concerns about measurement being used for performance evaluation. We had to "start to decouple the measurement from performance, because again, some of the measurement is just to measure the impact that these things are having. It doesn't necessarily have a reflection on the effort being put in by the team."

The key was establishing trust by focusing on programme effectiveness rather than team performance, and designing metrics that would drive beneficial behaviour even if teams tried to optimise for them specifically.

Source System Prioritisation: "Bang for Buck" Strategy

Rather than starting with technically interesting but narrow systems, we prioritised based on metric breadth and stakeholder impact. For example, ServiceNow covers all 12 capabilities to some extent and contains every incident, service item, and change - providing massive value in one integration. Compare this to Zephyr (testing data only) which, while interesting, delivers limited organisational impact. As Kai noted: "Your stakeholder's going to go great, you have Zephyr data. Not super useful, but if you say hey I have ServiceNow data which has every single incident, every single service item, every single change, that's an awesome win."

Serverless-First Architecture Philosophy

We made a core architectural decision that everything must be serverless to minimise operational overhead for the team. While this can increase cost overhead sometimes, it eliminates VM and server management completely. This proved crucial for a 7-person team supporting enterprise-scale data processing across multiple time zones.

Build vs Wait Decision: CDO Platform Timing

CBA's Chief Data Office was building their own data platform, but it wasn't ready and would take another 12 months. We couldn't wait for the DevSecOps transformation timeline, so we built our own platform designed to integrate with their future data mesh architecture. This "build now, integrate later" approach required additional complexity but delivered immediate value.

Data Vault 2.0 for Schema Evolution

We chose Data Vault 2.0 modelling specifically to handle the reality that "schemas change super regularly" in technology environments. Unlike traditional star schema approaches, Data Vault allows core business entities and relationships to remain stable while field types in hubs and satellites can evolve. This proved essential for maintaining data quality across diverse, evolving source systems.

File-Based Data Lake Strategy

Everything exists as flat files in S3 with no in-memory storage, making the solution very cost-effective on storage. Our ETL processes read from S3, transform, and output back to S3, minimising compute costs while providing enterprise scale capabilities.

QCE Disciplines Applied

Platform Engineering: Delivered a scalable, serverless data platform using AWS managed services, eliminating infrastructure overhead whilst providing enterprise-grade reliability and security for a regulated banking environment.
Developer Experience: Created self-service analytics capabilities allowing engineering teams to access insights about their own practices without requiring data engineering expertise, dramatically improving adoption and value realisation.
Quality Engineering: Embedded data quality controls throughout the pipeline using Data Vault 2.0 modelling principles, ensuring reliable insights for critical business decisions and regulatory reporting.

The Results: Measurable & Stakeholder-Centric Impact

Headline Success Metrics

Metric	Before Engagement	After Engagement	Improvement
Engineering Visibility	0%	100% across 10+ systems	Complete transformation
Data Sources Integrated	0	10+ critical systems	Full ecosystem coverage
Asset Ownership Visibility	~10%	90% identification rate	9x improvement
Regulatory Reporting	Manual/Incomplete	Automated APRA compliance	Risk mitigation achieved
DevSecOps Measurement	None	12 capability streams	Transformation enabled

Value Delivered by Stakeholder

For the CTO / Transformation Leaders:
- Enabled data-driven DevSecOps transformation across 12 capability streams with measurable progress tracking (transformation_enablement: "100% capability coverage")
- Achieved APRA regulatory compliance through automated reporting capabilities, reducing regulatory risk (compliance_achievement: "Automated APRA reporting")
- Provided enterprise-wide engineering visibility for strategic technology decisions (strategic_insight: "Complete engineering estate visibility")
For Capability Stream Leaders:
- Delivered specific metrics for each capability area, enabling targeted improvement programmes and resource allocation (programme_effectiveness: "Data-driven capability improvement")
- Provided leading indicators (like ObStack adoption) that correlate with lagging indicators (like system reliability) (predictive_analytics: "Leading indicator measurement")
- Enabled evidence-based programme prioritisation and impact measurement (decision_quality: "Metrics-driven programme decisions")
For Engineering Teams:
- Created self-service access to team-specific insights without requiring data engineering skills (self_service: "Team-level analytics access")
- Provided visibility into practices like PR merge times (average 1.5 days) and JIRA hygiene for continuous improvement (team_insights: "Practice visibility and improvement")
- Enabled teams to understand their contribution to broader DevSecOps transformation goals (alignment: "Team-to-transformation connection")
For Risk & Compliance Teams:
- Delivered comprehensive asset ownership visibility, addressing critical operational risk (risk_mitigation: "90% asset ownership identification")
- Automated APRA technology risk reporting, reducing manual effort and improving accuracy (compliance_automation: "Automated regulatory reporting")
- Provided audit trail and data lineage for regulatory requirements (audit_readiness: "Complete data governance")

Platform Economics

Platform Operating Cost: $20,000/month for comprehensive enterprise-wide engineering insights
Value Comparison: Replacing manual regulatory reporting efforts alone justifies platform investment
Scale Achievement: Supporting 40-person team across multiple geographies with serverless architecture
ROI Realisation: Enabling data-driven transformation decisions across $10B+ technology estate

Lessons, Patterns & Future State

What Worked Well

Source System Prioritisation Strategy: Focusing on systems with highest metric coverage (like ServiceNow covering all capabilities) rather than individual interesting metrics delivered maximum value per integration effort.
Serverless Architecture Choice: Using AWS managed services (Glue, Athena, S3) eliminated infrastructure overhead and provided automatic scaling for enterprise data volumes.
Data Vault 2.0 Modelling: Handling schema evolution and maintaining data quality across diverse source systems proved critical for long-term platform sustainability.
Early Risk Engagement: Involving risk and compliance stakeholders from September onwards ensured architectural decisions supported regulatory requirements from the start.

Challenges Overcome

Data Mesh Integration Complexity: The Chief Data Office had conflated marketplace and platform design in their data mesh approach. They didn't build an extensible platform where teams could "bring your own" solutions. We had to work with them over 12 months to position EIP as a BYO consumer whilst integrating with their centralised marketplace, essentially forcing them to decouple these concerns.
Measurement vs Performance Anxiety: Teams worried metrics would be used for performance evaluation and bonuses. Our solution was designing "gameable" metrics that drive desired behaviour even when gamed. For example, measuring "number of SLOs in ObStack" - teams might game it, but they're still onboarding to ObStack, achieving our goal. As Kai explained: "you want a metric that people can game... the right choice of metric means that even if people game the system, you're getting the outcome you want."
AppFlow vs Control Trade-offs: Initially chose AWS AppFlow for speed of development and cost benefits, but learned it trades off troubleshooting accuracy and control. For critical APRA reporting where "you need to get it right every time," this proved unacceptable. We moved to Glue ETL for better control and troubleshooting capabilities despite increased complexity.
Leading vs Lagging Metrics Strategy: Rather than just measuring DORA metrics (lagging indicators), we focused on "bleeding metrics" or leading indicators. For example, measuring ObStack onboarding rates as a leading indicator of reliability improvements. This gives programme teams specific levers to pull and helps squads understand "what do I need to do" to improve, not just "what should improve."
Timeline vs Quality Pressure: With multiple GMs demanding data "yesterday," we had to resist the temptation to "just pump out raw data." Instead, we invested time in proper data modelling, knowing that "longer term, your quality is massively higher, but obviously you have to spend the time and investment into your data."

Key Takeaways for Similar Engagements

Start Before Transformation: This platform should have been built before the DevSecOps programme began. "I think this should have been a core activity before the dev SEC OPS programme to start to actually have targeted pieces of work and to actually measure the impact of work being done." Having baseline metrics from day one would have significantly improved transformation programme effectiveness.
Source System ROI Analysis: Always prioritise integrations based on metric breadth over technical novelty. "It's gonna take me the same amount of effort... if I go after ServiceNow, that's probably the same amount of time as it will take for me to go get something very niche like Zephyr data... ServiceNow covers pretty much everything at the bank."
Design Gameable Metrics: Create metrics that drive desired behaviour even when teams try to game them. The goal is choosing metrics where "even if people game the system, you're getting the outcome you want." Focus on leading indicators that teams can actively improve.
Compliance Architecture Impact: Risk and compliance requirements materially impact architecture decisions (like data lineage requirements). "Having a dedicated risk person who we said look what do we need to do, what do we need to fill out and we keep them in the loop" proved essential. Engage these stakeholders during design, not implementation.
Build vs Wait Strategic Decision: When enterprise platforms aren't ready, consider building with integration in mind rather than waiting. We couldn't wait 12 months for CDO's platform, so built our own while planning integration with their future data mesh.
Early Customer Discovery: "If we could go out before we started this programme of work and find these customers, it's a massive win to be to already have those people aboard rather than people coming out of the woodwork saying great, you have this data." Identify consumers before building to prioritise effectively.

Replicable Assets Created

Serverless Data Lake Architecture: Complete AWS-based pattern for enterprise data platform with Data Vault 2.0 modelling
Engineering Metrics Framework: Methodology for identifying and measuring leading vs lagging indicators across DevSecOps capabilities
Source System Integration Patterns: Reusable approaches for integrating GitHub, JIRA, ServiceNow, and cloud platform data
Self-Service Analytics Framework: Pattern for enabling business users to access technical insights without data engineering expertise
Regulatory Compliance Data Architecture: Approach for ensuring data lineage and audit trails meet banking regulatory requirements

Client's Future State / Next Steps

With the Engineering Insights Platform now in production, CBA has fundamentally changed their approach to DevSecOps transformation from opinion-based to data-driven. The platform directly supports their ongoing APRA regulatory reporting requirements and is being used to drive strategic decisions across all transformation programmes. The established Center for Enablement is scaling these measurement patterns to additional business units, and the platform architecture provides a foundation for advanced analytics including predictive modelling of transformation success factors.

Organisational Impact Pattern

This engagement demonstrates how comprehensive engineering measurement platforms can transform large-scale organisational change programmes. The key success factor was treating measurement not as overhead but as fundamental infrastructure for transformation success - similar to how observability platforms are essential for production systems reliability.