How NEO Platform Reimagines Data Excellence Through a Unified Data Source
Introduction: In a world where data serves as the lifeblood of technology operations, the need for cohesive and reliable data management is paramount. Our tech-ops team, situated at the heart of the company's expansive suite of products, recognized the urgent requirement for a platform that not only supports the vast business infrastructure but accelerates innovation within our robust ecosystem.
Executive Summary
Thoughtworks strive to be a data informed organization. Inconsistent data across multiple systems and countries was threatening the quality of our data and creating trust issues amongst business stakeholders due to data inconsistency. MasterData was developed in 2021, it is a unified data storage via API Platform to NEO, Thoughtworks Award winning engineering portal. Currently supports 27 key master data. The benefit is an enhancement of accuracy and consistency of data across systems, a reduction of effort spent in reconciliation of inconsistent data and agility to change demonstrated by smooth data integration of ‘Organizational Structure’ & ‘Department List’ after the Thoughtworks restructuring in 2023.
Challenges
Data asymmetry
As our business rapidly and continuously develops, our system size grows accordingly. Consequently, the issue of data asymmetry (inconsistency of data across multiple systems and countries) has gradually surfaced. This challenge threatens the scalability and sustainability of business unit data, posing risks to our expanding business system.
Operational inefficiencies
Inconsistencies in master data across various systems and countries can result in operational inefficiencies. Interactions between different systems can be challenging or even disrupted, leading to high manual maintenance costs and low operational efficiency. This makes expansion difficult, weakens the ability to respond quickly to business changes, and can waste both financial and IT resources.
Solution:
In the face of growing complexity within our organization, TechOps team has embarked on a transformative journey to streamline and fortify our engineering operations.
After recognizing the critical importance of data consistency and accessibility, in 2021, we quickly integrated an MVP version of the master data management system with minimal effort, based on the existing engineering platform, API management, event source, and infrastructure services. This system aids our internal organization in continuously enhancing the seamless operation and management of master data. It’s carefully designed, allowing business stakeholders to manually curate master data, ensuring its maintenance as an authoritative source, and then propagating it through APIs and events, enabling downstream systems to easily consume the source master data. This accurately ensures the propagation of master data in all dependent systems. At the same time, when business data changes occur, this system can automatically update to reflect these changes, ensuring that all dependent systems remain synchronized and up-to-date.
The master data management system( named MasterData) enhances accuracy and consistency in TechOps, ensuring easy data access for business scalability. For Thoughtworks, it supports agility, compliance, and responsiveness to market changes. Furthermore, it builds a resilient data infrastructure to withstand environmental shocks, fostering sustainable development.
What we did:
Begin by developing the MasterData MVP in an agile manner, refining the data entry and editing functions.
Set up communication between the MasterData and the API & EVENTS platform to facilitate data transfer between systems.
Identify and promptly upload any unattended manual master data within the organization.
Masterdata played a pivotal role in the restructuring event at the end of 2023. Data managers efficiently updated the new organizational structure data through MasterData. They then synchronized it in real time to multiple downstream systems of TechOps. This process eliminated the need for extensive communication and manual changes, thus demonstrating significant benefits.
In the future, we aim to continually enhance the collaboration permission function for data management personnel, allowing business staff to directly and swiftly manage and maintain Master data.
Benefit:
Since the system went online, 27 pieces of key data information spanning multiple systems have been collected so far. It also brings considerable benefits.
Data Integrity and Consistency: Centrally managed data eliminates discrepancies across systems, ensuring all business operations utilize accurate and up-to-date information.
Innovation and Agility: With a shared data infrastructure, engineers can rapidly develop and deploy new features, adapting quickly to market demands or operational needs.
Cross-functional Collaboration: Unified data systems foster better teamwork across departments by simplifying communication and coordination on data-driven projects.
Operational Efficiency: Streamlined data access and management cuts down on the time and resources previously wasted in reconciling data across multiple platforms.
Risk Mitigation: Reduces the potential for errors and misalignments in critical business operations, lowering the risk of costly downtime or reputational damage.
Scalability: Has the ability to promptly adapt and provide support to rapidly evolving business needs and data.
Return of Investment
Consider the creation of an office scenario as an example. Without the MasterData system, adding new office data requires multiple system teams to spend nearly 60 person-days resolving inconsistent data problems across different systems. This single maintenance task is costly. However, with the MasterData system in place, the process of uploading, updating, and synchronizing business data to downstream systems can be completed in just 2 person-days.
Our centralized engineering platform strategy, coupled with a robust MasterData system, has yielded significant investment returns, both tangible and intangible. By consolidating our data resources, we've significantly reduced data communication time. Improved data accuracy and reliability have cut operating costs by reducing redundancy and error correction work. Better data quality has led to smarter decision-making, optimizing performance and enhancing customer satisfaction. Furthermore, this approach has facilitated scalable growth without a corresponding increase in operating costs, ensuring future expansion with minimal additional investment. These cumulative benefits have quickly offset the initial investment in the MasterData system.
The total development cost of the MasterData system is approximately 250 person-days.
The time to recover this investment is tied to the frequency of data updates. Therefore, the investment can be recouped after just four business data update requirements.
Since its launch three years ago, the MasterData system has collected 27 sets of data, undergone multiple modifications, and been maintained regularly. It played a crucial role in the organizational restructuring at the end of 2023.