Menu
close

Data Architecture with Databricks Transforms Credit Risk Engine

How a modern data strategy drove efficiency, governance, and cost reduction

In a context of increasing regulatory demand and competitiveness in the financial sector, building a robust data strategy is no longer a differential but essential.

It was in this scenario that a Large Financial Institution in Brazil relied on act to modernize its credit risk engine through an advanced data architecture based on Databricks.

The project was structured with a focus on Data Intelligence, capable of supporting dozens of models, allowing not only compliance with regulatory standards such as standard 4966, but also substantial gains in operational efficiency, data governance, and cost optimization.

The Challenge: Regulatory Complexity and Data Architecture Fragmentation

Regulation 4966 introduced new requirements for calculating PDD (Allowance for Doubtful Accounts) and PE (Expected Loss), in line with the national adoption of IFRS 9. For the financial institution, this translated into a large-scale challenge:

  • Creation from scratch of data pipelines for the credit risk engine, reconciling multiple data sources with different standards and architectures.
  • Development and modernization of more than 30 machine learning models, ensuring that all were adhering to the new calculation rules and regulatory documentation.
  • Entry into production in just 2 months, a critical deadline to meet legal requirements without compromising the credit operation.

In addition to the regulatory challenge, there were structural issues:

  • Fragmented legacy processes, with low automation and heavy reliance on manual treatments.
  • Lack of a central data platform that would support scalability, governance, and intensive use of ML models.

Solution: Data Intelligence Platform with Databricks

act digital implemented a modern data architecture platform based on Databricks, enabling data centralization, processing, and governance at scale.

This approach was instrumental in enabling a value-driven data strategy, supporting both analytics and advanced machine learning.

Data Architecture Implemented

act digital designed and implemented a cloud data platform (AWS), with Databricks as the core of the analytical processing layer and operationalization of credit risk models. The solution integrated:

  • Databricks@AWS as a unified environment for data engineering, data science, and running the more than 30 ML models in production.
  • Trino (EMR/K8S) as a distributed query engine over the Data Lake.
  • Data Lake in S3 as the central repository for the entire credit data lifecycle.
  • MLflow as a core component of MLOps, supporting model versioning, traceability of experiments, and deployment automation.

Two-Phase Implementation

Phase 01 – AS-IS implementation on the new platform

In the first phase, the priority was to ensure operational continuity and compliance within the regulatory deadline:

  • AS-IS mapping and migration of data processing and modeling flows to the new platform.
  • Orchestration of end-to-end workflows in Databricks, connecting ingestion, treatment, feature engineering and model scoring.
  • Integration with existing credit risk management systems, ensuring that the transition was transparent to the business areas.

Phase 02 – Optimization, MLOps, and FinOps

With the platform in place, the second phase focused on scale, efficiency, and governance:

  • Refining and optimizing the models to better exploit Databricks' distributed computing capabilities.
  • Application of good development practices and MLOps conveyors, with automated training, testing, approval and deployment pipelines.
  • Implementation of continuous monitoring of model performance (drift, accuracy, SLAs) and resource consumption, with a structured FinOps view.

This architecture consolidated Databricks as the core platform for the credit engine data lifecycle.

Results: Operational efficiency, data governance and cost reduction

Adopting the Data Intelligence platform with Databricks has brought measurable gains in process efficiency, costs, and data maturity.

1. Process Efficiency

90% reduction in processing time:

The execution time of flows in AWS/Databricks has dropped from more than 1 day to just 1 hour.

This allowed you to:

  • Increased frequency of calculation of PDD and PE.
  • Faster reaction to portfolio changes and risk behavior.
  • Reduced windows of downtime and pressure on operational teams.

2. FinOps & Financial Efficiency

87% reduction in total processing cost after Phase 02 optimizations.

The combination of Databricks, fine-tuning cloud resources, and FinOps practices has made it possible to:

  • More efficient utilization of clusters and workloads.
  • Elimination of processing redundancies and pipeline duplication.
  • Greater cost predictability for the financial and IT area.

3. Data Culture

The initiative went beyond the technical dimension and drove the wider adoption of the Data Platform within the institution:

  • It motivated the beginning of a structured data literacy journey, with an increase in the understanding and use of data by the business areas.
  • It stimulated the creation of new analytical use cases on the same platform, in addition to the credit engine.

4. Governance and Security

The new architecture made it possible to strengthen data governance:

  • Tighter access, traceability, and auditing controls, essential for regulated environments.
  • Greater observability of data processes, with monitoring of pipelines, data quality, and adherence to internal policies and external requirements.

Crosslink suggestions:

Conclusion: Data strategy as a lever for transformation

This case shows how a structured approach to data architecture and data strategy can transform regulatory challenges into opportunities for innovation and efficiency. By implementing a Data Intelligence platform with Databricks, act digital enabled the institution to:

  • Ensure compliance with complex regulatory standards
  • Scale your analytics efficiently
  • Dramatically reduce costs and processing time
  • Establish a solid data governance foundation
  • Evolve your data and analytics maturity

More than a technological modernization, the project consolidated a true data-driven transformation, aligned with act digital's positioning: transforming complex challenges into opportunities to generate value through scalable, intelligent, and business-centric solutions.

How a modern data strategy drove efficiency, governance, and cost reduction

In a context of increasing regulatory demand and competitiveness in the financial sector, building a robust data strategy is no longer a differential but essential.

It was in this scenario that a Large Financial Institution in Brazil relied on act to modernize its credit risk engine through an advanced data architecture based on Databricks.

The project was structured with a focus on Data Intelligence, capable of supporting dozens of models, allowing not only compliance with regulatory standards such as standard 4966, but also substantial gains in operational efficiency, data governance, and cost optimization.

The Challenge: Regulatory Complexity and Data Architecture Fragmentation

Regulation 4966 introduced new requirements for calculating PDD (Allowance for Doubtful Accounts) and PE (Expected Loss), in line with the national adoption of IFRS 9. For the financial institution, this translated into a large-scale challenge:

  • Creation from scratch of data pipelines for the credit risk engine, reconciling multiple data sources with different standards and architectures.
  • Development and modernization of more than 30 machine learning models, ensuring that all were adhering to the new calculation rules and regulatory documentation.
  • Entry into production in just 2 months, a critical deadline to meet legal requirements without compromising the credit operation.

In addition to the regulatory challenge, there were structural issues:

  • Fragmented legacy processes, with low automation and heavy reliance on manual treatments.
  • Lack of a central data platform that would support scalability, governance, and intensive use of ML models.

Solution: Data Intelligence Platform with Databricks

act digital implemented a modern data architecture platform based on Databricks, enabling data centralization, processing, and governance at scale.

This approach was instrumental in enabling a value-driven data strategy, supporting both analytics and advanced machine learning.

Data Architecture Implemented

act digital designed and implemented a cloud data platform (AWS), with Databricks as the core of the analytical processing layer and operationalization of credit risk models. The solution integrated:

  • Databricks@AWS as a unified environment for data engineering, data science, and running the more than 30 ML models in production.
  • Trino (EMR/K8S) as a distributed query engine over the Data Lake.
  • Data Lake in S3 as the central repository for the entire credit data lifecycle.
  • MLflow as a core component of MLOps, supporting model versioning, traceability of experiments, and deployment automation.

Two-Phase Implementation

Phase 01 – AS-IS implementation on the new platform

In the first phase, the priority was to ensure operational continuity and compliance within the regulatory deadline:

  • AS-IS mapping and migration of data processing and modeling flows to the new platform.
  • Orchestration of end-to-end workflows in Databricks, connecting ingestion, treatment, feature engineering and model scoring.
  • Integration with existing credit risk management systems, ensuring that the transition was transparent to the business areas.

Phase 02 – Optimization, MLOps, and FinOps

With the platform in place, the second phase focused on scale, efficiency, and governance:

  • Refining and optimizing the models to better exploit Databricks' distributed computing capabilities.
  • Application of good development practices and MLOps conveyors, with automated training, testing, approval and deployment pipelines.
  • Implementation of continuous monitoring of model performance (drift, accuracy, SLAs) and resource consumption, with a structured FinOps view.

This architecture consolidated Databricks as the core platform for the credit engine data lifecycle.

Results: Operational efficiency, data governance and cost reduction

Adopting the Data Intelligence platform with Databricks has brought measurable gains in process efficiency, costs, and data maturity.

1. Process Efficiency

90% reduction in processing time:

The execution time of flows in AWS/Databricks has dropped from more than 1 day to just 1 hour.

This allowed you to:

  • Increased frequency of calculation of PDD and PE.
  • Faster reaction to portfolio changes and risk behavior.
  • Reduced windows of downtime and pressure on operational teams.

2. FinOps & Financial Efficiency

87% reduction in total processing cost after Phase 02 optimizations.

The combination of Databricks, fine-tuning cloud resources, and FinOps practices has made it possible to:

  • More efficient utilization of clusters and workloads.
  • Elimination of processing redundancies and pipeline duplication.
  • Greater cost predictability for the financial and IT area.

3. Data Culture

The initiative went beyond the technical dimension and drove the wider adoption of the Data Platform within the institution:

  • It motivated the beginning of a structured data literacy journey, with an increase in the understanding and use of data by the business areas.
  • It stimulated the creation of new analytical use cases on the same platform, in addition to the credit engine.

4. Governance and Security

The new architecture made it possible to strengthen data governance:

  • Tighter access, traceability, and auditing controls, essential for regulated environments.
  • Greater observability of data processes, with monitoring of pipelines, data quality, and adherence to internal policies and external requirements.

Crosslink suggestions:

Conclusion: Data strategy as a lever for transformation

This case shows how a structured approach to data architecture and data strategy can transform regulatory challenges into opportunities for innovation and efficiency. By implementing a Data Intelligence platform with Databricks, act digital enabled the institution to:

  • Ensure compliance with complex regulatory standards
  • Scale your analytics efficiently
  • Dramatically reduce costs and processing time
  • Establish a solid data governance foundation
  • Evolve your data and analytics maturity

More than a technological modernization, the project consolidated a true data-driven transformation, aligned with act digital's positioning: transforming complex challenges into opportunities to generate value through scalable, intelligent, and business-centric solutions.

Related

chevron-right