Telecommunications Operator Revolutionizes Data Management with Migration to BigQuery

In a scenario where speed and efficiency in data management have become essential for business competitiveness, major telecommunications operators face the challenge of adapting their technological infrastructures to meet growing demands. This was the challenge faced by one of Brazil’s largest telecommunications operators, which decided to make a significant change in its data management system: migrating from the traditional SQL Server to the innovative and scalable BigQuery, a Google Cloud data storage solution.

Migrating to BigQuery is a solution for various challenges faced by companies, including the need to process and analyze massive volumes of data generated daily. Although SQL Server is effective in many aspects, it presents limitations in terms of scalability and flexibility, especially in dealing with the growing demands of Big Data. Choosing BigQuery overcomes these limitations and allows organizations to explore the advantages of a cloud infrastructure, such as reduced operational costs and greater agility in data processing.

However, a transition like this can be complex, especially considering the need to ensure data security and protection throughout the process. This is why the operator sought the support of act digital, which was responsible for conducting this project from start to finish, from planning to complete migration. The solution was carefully designed to ensure data integrity, minimizing service interruptions, risks, and costs.

With the migration to BigQuery completed, the operator now has a more agile and efficient data management system, favoring real-time strategic decision-making. This resulted in productivity and efficiency gains, with lower costs for maintaining the data infrastructure.

Client: Leadership and Pioneering in Telecommunications

Operating nationally, the telecommunications company that hired act digital to migrate the data management system is committed to innovation, customer experience, and agility. It is a company that has positioned itself as a leader in significant movements in the telecommunications market since its inception in Brazil.

Since 2015, it has led 4G coverage in the country, expanding its connectivity to rural areas, thus contributing to the innovation of agribusiness. As a pioneer in activating 5G networks in Brazil, the company continues to promote digital transformation in the sector.

In addition to its focus on technology and innovation, the company stands out for its environmental, social, and governance (ESG) practices. Its performance in this sphere has been recognized not only by Brazilian society, benefiting from these actions, but also by the capital market, attesting to the rigor and effectiveness of the adopted practices.

With significant operations in the country, the company places data at the center of its operations. Daily, its activities involve processing a broad set of information that needs to be stored and used securely, in compliance with the General Data Protection Law (LGPD). In this sense, migrating from SQL Server to BigQuery positively impacts both the operations and strategy of the organization.

Challenge: Complexity in Migrating from SQL Server to BigQuery

Migrating from SQL Server to BigQuery was the challenge faced by the company. Although offering numerous benefits, this transition presented significant challenges due to the inherent complexity of the process.

To ensure that the change would occur efficiently, generating the expected results, the act digital team developed the entire work framework, considering the challenges that needed to be overcome throughout the project, such as:

• Data schema compatibility
• Data volume and integrity
• Security and compliance
• Operational, performance, and financial optimizations

Solution: Data Bridge and Backup in System Migration

The act digital team, responsible for migrating data from SQL Server to BigQuery, adopted an agile, secure, and strategic approach to execute the transition. The team took on responsibilities ranging from developing the architecture to implementing the complete pipeline.

Methodology

The methodology employed in transitioning between data storage systems considered two distinct approaches. In the first stage, the team’s focus was on creating a “data bridge,” an interconnection solution developed to transfer data efficiently and securely between the two systems. Subsequently, the team developed an alternative backup method, an essential process to ensure data redundancy and security during migration.

• Version 1

The team developed a “data bridge” using Python with the pandas_gbq library, providing direct integration with BigQuery. Additionally, SQLAlchemy was used to extract data from SQL Server.

• Version 2

As an alternative backup method, the team implemented a system using Shell Script, centered on using the BCP utility, a SQL Server command-line tool. This utility allows data to be extracted directly from SQL Server to CSV files without occupying server memory, offering superior performance compared to using Python, although it requires more careful data handling and adds complexity to the process.

Technology

From planning to system migration, the technologies used were:

• Python
• SQL
• Shell Script
• Kanban

Results: Efficient Transition Between Systems

The successful migration from SQL Server to BigQuery provided significant results for the client, reflected in various areas of its operation. Executed efficiently, the project itself offered a solution with reduced costs. However, there are several other benefits for the operator.

- Resolution of critical challenges for cloud migration

The team overcame technical and operational obstacles, facilitating the transition from SQL Server to a modern and efficient cloud environment. The project was developed to address demands by optimizing resources and costs.

- Reduction of operational costs

Significant decrease in infrastructure and maintenance expenses, thanks to the use of efficient migration tools from the Google Cloud Platform (GCP).

- Improvement in efficiency and scalability

Increased operational efficiency using BigQuery, enabling faster and scalable data processing.

- Enhancement in data security and management

Improvements in data security and management with the adoption of advanced security practices and tools from GCP.

- Improvement in data analysis and decision-making

Enhanced capacity for data analysis, providing deeper insights and supporting data-driven decision-making.

How We Can Help Your Company

In the telecommunications and media sector, the demand for innovation and digital efficiency is relentless. Therefore, our approach is to offer solutions that not only meet current needs but also prepare companies for future challenges.

With specialized professionals who are well-versed in this sector, act digital is ready to assist companies in facing challenges similar to SQL to BigQuery migration, ensuring a safe, efficient, and innovative digital transformation.

We offer a wide range of solutions that are tailored to each project, encompassing its specific demands and exploring its maximum potential. See how we can help your company!

- Development of digital journeys: We create customized paths for digital transformation, ensuring that the transition to new technologies is efficient and aligned with business objectives.

- Streaming strategies: With the increasing demand for digital content, we help companies develop and optimize streaming strategies, ensuring efficiency in content delivery and a better user experience.

- Omnichannel culture: We implement an omnichannel approach, ensuring that businesses are prepared to interact with customers across multiple digital channels in an integrated and effective manner.

- Process and data security: Security is a priority in all our solutions. We focus on protecting data and processes, ensuring that the transition to platforms like BigQuery is secure and reliable.

- APIs and applications: We develop custom APIs and applications that help optimize operations, improve customer interaction, and enhance data analysis.

We are prepared for the future that is revolutionizing the media and telecommunications sector. Our expertise combines the adoption of agile methods with the implementation of innovative technologies to drive companies toward their business results.

Digital transformation has been unfolding at an increasingly rapid pace, demanding efficiency in technology transitions and data processing. That’s our specialty! If your company needs to advance in this process, talk to the act digital team.