Migrating on-premises data to Databricks can be a complex and time-consuming process. However, with proper planning and execution, the benefits of leveraging the power of the cloud can be significant. This post will discuss some best practices for successfully migrating on-premises data to Databricks.
At the outset of planning the migration process, it is crucial to understand the current state of your on-premises data. This includes identifying the data sources and format, as well as any potential roadblocks or edge cases.
The next step is cataloging the data to be migrated. This should include the following:
· Data sources
· Formats
· Schemas
· Synchronization requirements
Once this is compiled, it will help shed light on the project's scope and allow it to be broken into phases. Smaller phases help validate the process and identify unknown issues early in the project timeline.
After the catalog of data sources is created, the next step is to set up a timeline for each phase and a strategy for the initial load of data and the ongoing data sync. The requirements for the ongoing sync vary based on the business needs and can go from immediately streamed to hourly, daily, weekly, etc. It is imperative to validate with the business the requirements for the freshness of the synchronized data.
Security and efficiency are two factors that cannot be overlooked and should be planned and prioritized from the beginning. Encryption and authentication are a must.
There are many ways to get data into Databricks. Exported files (CSV,JSON, etc...), native tools, third-party connector platforms, and custom-developed applications. Each scenario is different and having an experienced partner to help guide using best practices is invaluable.
Identifying where the integration process runs is critical. A key distinction is whether it needs to be run on-premise to push data or from a cloud provider that can pull data. This will depend on your network security and how and where your data is currently stored.
A monitoring strategy is a valuable piece of the system. Identifying as soon as possible if something fails or data comes through inaccurate or missing is key to maintaining the integrity of the warehouse.
When moving data into Databricks, it is simpler to move the data as is without transforming it during this phase. This makes updating and adding new data easier and comparing it with source data for validation. The transformation should happen in the next phase, inside the Databricks environment.
Create an intuitive and organized naming convention for the structure within the Databricks lakehouse. This will save time and minimize the risk of errors when kept clean and organized.
Once the data has been migrated to Databricks, it is important to validate the accuracy and completeness of the data. It is also essential to regularly monitor the migrated data to ensure that it remains accurate and up-to-date.
We recommend utilizing a Medallion Architecture in Databricks. Learn more here.
Medallion Architecture – Databricks
When the raw data is ingested into the Databricks platform, it is considered in a "Bronze" state. The next step is to clean, transform, join, and possibly aggregate the data into a "Silver" state. From the "Silver" state, the data is typically integrated into a data warehouse-friendly structure, such as a star schema for consumption by end users. This transformation step is a vast topic that is out of the scope of this article.
With a solid plan, several technologies can be used to get on-premises data into Databricks, including data integration platforms, cloud data transfer services, and secure network connections. The specific approach will depend on the characteristics of the data and the infrastructure of the on-premises environment.
