Context: We are working on a data migration project involving the synchronization of data between an on-premises NFS filesystem and an Azure NFS-based file share. The goal is to ensure a seamless transition from the on-premises environment to Azure while maintaining data integrity and efficiency.
Background:
Source: On-premises NFS filesystem.
Destination: Azure NFS-based file share.
Data Size: Approximately 350GB.
Tool Usage:
AzCopy (Not Supported): We initially attempted to use AzCopy for data migration but discovered it is not supported with Azure NFS file systems.
Rsync (Storage Growth Issue): We then turned to Rsync for data synchronization. However, we encountered significant storage growth at the Azure destination, and the process was never completed. The storage kept increasing without apparent reason, forcing us to abandon the Rsync process.
Fpsync (Successful First Attempt): To address the storage growth issue, we transitioned to Fpsync for data synchronization. In the first attempt, it showed promise as it successfully completed the initial synchronization.
The Issue: Unexplained Storage Growth: Our primary challenge is the unexplained growth in storage utilization at the Azure NFS destination, especially with Rsync. Even when the source data size remains the same, the destination storage increases significantly, making the process unmanageable.
Objective: We are seeking insights, advice, or solutions from the community to help troubleshoot and resolve this storage growth issue. Our goal is to ensure efficient data synchronization and minimal storage usage at the Azure destination.
Additional Information: The source data, including hidden files and directories, is correctly formatted and named.
Permissions are preserved during synchronization.
While we had initial success with Fpsync for the first synchronization, subsequent synchronizations still exhibited storage growth issues. Any suggestions, insights, or experiences related to this issue would be greatly appreciated. We are looking to resolve this challenge and ensure a successful data migration to Azure NFS.
Update:
Now i have used the rclone utility and hit the same issue.