A scalable Message Queuing Service that supports asynchronous message-based communication between clients and the Synchronization Service instances best fits the requirements of our application. we just need to make sure we meet the data consistency.Īn important part of our reference architecture is a messaging middleware that should be able to handle a substantial amount of reads and writes. The Metadata Database can be a relational database such as MySQL, or a NoSQL database. The Metadata Database is responsible for maintaining the versioning and metadata information about files/chunks, users, and workspaces.

Internal Database keeps track of the chunks, files, their versions, and their location in the file system Metadata Database.

Once the chunks are successfully submitted to the Cloud Storage, the Indexer will communicate with the Synchronization Service using the Message Queuing Service to update the Metadata Database with the changes.

Indexer processes the events received from the Watcher and updates the internal database with information about the chunks of the modified files.

A chunking algorithm can detect the parts of the files that have been modified by user and only transfer those parts to the Cloud Storage, saving on cloud storage space, bandwidth usage, and synchronization time. To reconstruct a file, chunks will be joined back together in the correct order.

Chunker splits the files into smaller pieces called chunks.

Watcher monitors the sync folders and notifies the Indexer of any action performed by the user for example when user create, delete, or update files or folders.

The main components of the desktop client are Watcher, Chunker, Indexer, and Internal DB as described below. Some of the most important requirements of the Desktop Client include upload and download of the files, detecting file changes in the sync folder, and handling conflicts due to offline or concurrent updates.

It also interacts with the backend Cloud Storage for storing the actual files. file name, size, modification date, etc.). The Desktop Client interacts with the Synchronization Service to handle file metadata updates (e.g.

Lets assume we have file upload client installed on computer/mobileĭesktop Client Application monitors the folders that are identified as workspace or sync folders and synchronizes them with the remote Cloud Storage.

Dropbox to google photos