Sync Logic

Now the magic that will sync everything.

Process is simple:

1. Download data after we have saved in out local db.

2. Upload all isDirty data to Online DB (Firestore).

3. Add/Update this device installation id to Devices collection in Firestore.

4. Upload db file to S3 using Big Brother logic.

1. Login

After user log-in to his phone, first thing we will do is, we restore his DB from S3 Bucket.

After downloading, we will read DownloadedTracker, in next sync cycle, we will download whatever is missing from this restored DB.

2. Download Data

1. Get all devices that uploaded after you last time uploaded by getting max(downloadedTill) from DownloadedTracker .

2. If you DO NOT find any device, then you can stop your sync as no other device is probably in use. Put a entry in downloadedTracker table with max(updatedAt) for all tables one by one.

3. If you find other devices,

4. Read downloadedTill from DownloadedTracker for your the collection you are trying to download.

5. Download data from server with: SELECT * FROM table_name where updatedOn > downloadedTill AND updatedBy != myInstallationId

6. Save each entry one by one, update documents already present using serverId field.

7. Do this for all other collection/tables

   • Make sure you download parent data before download a child document, for example you should download a Note before you download its Comment document.

3. Upload Data

1. Find all dirty documents using isDirty=true field.

2. Upload documents one by one.

3. Make sure you fire create request only when serverId field is null.

4. After a successful upload, you have to save id returned by server in serverId field on your local DB data.

5. Repeat this process for every collection you have.

   1. Again, make sure to upload parent document before you upload child document.

Sometimes, we face a problem where, after S3 DB restore, a document is missing from local DB but is not actually a new document, locally we a similar document but does not have serverId attached to it. In such cases, we should have a fall-back field, in our case we had a combination of createdOn and uploadedOn.

If we found a document with same createdOn and updatedOn, we assume this document is same and we can overwrite server content on this document.

Now we can move on to upload our DB file to S3 bucket or wherever you want to store them.

This process is simple, you can encrypt your DB file (or not, who am I to judge).

But, now we have a problem, each device is independent and does not know which device has latest data. We only want to upload a DB with latest data because while restoring, we only want download minimal amount and avoid heavy download.

We will read more on this in Big Brother Strategy.