Details about Incremental Uploads

Up to version 4.0.5, F-Stop was able to perform incremental uploads. There was, however, a significant limitation. Incremental upload information about what was successfully uploaded or not, was stored in memory. As memory is volatile, this information was not preserved among F-Stop runs leading to duplicate file uploads when running, for two times in a row, the same folder upload.

Since version 4.1.0, F-Stop overcomes this limitation storing information locally on your hard drive. This is an opt-in/out feature that is activated within F-Stop preferences.

Incremental Uploads Preferences

How it works

To begin, this option has no meaning for single files added to a batch as they are usually added for a specific need. Instead, it works for folders. When this option is enabled, F-Stop will slightly change its behaviour during folders uploads.

Current directory upload process can be simplified as follows: when a batch starts, a new background thread is spawned. This thread performs several administrative tasks such as checking for other running upload threads. Then, if everything is ok, the thread builds a file list of files to upload for each folder present inside the batch itself.

Up to version 4.0.5 this file list was built creating 2 different sets. The first set contained all files stored inside the directory. The second set contained all files already uploaded to Flickr. From the subtraction of set 1 and set 2 F-Stop determined which files needed to be uploaded.

Since version 4.1.0, this process is almost the same but set 2 is stored locally and it’s reread every time a batch starts. Furthermore, this “information store” grows every time a file is successfully uploaded. In this way, it’s always up-to-date with the current batch progress.

N.B. F-Stop will behave as version 4.5.0 and below if “Allows incremental upload” preference is disabled. This preference can be changed at any time but changing this while upload batches are running can have some side effect. Please disable/enable it while no uploads are running

Information stores are placed inside this directory:

~/Library/Containers/com.zamioware.F-Stop/Data/Library/Application Support/com.zamioware.F-Stop/uploadStatus

There a 2 kind of stores:
The first one store is a file with a iue extension used to keep track of uploaded files for each folder. Store file name is a generated name built from the following components:

  • Flickr account nsid
  • MD5 Hash of full directory path
  • Legible directory name

You see that there is a strict correlation between Flickr user name and directory path. In this way, F-Stop can keep track of different uploads of the same folder in multiple accounts. MD5 hash of full path guarantees, instead, that 2 different folders with the same name are not treated as they were the same.

The second information store is a file with sfe extension. This store contains information about user settings applied to a folder (in upload view). This settings are:

Name, Description, Tags, Groups, Photosets, Safety Level, etc.

SFE files are read when a folder is added to a batch so that user settings can be restored.

N.B. F-Stop will read/write SFE only if “Allows incremental upload” preference is enabled. SFE file is stored when at a least one file is successfully uploaded to Flickr.

Managing Incremental Uploads

Incremental uploads can be managed within F-Stop Preferences. To do this, simply click Details button on the right of “Allows incremental upload” preference. When you click details a new window appears:

Incremental Uploads View

 

Within this view, you can see all information stored by F-Stop for incremental uploads. For each folder there are generally 2 different information store identified by “File Type” column. “Folder Information” are SFE files described above. “Files Upload DB” are IUE files described above.

From this interface you can remove files you don’t need anymore. Be careful that, removing “Folder Information” will prevent F-Stop to populate automatically fields for this folder in upload view, while removing “Files Upload DB” will prevent F-Stop to make incremental upload for this folder. In other words, a full upload will run again.

Performance consideration

All tests I have made using my developing machine (Macbook Pro, Early 2011, 2,3 GHz Intel Core i5, Samsung 840EVO SSD Hard Drive, 8GB RAM) show negligible performance impact in incremental upload calculation for a folder with 100.000 files. Something around 3 to 5 seconds every time a batch is started.

My suggestion is to avoid upload of huge directories with hundred of thousand of files.

Caveats

Also with Incremental Upload feature, F-Stop cannot be considered a synchronization utility. This because F-Stop doesn’t keep track of relationship between local file names and remote Flickr photos NSID.

Primary effect of this is that F-Stop will not try to re-upload photos uploaded and subsequently removed from Flickr.  To illustrate better the implication of this, imagine the following scenario:

You upload directory named “A”. Directory A contains file photo1, photo2 and photo3. After a full, completed upload, you remove photo2 from Flickr by mistake. Then you decide to start again upload of directory “A” in the assumption that F-Stop will upload photo2 again.

Well, this is not going to happen! you have to manually re-upload photo2.

Posted in Technical details and tagged , , , .