Version 5.0 on Yosemite

Due to a mismatch between some Xcode settings and F-Stop info files, Mac App Store is reporting that F-Stop version 5.0 is able to run on Yosemite and App Store application on Yosemite could propose an upgrade to version 5.

Please do not upgrade as F-Stop version 5 is not compatible with Yosemite. In case you already upgraded, please restore version 4.1.4 from a backup.

A fix is ongoing for this problem and I apologize for any issue it caused. If you need more support on this matter, please drop me an email to bugs at zamioware.com

Update: Version 5.0 has been removed from the list of downloadable version. If you are on Yosemite and need to restore version 4.1.4 please follow actions described in this Apple support article https://support.apple.com/en-us/HT201377

Version 5.0 – Technical Details

F-Stop version 5.0 introduces several changes. In this post I’m going to explain reasons and technical details behind some of these changes.

OSX 10.11 El Capitan requirement

This requirement wasn’t planned at the beginning of version 5.0 brainstorming but, during WWDC 2015, Apple announced something I was waiting since 3 years: a lot of improvements in NSCollectionView. With these improvements, NSCollectionView is able to display images in a more flexible way than the traditional IKImageBrowserView used by F-Stop since version 1. Unfortunately Apple has decided to do not back-port these improvements to NSCollectionView running on Yosemite and this is only reason why F-Stop version 5 requires OSX 10.11 El Capitan.

New graphical interface for Image Browser

IKImageBrowserView, a venerable component developed by Apple a lot of years ago, was the foundation of F-Stop image browser. At the time I started writing F-Stop, I had to do a choice between 2 components available: IKImageBrowserView and NSCollectionView. I went for IKImageBrowserView for 2 main reasons:

  • Performance. IKImageBrowserView uses layers and it’s very fast in displaying a lot of images with beautiful animations
  • Ability to display multiple groups of images

NSCollectionView was lacking support for multiple groups of images and I found it slower than IKImageBrowserView. But during these years I found a lot of limitations in IKImageBrowserView too:

  • Customization: IKImageBrowserView is cell based. There is no way for IKImageBrowserView to use subviews to render images. Every image displayed has a fixed layout and there is a limited degree of customization in cell layout too.
  • No custom layouts: IKImageBrowserView is able to display images in a grid and only in a grid.
  • Bugs/Weird behaviours. Although IKImageBrowserView is layer based, I haven’t been able to draw additional layers consistently. I have tried for several months to add a video playback layer to a cell without any luck.
  • No improvements for years: Apple has forgotten this component since its inception. No new methods, nor improvements to its deficiencies in years.

During the same years, thanks to its iOS counterpart (UICollectionView), NSCollectionView has gained a lot of focus. This year, during WWDC 2015, Apple announced some important improvements: Ability to display different groups of images within the same NSCollectionView and the possibility to use custom layouts. Time to ditch IKImageBrowserView…

Implementation of a new layout engine to emulate Flickr explore page hasn’t been easy and I have to credit this guy and its blog post for pointing me in the right direction. In addition, refactoring all F-Stop to use NSCollectionView was harder than I expected. Anyway… I did it and this is how F-Stop looks like now:

Main View version 5.0

 

Is NSCollectionView perfect? Not yet. I have found some limitations such as a more complex way to deal with selections (keyboard and drag&drop). And I’m still looking for a way to solve some crashes when images are added using insertItemsAtIndexPaths: method. To avoid these crashes, F-Stop reloads images every time but animations are ugly. But I’m confident that Apple will solve these defects in the future because NSCollectionsView is (loud and clear today) the way to go.

There is another subtle graphical change I have introduced. Windows have a full size content view. In simpler words, F-Stop title bar appearance is like the one seen in Safari when colored content is scrolled (blurred behind the title bar).

Full Content View

 

Improved performance of Photo Downloader. Downloads are executed concurrently with a progress bar for each download

There are some improvements to Photo Downloader too. The main one is that photo downloads are now executed concurrently: this increase download performance for broadband users. For each download there is also a progress bar to monitor the process.

Another improvement is related to failed downloads: Photo Downloader used to silently discard failed downloads of photos. This is not anymore: now F-Stop adds an error to its error window (describing problem occurred) and the download can be restarted once the problem is fixed.

Improved upload performance with concurrent uploads

During last year, I have received several complains regarding upload speed. Some broadband users were reporting slow uploads. Having a slow internet connection, I had hard times to understand the issues. What I found is that Flickr upload api performs some post-processing operations before returning a photo identifier (another way to say: “upload completed”). These operations include photo validation and resizing and (probably) metadata synchronization in the cluster and takes something around 10-15 seconds per upload.

For broadband users, a 10 to 15 seconds delay represents a huge increase of upload time. For example, for a 2MB file that takes 2 seconds to upload, entire upload takes (in the worst case) 17 seconds. That is 8.5 times!! So I have decided to allow for concurrent uploads. Users can tune this value in F-Stop preferences from a minimum value of 1 (serialized uploads good for slow internet connections) to 10 concurrent uploads (fast internet connections).

It’s important to note that concurrent uploads may not respect upload ordering defined in F-Stop upload sidebar. This could also affect photostream ordering in Flickr because Flickr orders photos using their upload time (except for Camera roll).

Improved editing of tags when Photos View is displaying a selection

Metadata Editor in Photos View gives users the ability to change tags of a photo or a selection of photos. F-Stop versions before 5.0 managed the latter case in the wrong way: all tags of a selection were replaced with the new set of tags specified in metadata editor. Imagine the following scenario

Selection of 3 photos:

Photo A – Tags: A,B,C

Photo B – Tags: D,E,F

Photo C – Tags: G,H,I

Setting tags X,Y,Z in metadata editor lead to the following result:

Photo A – Tags: X,Y,Z

Photo B – Tags: X,Y,Z

Photo C – Tags: X,Y,Z

Now in F-Stop version 5, tags field of Metadata Editor displays (for the selection above) the following tags: A,B,C,D,E,F,G,H,I. User can add new tags and/or remove them. Adding X,Y,Z and removing E produces the following result:

Photo A – Tags: A,B,C,X,Y,Z

Photo B – Tags: D,F,X,Y,Z

Photo C – Tags: G,H,I,X,Y,Z

Removed support for Google Maps

F-Stop 5.0 doesn’t use Google Maps anymore to display user photos on a map. Apple map is the only choice available. Google Maps in F-Stop used to work using a WebKit view rendering a javascript content described in a file. With El Capitan there are additional restrictions to access this file and, honestly, I’m not very good with HTML and Javascript. Considering how Apple Maps have improved during the last couple of years, I believe that this is not going to hurt anyone.

Removed all deprecated methods of old releases of OSX including NSURLConnection

Every year Apple introduces so many changes to its platform that new deprecated methods are a fact life when an application is recompiled using the latest and the greatest developer tools. Compiler embedded in Xcode 7 emitted more that fifty deprecated methods warnings with F-Stop version 4.x… time for a definitive cleanup. Generally speaking, replacement of a deprecated method is easy: there is a new method to call. Nothing more, nothing less. But with OSX 10.11 El Capitan NSURLConnection is going to be deprecated and this is huge. Its replacement, named NSURLSession, is more efficient, more elegant, more sophisticated and in the most of cases very easy to be introduced as replacement for NSURLConnection. In most of cases apart one: synchronous connections. NSURLSession has no synchronous mode of operation: everything is managed asynchronously.

Unfortunately is not always easy to deal with an asynchronously logic. Especially if you have spent your last 5 years to use NSOperationQueue with synchronous operations to perform all your background tasks. A lot of people out of there argues that synchronous logic is evil. I disagree. Synchronous is the perfect fit for sequential operations if you can perform them in background. Synchronous sequential operations are easier to understand, easier to debug and, if there is a real reason to have them sequentially executed, performance difference is negligible in comparison to asynchronous operations.

In any case F-Stop uses now NSURLSession everywhere but it costed me a lot of headaches. I hope Apple will reintroduce Synchronous methods to NSURLSession.

Apple Photos and F-Stop integration

Some days ago, a user notified me a new different behaviour of Apple Photos app in comparison to iPhoto when dragging photos to a batch in F-Stop. With new Apple Photos app, nothing happens. After some investigations, I have discovered that new Photos app is more complex than before when copying photos.

You can check it by yourself giving a look at the screenshot below.

Photos and Mail

Photos and Mail

 
What is the difference between these 2 images? I can assure that they are the same photo stored in Photos.app copied to mail.

Well, in the first case (upper image) the photo is copy/pasted. Under the hood, Photos app sends a File URL to target application (in this case mail). This File URL references an existing file on disk which contains a thumbnail (1024px) of the original image. This file is stored inside Photos app library.

In the second case the photo is drag/dropped; Under the hood, Photos app sends (again to mail) a File Promise. A File Promise is a file which doesn’t exist during drag&drop session but it’s created later (generally asynchronously) in a path specified by target application. This File Promise will become eventually a new file on disk that can be read by target application (again mail).

At a first glance, this approach is a bit awkward. Performing what, on the user side, is the same operation (have the same image on 2 different applications), leads to completely different results.
In addition to that, the use of Files Promise has a lot of other important side effects:

1) A proliferation of duplicate files on disk. Imagine you want to drag&drop a lot of photos (may be thousand) from Photos app to another application: doing this, is going to increase (potentially double) space occupation of photos.
2) Creating these copies has also a great impact on CPU because each photo is rendered again in an independent file. Do a test by yourself dragging thousand of photos from Photos app to a finder folder. You’ll see your mac sustain a 100% utilization (with a huge impact on battery life if your using a laptop)
3) These copies are not temporary. It’s up to the destination application an eventual removal.

Now, speaking about F-Stop, there are additional problems because I engineered it to upload EXISTING files. So when you add a file to a batch, F-Stop immediately checks for its existence on disk. But, as a File Promise is managed asynchronously, it could happen that the file still doesn’t exist when F-Stop tries the addition. And while I could think to apply a workaround for this behavior, it’s far harder to apply a workaround if File Promise is still a promise when the upload batch is started.
Last but not least, F-Stop should remove all these temporary files but it cannot do immediately because it should still allow a retry in case of upload failure. So it should clean these files on exit or when a file is manually removed from a batch. Both cases have their own tradeoffs too.

Probably all of above are not an issue if a user is uploading a small batch of files. But F-Stop users use it to batch upload thousand of photos and I’m not confident in messing up the upload engine to support this change in photos.app.

So, considering all of the above, if 1024px resolution is fine for your needs, I suggest to use copy/paste to copy photos from Photos.app to F-Stop instead of drag&drop. Otherwise export images manually to a finder folder, then upload using F-Stop.

Details about Incremental Uploads

Up to version 4.0.5, F-Stop was able to perform incremental uploads. There was, however, a significant limitation. Incremental upload information about what was successfully uploaded or not, was stored in memory. As memory is volatile, this information was not preserved among F-Stop runs leading to duplicate file uploads when running, for two times in a row, the same folder upload.

Since version 4.1.0, F-Stop overcomes this limitation storing information locally on your hard drive. This is an opt-in/out feature that is activated within F-Stop preferences.

Incremental Uploads Preferences

How it works

To begin, this option has no meaning for single files added to a batch as they are usually added for a specific need. Instead, it works for folders. When this option is enabled, F-Stop will slightly change its behaviour during folders uploads.

Current directory upload process can be simplified as follows: when a batch starts, a new background thread is spawned. This thread performs several administrative tasks such as checking for other running upload threads. Then, if everything is ok, the thread builds a file list of files to upload for each folder present inside the batch itself.

Up to version 4.0.5 this file list was built creating 2 different sets. The first set contained all files stored inside the directory. The second set contained all files already uploaded to Flickr. From the subtraction of set 1 and set 2 F-Stop determined which files needed to be uploaded.

Since version 4.1.0, this process is almost the same but set 2 is stored locally and it’s reread every time a batch starts. Furthermore, this “information store” grows every time a file is successfully uploaded. In this way, it’s always up-to-date with the current batch progress.

N.B. F-Stop will behave as version 4.5.0 and below if “Allows incremental upload” preference is disabled. This preference can be changed at any time but changing this while upload batches are running can have some side effect. Please disable/enable it while no uploads are running

Information stores are placed inside this directory:

~/Library/Containers/com.zamioware.F-Stop/Data/Library/Application Support/com.zamioware.F-Stop/uploadStatus

There a 2 kind of stores:
The first one store is a file with a iue extension used to keep track of uploaded files for each folder. Store file name is a generated name built from the following components:

  • Flickr account nsid
  • MD5 Hash of full directory path
  • Legible directory name

You see that there is a strict correlation between Flickr user name and directory path. In this way, F-Stop can keep track of different uploads of the same folder in multiple accounts. MD5 hash of full path guarantees, instead, that 2 different folders with the same name are not treated as they were the same.

The second information store is a file with sfe extension. This store contains information about user settings applied to a folder (in upload view). This settings are:

Name, Description, Tags, Groups, Photosets, Safety Level, etc.

SFE files are read when a folder is added to a batch so that user settings can be restored.

N.B. F-Stop will read/write SFE only if “Allows incremental upload” preference is enabled. SFE file is stored when at a least one file is successfully uploaded to Flickr.

Managing Incremental Uploads

Incremental uploads can be managed within F-Stop Preferences. To do this, simply click Details button on the right of “Allows incremental upload” preference. When you click details a new window appears:

Incremental Uploads View

 

Within this view, you can see all information stored by F-Stop for incremental uploads. For each folder there are generally 2 different information store identified by “File Type” column. “Folder Information” are SFE files described above. “Files Upload DB” are IUE files described above.

From this interface you can remove files you don’t need anymore. Be careful that, removing “Folder Information” will prevent F-Stop to populate automatically fields for this folder in upload view, while removing “Files Upload DB” will prevent F-Stop to make incremental upload for this folder. In other words, a full upload will run again.

Performance consideration

All tests I have made using my developing machine (Macbook Pro, Early 2011, 2,3 GHz Intel Core i5, Samsung 840EVO SSD Hard Drive, 8GB RAM) show negligible performance impact in incremental upload calculation for a folder with 100.000 files. Something around 3 to 5 seconds every time a batch is started.

My suggestion is to avoid upload of huge directories with hundred of thousand of files.

Caveats

Also with Incremental Upload feature, F-Stop cannot be considered a synchronization utility. This because F-Stop doesn’t keep track of relationship between local file names and remote Flickr photos NSID.

Primary effect of this is that F-Stop will not try to re-upload photos uploaded and subsequently removed from Flickr.  To illustrate better the implication of this, imagine the following scenario:

You upload directory named “A”. Directory A contains file photo1, photo2 and photo3. After a full, completed upload, you remove photo2 from Flickr by mistake. Then you decide to start again upload of directory “A” in the assumption that F-Stop will upload photo2 again.

Well, this is not going to happen! you have to manually re-upload photo2.