Details

I started with the style transfer model described in the Real-Time Style Transfer paper, which proved too slow/computationally expensive for low power and edge devices.

To bring down the computational cost, I first replaced the convolutional layers with separable convolutional filters (as in MobileNet architectures). This allowed for significant reduction in computational cost at little to no significant loss to quality of the output.

To further bring down the cost and tune the models; model architecture search and pruning (as described in this paper) was employed. This led to another order of magnitude drop in the total number of parameters at the cost of some overall output quality for some styles (see mosaic). Finally, all the layers in the models that could be fused were fused and the models quantised to give the final optimised models.

All these techniques combined allowed for orders of magnitude reduction in model parameters (as shown in the table below) making realtime inference on low power devices feasible. All models were trained and optimised in Pytorch and then converted for deployment on TensorFlow.js .

Currently working on

• Deploying arbitrary style transfer model to allow for style transfer from any user defined image
• Deploying the demo as a React Native app

Credits

This demo was put together by Zawar Qureshi but could not have been done without the following:

• Authors of the Real-Time Style Transfer paper.
• Authors of the PSP net paper.
• The onnx2keras package.
• The TensorFlow.js library.

I'd love to hear from people interested in making tools/apps using these models!