Close

Style

Real-time style transfer

Real-time image stylisation using optimised deep style transfer models. You can select different styles in the Settings section to experiment with, please select video mode if you would like to see the model output in real-time.

Note: Since these models run locally on device, any uploaded pictures stay safely on the device. Please note that the first run may be much slower than subsequent runs as Tensorflow loads the necessary files. (WIP so there may be bugs!)

image mode:

video mode:


Settings:

Style:

Style quality:


Upload an image:

Enter URL:


Output:

Details

I started with the style transfer model described in the Real-Time Style Transfer paper, which proved too slow/computationally expensive for low power and edge devices.

To bring down the computational cost, I first replaced the convolutional layers with separable convolutional filters (as in MobileNet architectures). This allowed for significant reduction in computational cost at little to no significant loss to quality of the output.

To further bring down the cost and tune the models; model architecture search and pruning (as described in this paper) was employed. This led to another order of magnitude drop in the total number of parameters at the cost of some overall output quality for some styles (see mosaic). Finally, all the layers in the models that could be fused were fused and the models quantised to give the final optimised models.

All these techniques combined allowed for orders of magnitude reduction in model parameters (as shown in the table below) making realtime inference on low power devices feasible. All models were trained and optimised in Pytorch and then converted for deployment on TensorFlow.js .

Model Prams. GFLOPs @384x216 Size Size(quantised)
Original ~1.68M ~25.56 ~6.4MB ~3.2MB
Separable ~211.07K ~3.52 ~799KB ~399KB
Optimised ~22.2K ~0.74 ~87.4KB ~43.7KB

Currently working on

  • Deploying arbitrary style transfer model to allow for style transfer from any user defined image
  • Deploying the demo as a React Native app

Credits

This demo was put together by Zawar Qureshi but could not have been done without the following:

I'd love to hear from people interested in making tools/apps using these models!