ControlNet-Canny: Optimized for Qualcomm Devices

On-device, high-resolution image synthesis from text and image prompts. ControlNet guides Stable-diffusion with provided input image to generate accurate images from given input prompt.

This is based on the implementation of ControlNet-Canny found here. This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the Qualcomm® AI Hub Models library to export with custom configurations. More details on model performance across various devices, can be found here.

Qualcomm AI Hub Models uses Qualcomm AI Hub Workbench to compile, profile, and evaluate this model. Sign up to run these models on a hosted Qualcomm® device.

Getting Started

There are two ways to deploy this model on your device:

Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

Runtime Precision Chipset SDK Versions Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-8275 QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-6690 QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-9075 QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Mobile QAIRT 2.45, ONNX Runtime 1.27.1 Download
PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile QAIRT 2.45, ONNX Runtime 1.27.1 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-8275 QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-6690 QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-9075 QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Mobile QAIRT 2.45 Download
QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile QAIRT 2.45 Download

For more device-specific assets and performance metrics, visit ControlNet-Canny on Qualcomm® AI Hub.

Option 2: Export with Custom Configurations

Use the Qualcomm® AI Hub Models Python library to compile and export the model with your own:

  • Custom weights (e.g., fine-tuned checkpoints)
  • Custom input shapes
  • Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for ControlNet-Canny on GitHub for usage instructions.

Model Details

Model Type: Model_use_case.image_generation

Model Stats:

  • Conditioning Input: Canny-Edge
  • ControlNet Number of parameters: 361M
  • Input: Text prompt and input image as a reference
  • Model size: 1.4GB
  • Text Encoder Number of parameters: 340M
  • UNet Number of parameters: 865M
  • VAE Decoder Number of parameters: 83M

Performance Summary

Model Runtime Precision Chipset Inference Time (ms) Peak Memory Range (MB) Primary Compute Unit
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 29.115 ms 33 - 33 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 70.864 ms 352 - 352 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 59.84 ms 33 - 45 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-8275 71.106 ms 30 - 35 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 75.918 ms 30 - 32 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-9075 70.966 ms 30 - 34 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-X7181 70.864 ms 352 - 352 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-6690 1032.002 ms 2 - 13 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-7790 120.798 ms 32 - 42 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-8750 44.083 ms 31 - 44 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Mobile 44.083 ms 31 - 44 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 28.81 ms 35 - 42 MB NPU
controlnet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 120.798 ms 32 - 42 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 29.44 ms 2 - 2 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 70.992 ms 2 - 2 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 57.107 ms 2 - 10 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-8275 138.234 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 75.676 ms 2 - 17 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 70.414 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8650P 70.414 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8255P 70.414 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-9075 70.532 ms 2 - 18 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-X7181 70.992 ms 2 - 2 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-6690 893.459 ms 0 - 6 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-7790 120.358 ms 2 - 8 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-8750 44.001 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 138.234 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Mobile 44.001 ms 2 - 11 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 28.377 ms 2 - 10 MB NPU
controlnet QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 120.358 ms 2 - 8 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 1.519 ms 1 - 1 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 3.435 ms 157 - 157 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 2.306 ms 0 - 12 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-8275 3.197 ms 0 - 4 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 3.246 ms 0 - 162 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-9075 3.347 ms 0 - 3 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-X7181 3.435 ms 157 - 157 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-6690 22.092 ms 0 - 10 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-7790 3.694 ms 0 - 6 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-8750 1.753 ms 0 - 7 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Mobile 1.753 ms 0 - 7 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 1.552 ms 0 - 14 MB NPU
text_encoder PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 3.694 ms 0 - 6 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 1.725 ms 0 - 0 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 3.611 ms 0 - 0 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 2.313 ms 0 - 7 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-8275 7.504 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 3.247 ms 0 - 1 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 3.497 ms 0 - 10 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8650P 3.497 ms 0 - 10 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8255P 3.497 ms 0 - 10 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-9075 3.341 ms 0 - 2 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-X7181 3.611 ms 0 - 0 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-6690 21.993 ms 0 - 6 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-7790 3.67 ms 0 - 6 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-8750 1.746 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 7.504 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Mobile 1.746 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 1.542 ms 0 - 9 MB NPU
text_encoder QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 3.67 ms 0 - 6 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 46.703 ms 13 - 13 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 114.147 ms 829 - 829 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 82.833 ms 13 - 25 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-8275 106.904 ms 11 - 28 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 112.935 ms 0 - 883 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-9075 106.042 ms 13 - 28 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-X7181 114.147 ms 829 - 829 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-6690 2170.557 ms 13 - 24 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-7790 179.576 ms 13 - 24 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-8750 63.54 ms 6 - 17 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Mobile 63.54 ms 6 - 17 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 48.983 ms 6 - 17 MB NPU
unet PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 179.576 ms 13 - 24 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 46.911 ms 13 - 13 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 114.027 ms 13 - 13 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 80.549 ms 13 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-8275 266.608 ms 11 - 19 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 112.411 ms 13 - 15 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 107.12 ms 11 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8650P 107.12 ms 11 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8255P 107.12 ms 11 - 20 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-9075 106.917 ms 11 - 25 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-X7181 114.027 ms 13 - 13 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-6690 2069.299 ms 8 - 14 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-7790 179.277 ms 13 - 19 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-8750 61.869 ms 6 - 19 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 266.608 ms 11 - 19 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Mobile 61.869 ms 6 - 19 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 46.35 ms 6 - 15 MB NPU
unet QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 179.277 ms 13 - 19 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X2 Elite 126.952 ms 4 - 4 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® X Elite 258.679 ms 62 - 62 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Gen 3 Mobile 197.364 ms 3 - 11 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-8275 262.233 ms 3 - 7 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 262.765 ms 4 - 5 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-9075 240.571 ms 3 - 6 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ IQ-X7181 258.679 ms 62 - 62 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-6690 6315.343 ms 0 - 7 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-7790 505.592 ms 2 - 8 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Qualcomm® Dragonwing™ Q-8750 172.672 ms 3 - 15 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Mobile 172.672 ms 3 - 15 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 8 Elite Gen 5 Mobile 120.875 ms 3 - 10 MB NPU
vae PRECOMPILED_QNN_ONNX w8a16 Snapdragon® 7 Gen 4 Mobile 505.592 ms 2 - 8 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® X2 Elite 127.18 ms 0 - 0 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® X Elite 259.674 ms 0 - 0 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Gen 3 Mobile 197.427 ms 3 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-8275 711.8 ms 0 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ QCS8550 (Proxy) 262.561 ms 0 - 2 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8775P 243.768 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8650P 243.768 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA8255P 243.768 ms 0 - 9 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-9075 240.993 ms 0 - 3 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ IQ-X7181 259.674 ms 0 - 0 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-6690 6272.479 ms 0 - 7 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-7790 503.305 ms 1 - 7 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® Dragonwing™ Q-8750 171.99 ms 1 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Qualcomm® SA7255P 711.8 ms 0 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Mobile 171.99 ms 1 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 8 Elite Gen 5 Mobile 121.193 ms 0 - 10 MB NPU
vae QNN_CONTEXT_BINARY w8a16 Snapdragon® 7 Gen 4 Mobile 503.305 ms 1 - 7 MB NPU

License

  • The license for the original implementation of ControlNet-Canny can be found here.

References

Community

Usage and Limitations

This model may not be used for or in connection with any of the following applications:

  • Accessing essential private and public services and benefits;
  • Administration of justice and democratic processes;
  • Assessing or recognizing the emotional state of a person;
  • Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics;
  • Education and vocational training;
  • Employment and workers management;
  • Exploitation of the vulnerabilities of persons resulting in harmful behavior;
  • General purpose social scoring;
  • Law enforcement;
  • Management and operation of critical infrastructure;
  • Migration, asylum and border control management;
  • Predictive policing;
  • Real-time remote biometric identification in public spaces;
  • Recommender systems of social media platforms;
  • Scraping of facial images (from the internet or otherwise); and/or
  • Subliminal manipulation
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for qualcomm/ControlNet-Canny