How to Deploy Edge AI: Quantization, Pruning, and ONNX for Mobile and IoT

Step-by-Step Guide

1. 1

   Understand Edge Constraints

   Edge devices have strict constraints: phones have 4-12 GB RAM and limited battery, IoT devices may have 256KB RAM and no OS, embedded GPUs (Jetson) have 4-16 GB shared memory. Your model must fit within these limits while meeting latency targets.

   💡

   Tip: Profile your model on the actual target device — emulators and simulators can be misleading.
2. 2

   Choose Your Optimization Strategy

   The three main optimization techniques are complementary: Quantization reduces precision (4-8x smaller), Pruning removes weights (2-10x smaller), Distillation trains a smaller model (10-100x smaller). Most production deployments use all three in sequence.

3. 3

   Implement Post-Training Quantization

   PTQ is the simplest optimization — convert FP32 weights to INT8 without retraining. PyTorch provides torch.quantization.quantize_dynamic for weights-only and quantize_fx for full quantization. Accuracy drop is typically 0.5-2%.

   python Copy

   import torch.quantization as quant
   
   model.eval()
   model = quant.quantize_dynamic(
       model,
       {nn.Linear, nn.LSTM},
       dtype=torch.qint8
   )
   # Model is now 4x smaller with <1% accuracy loss

4. 4

   Implement Quantization-Aware Training

   QAT fine-tunes the model with simulated quantization, allowing it to adapt to quantization noise. This recovers most of the accuracy lost in PTQ. Use this when PTQ accuracy drop exceeds your tolerance (>2%).

   ⚠️

   Warning: QAT requires a training pipeline and representative dataset. Budget 1-3 days of training time.
5. 5

   Apply Structured Pruning

   Structured pruning removes entire channels or layers, resulting in actual speedup on hardware. Unstructured pruning removes individual weights but requires sparse matrix support for speedup. Use torch.nn.utils.prune for PyTorch or TensorFlow Model Optimization Toolkit for TF.

   python Copy

   from torch.nn.utils import prune
   
   # Prune 30% of channels in all Linear layers
   for name, module in model.named_modules():
       if isinstance(module, nn.Linear):
           prune.ln_structured(module, name="weight", amount=0.3, n=2, dim=0)
           prune.remove(module, "weight")  # make permanent

6. 6

   Use Knowledge Distillation

   For maximum compression, train a smaller student model using the teacher's soft probabilities as targets. The student learns not just the correct answer but the teacher's confidence distribution, which transfers "dark knowledge" that improves generalization.

   💡

   Tip: DistilBERT (66% of BERT size, 97% of BERT performance) was created using knowledge distillation. The technique works remarkably well for transformer models.
7. 7

   Export to ONNX Format

   ONNX (Open Neural Network Exchange) is the standard format for cross-platform deployment. Export your optimized model to ONNX, then run it with ONNX Runtime which provides hardware-specific optimizations for CPU, GPU, NPU, and mobile.

   python Copy

   dummy_input = torch.randn(1, 3, 224, 224)
   torch.onnx.export(
       model, dummy_input, "model.onnx",
       input_names=["input"],
       output_names=["output"],
       dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
       opset_version=17
   )

8. 8

   Optimize with ONNX Runtime

   ONNX Runtime provides execution providers for different hardware: CUDA for NVIDIA GPUs, TensorRT for maximum GPU performance, CoreML for Apple devices, NNAPI for Android, and DirectML for Windows. Use session options to enable graph optimizations and select the best provider.

   

9. 9

   Benchmark on Target Hardware

   Always benchmark on the actual target device. Key metrics: latency (p50, p95, p99), throughput (inferences/sec), memory footprint (model + runtime), and power consumption. Compare against your requirements and iterate on optimization if needed.

   ⚠️

   Warning: Thermal throttling can cause 2-3x latency increase after sustained inference. Test under realistic thermal conditions.
10. 10

    Implement Model Versioning and OTA Updates

    Edge models need over-the-air updates. Implement: model versioning (semantic versioning), delta updates (only send changed weights), rollback capability, and A/B testing for new models. Use a model registry like MLflow or a custom solution with CDN distribution.

11. 11

    Handle Privacy and Security

    Edge AI can improve privacy by keeping data on-device. But the model itself may contain sensitive information (training data leakage). Use model encryption, secure enclaves (TEE), and differential privacy during training. For mobile, use iOS Keychain or Android Keystore for model storage.

12. 12

    Monitor Production Models

    Deploy telemetry to track: inference latency distribution, error rates, model confidence distributions (for drift detection), and device-specific performance. Use federated learning to improve models from edge data without sending raw data to the cloud.