On-Device Inference with Core ML and Vision

Plenty of apps still ship a photo to a backend and wait for a classification response. That buys a network dependency and added latency, and it sends a copy of the user's image off the device for work the phone's own silicon can do.

#The old way

import Foundation

func classifyRemote(imageData: Data) async throws -> String {
    var request = URLRequest(url: URL(string: "https://api.example.com/classify")!)
    request.httpMethod = "POST"
    request.httpBody = imageData

    let (data, _) = try await URLSession.shared.data(for: request)
    let result = try JSONDecoder().decode(ClassificationResponse.self, from: data)
    return result.label
}

struct ClassificationResponse: Decodable {
    let label: String
}

#The new way

import Vision
import CoreML
import UIKit

func classify(image: UIImage) throws -> String {
    guard let cgImage = image.cgImage else {
        throw ClassificationError.invalidImage
    }

    let model = try VNCoreMLModel(for: MobileNetV2(configuration: MLModelConfiguration()).model)
    let request = VNCoreMLRequest(model: model)

    let handler = VNImageRequestHandler(cgImage: cgImage)
    try handler.perform([request])

    guard let results = request.results as? [VNClassificationObservation],
          let top = results.first else {
        throw ClassificationError.noResult
    }

    return top.identifier
}

enum ClassificationError: Error {
    case invalidImage
    case noResult
}

#Why it matters

• Inference runs on-device, so there is no network dependency and no round-trip latency, and the image data never leaves the phone
• Vision handles image preprocessing (cropping, scaling, orientation) that the model expects, so pixel buffer conversion isn't hand-rolled
• Core ML automatically routes execution across the CPU, GPU, and Neural Engine for the best available performance on that device
• Works fully offline, which matters for any camera-driven feature that needs to respond instantly

#Gotcha

VNCoreMLRequest runs synchronously on whatever thread calls handler.perform(_:). Call it off the main thread, on a background queue or in a detached Task, or the UI stalls on every frame for anything camera-driven. A bundled .mlmodel file also adds real weight to the app's download size, and swapping in a bigger, more accurate model can meaningfully increase both binary size and per-inference latency on older devices. Profile on the oldest device supported rather than a current-generation phone. Exact class and initializer names depend on the model compiled in, so verify against the generated interface for the actual .mlmodel in use.