There was a great WWDC video back in 2018 titled Vision with Core ML. The example app used live video capture to feed an image through the model using scene stability. There is sample code out there for UIKit, but always wanted to try and re-make using SwiftUI as a starting point. Any tips or pointers on making an image recognition app with live video capture using SwiftUI as a starting point ?

You have a couple choices here. You can use the techniques you already know, and use UIViewRepresentable to import the UIViews into your app. That will still work really well!

I've actually done exactly that using UIViewControllerRepresentable in the app I'm working on, and used a modified version of the VisionViewController which I think came from that video's sample code. It works perfectly... the entire app is SwiftUI except for the VisionViewController part.

Here's a link to the original sample code that has the VisionViewController (click the download button from that page): Training a Create ML Model to Classify Flowers. Unfortunately I don't have a publicly available version that shows where I've used that with UIViewControllerRepresentable.

Alternatively, you can display the camera feed by sending the image from each frame into a SwiftUI Image. The new Create ML Components feature actually includes a VideoReader.readCamera method which provides an asynchronous stream of image buffers, which is a great way to get started with this approach. Alternatively you can use your existing AVCaptureDevice logic and delegate methods to provide a series of images. You can see an example of this approach in the rep counting demo app which will be available soon as part of this year's WWDC session What's new in Create ML

Recently I refactored some UIKit code into SwiftUI and found that the ViewController could be relatively easily transformed into an ObservableObject class, changing @IBOulets into @Published properties.

I've been experimenting with using AVCaptureVideoPreviewLayer too. Using a preview layer is likely to be a very CPU / memory efficient way to do it. I don't yet have a clear picture of the efficiency of using SwiftUI views to run the preview directly in the way I proposed, so while it can be a lot of fun to do it purely in SwiftUI it may not be the best choice just yet.

Tagged with: