There was a great WWDC video back in 2018 titled Vision with Core ML. The example app used live video capture to feed an image through the model using scene stability. There is sample code out there for UIKit, but always wanted to try and re-make using SwiftUI as a starting point. Any tips or pointers on making an image recognition app with live video capture using SwiftUI as a starting point ?
You have a couple choices here. You can use the techniques you already know, and use UIViewRepresentable
to import the UIView
s into your app. That will still work really well!
I've actually done exactly that using UIViewControllerRepresentable
in the app I'm working on, and used a modified version of the VisionViewController
which I think came from that video's sample code. It works perfectly... the entire app is SwiftUI except for the VisionViewController
part.
Here's a link to the original sample code that has the VisionViewController
(click the download button from that page): Training a Create ML Model to Classify Flowers. Unfortunately I don't have a publicly available version that shows where I've used that with UIViewControllerRepresentable
.
Alternatively, you can display the camera feed by sending the image from each frame into a SwiftUI Image
. The new Create ML Components feature actually includes a VideoReader.readCamera
method which provides an asynchronous stream of image buffers, which is a great way to get started with this approach. Alternatively you can use your existing AVCaptureDevice
logic and delegate methods to provide a series of images. You can see an example of this approach in the rep counting demo app which will be available soon as part of this year's WWDC session What's new in Create ML
Recently I refactored some UIKit code into SwiftUI and found that the ViewController
could be relatively easily transformed into an ObservableObject
class, changing @IBOulet
s into @Published
properties.
I've been experimenting with using AVCaptureVideoPreviewLayer
too. Using a preview layer is likely to be a very CPU / memory efficient way to do it. I don't yet have a clear picture of the efficiency of using SwiftUI views to run the preview directly in the way I proposed, so while it can be a lot of fun to do it purely in SwiftUI it may not be the best choice just yet.