WWDC22 - Augmented Reality Digital Lounge

For creating 3D models from images captured on device, this should be a helpful resource to get more inspiration and help:

Bring your world into augmented reality

Capturing images with LiDAR devices will give you automatic scale estimation and gravity vector information on your final usdz output

There’s no API in RealityKit to capture video. That said there are system level APIs to capture screen recordings and I wonder if that would be useful for you:

ScreenCaptureKit

I’d suggest filing a feature request with your use-case. Thanks!

Bug Reporting

recommendedVideoFormatForHighResolutionFrameCapturing is used for capturing high resolution still images while the session is running.

For 4K video, you should use recommendedVideoFormatFor4KResolution

Note that this feature is only supported on iPad with M1

Yes, using 4k resolution may result in more power being consumed. It may also result in thermal mitigation engaging to keep the device from getting too hot, which may impact performance. At the moment, we are only supporting 4k @ 30hz.

To capture and replay an ARKit session, see an example here:

Recording and Replaying AR Session Data

If you want to capture video in your app in order to do post processing later, you could use and configure an AVAssetWriter to capture a video.

We also provide a camera frame with every ARFrame, see:

ARFrame.capturedImage

ARFrame.capturedImage is just the ‘clean slate’, it doesn’t contain any virtual content rendered on top of it. If you are doing your own rendering and your Metal textures are backed by IOSurfaces. then you can easily create CVPixelBuffers using the IOSurfaces and then pass those to AVFoundation for recording.

We don’t have QR code detection in ARKit. However, you can use the Vision APIs to do QR code detection on the captured image. This VisionKit talk and article might be of interest to you:

• Capture machine-readable codes and text with VisionKit
• Scanning data with the camera

You can take the next photo right after the completion handler of your previous captureHighResolutionFrame call - or even from within the completion handler.

If you try taking a new photo before the previous call completed, you will receive an ARError.Code.highResolutionFrameCaptureInProgress error in the completion handler/

Please file a bug report for this. Any additional information you can provide, such as a video reproducing the issue would be hugely helpful.

https://developer.apple.com/bug-reporting/

Great question - and we have good news for you 🙂

We are releasing new lighting to AR Quick Look which is brighter with enhanced contrast and improved shape definition to make your assets look even better.

Please check tomorrow's session - Explore USD tools and rendering - with examples and how to implement it!

That is a standard message and unfortunately there’s currently no way to customize the text.

Alternatively, you can create your own banner within your asset.

For example, you can check out the first asset on the gallery page

Please file a feedback report, if you haven't already 🙏

This is not supported at this point. The team is still investigating this possibility.

Yes this is something that is supported in AR Quick Look. You can place objects on the ceiling by dragging them there. This can be done using a regular horizontal (or vertical) anchor.

However, there are potential challenges to be aware, the biggest is that ceilings usually lack a lot of feature points, which makes it difficult to detect a proper plane. Using a device with LiDAR can improve the results that you get.

You could use Reality Composer's interactions to make an interactive USD where you can tap on different colors to change the model

This would need to be done in 3D. There’s a previous session, Building AR Experiences with Reality Composer, that has some examples.

I don't think that it's possible. But you should submit the idea for supporting virtual lights on:

https://feedbackassistant.apple.com

Multiuser AR experiences can be created using the SynchronizationComponent

This is a good tutorial (along with sample code) on building collaborative AR sessions between devices:

https://developer.apple.com/documentation/arkit/creatingacollaborative_session

No there is no re-localization in RoomPlan. But we expose the ARSession so you could fallback to ARKit for re-localization.

We don’t currently have a recommend method for doing this and as such would love to see a feedback item explaining what you need and a use case explaining it. That would be wonderful.

That said your method should in theory work and we’d also love to see feedback item describing the issues you’re seeing.

Bug Reporting

We don’t discuss future plans, but gathering developer feedback is important to us so we’d ask you to post your request to

Bug Reporting

While we don't discuss future plans, we always hope to gather this sort of feedback during WWDC. Thanks for taking the time to share 🙏

We do support collaborative sessions over the same network, more details and sample code can be found here:

Creating a Collaborative Session

Is this what you were looking for?

Check out this page:

Augmented Reality

To learn more about RealityKit and ARKit, I would recommend starting with our documentation and videos. Here are a few links to help you get started:

• RealityKit
• More to explore with ARKit 6
• AR Creation Tools
• Videos > Augmented Reality

You can also always ask questions on Developer Forums 💬

Suggestion from non-Apple developer:

Reality School on YouTube

Yes, if you can export your assets to FBX, glTF or OBJ, you can convert them to USDZ using Reality Converter, which is compatible with ARKit and RealityKit

Hey there, we don’t discuss future releases of Apple products. But we’d love to hear your feedback and suggestions. Please file your feedback here to get it into our system.

Bug Reporting

That’s a good question.

I think you can get some of the way there via the ARKit APIs to get the current frame.

You can also toggle the mode of an ARView to switch it to .nonAR view - then use ARView.snapshot() to grab a snapshot of the virtual content. And then switch it back.

However, I don’t believe that would give you exactly what you want - I think the ARView snapshot would not necessarily have a transparent background (if that’s what you need). And even then the performance of this may not be great.

You could also try setting the Environment background color to something with 100% alpha.

I’d suggest filing a feature request for this with Bug Reporting

Using data scanner via VisionKit is possible using ARKit. ARKit provides the captured image on the ARFrame. One can inject the ARFrame's captured image into data scanner and obtain information about text.

However, the result will be two-dimensional. If the use-case is to bring the detected text into the AR world in three dimensions one needs to estimate a transform for the 2D text. ARKit does not support this natively but does support custom anchoring.

ARMeshAnchor transforms are already aligned with the wide camera, which is also what the camera transform is relative to.

Yes, once you turn on scene reconstruction by setting the sceneReconstruction property on ARWorldTrackingConfiguration. The meshes are available as ARMeshAnchor through the ARKit's anchor delegate methods

Reference APIs:

• ARSessionDelegate
• ARSceneReconstruction
• sceneReconstruction

recommendedVideoFormatForHighResolutionFrameCapturing is used for capturing high resolution still images while the session is running.

For 4K video, you should use recommendedVideoFormatFor4KResolution

Note that this feature is only supported on iPad with M1

We recommend adding an ARAnchor in the position where you want to place and object and then associate your node/entity with that anchor. This should help prevent drifting.

You can use ARKit inside a SwiftUI app. You can also use RealityKit to build ARKit apps in a declarative way.

Here are the links to resources and sample code to help you get started:

• ARKit
• RealityKit

ARKit offers functionality to add custom anchors which is the preferred and recommended way to place content.

See the add(anchor:) method.

Custom anchors are used internally for drift correction. We cannot guarantee absolutely no drift. However, using your own anchors will use the system's best knowledge to correct for any drift.

We really appreciate the feedback and are glad that you are already starting to put these API changes to good use! At the moment, we do not expose the ability to pass in AVCapturePhotoSettings through our API, but this would be a great feature request to submit via Bug Reporting

The depth map surfaced through the Scene Depth API does align with the wide angle camera and should correspond to the camera transform available through the ARFrame.

Here is a sample code that generates a colored point cloud by combining the wide angle camera image and depth map:

Displaying a Point Cloud Using Scene Depth

If you still see some issues, I recommend filing a bug through the feedback assistant at Bug Reporting

ARKit itself has no notion of rendered content. Content (USDZ) is commonly handled by the rendering engine on top of ARKit like RealityKit, SceneKit, Metal, etc.

In order to learn more about USDZ and how to efficiently use it we recommend this talk.

Understand USD fundamentals

"SubjectArea": ( 2013, 1511, 2116, 1270 )
"LensSpecification": ( 4.2, 4.2, 1.6, 1.6 )

The subject area is defined as rectangle with center coordinate and its dimensions. In this case, the center is at (2013, 1511) with rectangle dimensions 2116 x 1270.

For more details, you may refer to Exif standard tags.

No, any modifications to ARConfiguration object does not affect a running session. You need to call run(_:options:) after modifying configuration for it to be used.

You can change capture device settings such as exposure, white balance etc, that will be reflected in the output of ARSession. However, you cannot change the input/output configurations on capture session.

We do not have any resources for this.

You should be able to use the wide angle camera and camera transform to generate texture maps for the meshes but unfortunately we do not have any resources or sample code showing that.

We do have this sample code showing how to generate colored point clouds using the scene depth API, hope it is of some help.

Displaying a Point Cloud Using Scene Depth

ARKit does not have any hand tracking feature. The Vision framework offers functionality for hand gesture detection.

Detect Body and Hand Pose with Vision - WWDC20

You may find the camera's captured images on the ARFrame and can inject this into Vision. So by combining multiple frameworks you could achieve something close to the requested feature.

ARKit returns discrete tracking state for every frame update. You can read more about it here:

Managing Session Life Cycle and Tracking Quality

It is highly recommended to integrate standard coaching view in your app to guide users when tracking is limited. More details at:

ARCoachingOverlayView

3D body tracking is only supported using the ARBodyTrackingConfiguration. However, we support the detection of 2D bodies on multiple configurations; the ARWorldTrackingConfiguration is one of them.

In order to check which configuration supports this you may use the supportsFrameSemantics(_:) function.

ARKit only supports wide camera as the primary camera. It is not possible use other cameras for rendering.

ARFaceTrackingConfiguration, however, uses the front facing camera.

If you have a need to use a different camera, please file a feature request through the feedback assistant at Bug Reporting

There is no integration of MapKit into the ARView. If you know the building footprint (i.e. the polygon in lat/lon coordinates) or even exact geometry anchored to a lat/lon coordinate you can transform these coordinates by placing ARGeoAnchors at a that location. If they are tracked in ARKit you get the local coordinates and can build an occlusion/collision mesh.

RoomPlan is not an ARAnchor in current design. Thanks for suggestion. We will take into consideration.

From a non-Apple developer:

I created a demo where custom ARAnchors are created for RoomPlan objects. The same could be done for surfaces and then saved to a world map:

https://github.com/jmousseau/RoomObjectReplicatorDemo

Yes, using 4k resolution may result in more power being consumed. It may also result in thermal mitigation engaging to keep the device from getting too hot, which may impact performance. At the moment, we are only supporting 4k @ 30hz.

We are not exposing the heading directly.

You can create any ARGeoAnchor in your vicinity and then compare its transform with your camera’s transform. Since ARGeoAnchors are always aligned to East-Up-South, you can derive any global camera orientation by comparing the camera’s transform to the ARGeoAnchor transform.

Thanks for the suggestion. Here are some existing sample code that uses Metal with ARKit:

• Displaying a Point Cloud Using Scene Depth
• Creating a Fog Effect Using Scene Depth
• Displaying an AR Experience with Metal

No, ARKit runs standalone on iPhone and iPad devices only and does not take any external inputs.

RealityKit exposes the ARSession through this API:

https://developer.apple.com/documentation/realitykit/arview/session

You can set the delegate on it to listen to ARKit delegate events.

To capture and replay an ARKit session, see an example here:

Recording and Replaying AR Session Data

If you want to capture video in your app in order to do post processing later, you could use and configure an AVAssetWriter to capture a video.

We also provide a camera frame with every ARFrame, see:

ARFrame.capturedImage

ARFrame.capturedImage is just the ‘clean slate’, it doesn’t contain any virtual content rendered on top of it. If you are doing your own rendering and your Metal textures are backed by IOSurfaces. then you can easily create CVPixelBuffers using the IOSurfaces and then pass those to AVFoundation for recording.

We recommend adding ARAnchors and associating your virtual content with ARAnchors. In case there is a drift, the session delegate session(_didUpdate:) would update the anchor such that the virtual content stays in the same location in the real world.

We do not have a sample code that uses ARKit together with spatial audio (PHASE). However, this is a great question, can you please send us a request through Bug Reporting

Body tracking runs at 60 Hz at the same cadence of the camera. We cannot compute this faster than this. However, by changing the ARVideoFormat you may change this to 30 Hz or other supported frame rates.

We do not offer a functionality to write the motion capture data to file. However, our data is very well compatible with a format called BVH. By following the topology of the skeleton given by the ARSkeletonDefinition and the data coming from an ARBodyAnchor one could generate such a file output.

No, there haven’t been changes to light estimation in ARKit this year.

Thanks for your excitement about that app idea. We do not have the sample code, but I recommend going to our Explore ARKit 4 session where we explain how to pick a coordinate in Maps and create an ARGeoAnchor based on it. For alignment with the real world we have the example with the Ferry Building in SF. We followed that exact workflow with the focus square example.

In general we recommend to use/add anchors independent from the device you are using.

We don’t have QR code detection in ARKit. However, you can use the Vision APIs to do QR code detection on the captured image. This VisionKit talk and article might be of interest to you:

• Capture machine-readable codes and text with VisionKit
• Scanning data with the camera

You can take the next photo right after the completion handler of your previous captureHighResolutionFrame call - or even from within the completion handler.

If you try taking a new photo before the previous call completed, you will receive an ARError.Code.highResolutionFrameCaptureInProgress error in the completion handler/

Creating a model for Object Detection and creating a textured mesh with Object Capture are two different use cases with separate workflows, we do not offer a tool to convert from one to another. That sounds like a great use case though, I encourage you to file a feature request.

Feedback Assistant

ARKit detects one body at a time. If multiple people are in the scene, the most prominent one is returned.

Unfortunately not. ARKit consumes the UW camera internally for certain processing tasks in a specific configuration.

Though I encourage you to file a feature request. Feedback Assistant

ARKit offers several ways to measure distances. You can either evaluate distances from the device to its environment or between ARAnchors.

Please see this documentation to get an overview:

Environmental Analysis

I recommend checking out our documentation of our SceneGeometry API that we presented in ARKit 3.5. A good overview is given in this tech talk:

Advanced Scene Understanding in AR

After getting a geometry that is good enough you still have to solve the task of isolating your object of choice and computing its volume. There would be several ways of doing it. For example cutting everything off above the ground level, or letting the user create a cube objects and then intersect it with the scene geometry, we do not have any code sample for these tasks, though.

Setting videoHDRAllowed means that HDR will be enabled on the formats supporting it; however this is not the case for all video formats.

In iOS 16, ARVideoFormat has a new property isVideoHDRSupported. You can filter the list of the configuration’s supportedVideoFormat to find one where videoHDRSupported is true, and set this format as the configuration’s videoFormat before running the session.

ARKit doesn’t support physics but rather detects the surrounding scene to allow RealityKit to handle virtual objects. RealityKit does support rigid body physics and a good place to start looking is at the physics APIs here:

Preliminary_PhysicsRigidBodyAPI

We have a limited pool of resources for our ARFrames and in order to keep some available for ARKit to process, we recommend processing the frames as quickly as possible. If you need to perform longer computations, you can copy an ARFrame and release the ARFrame from the delegate method.

ARKit configures the cameras according to the selected configuration. Capturing from another camera while an ARKit session is running is not supported.

Glad you’re also a fan of the new desk view feature. There are potentially two solutions to this:

1. Do a single perspective projections for the whole image
2. Use a per-pixel correction like you suggested

Both come with their own benefits and drawbacks. Please check out our documentation for implementing the second approach:

Displaying a Point Cloud Using Scene Depth

ARKit has no built-in hand or finger detection, but you can use Vision to track hands or detect hand poses. Here is a developer sample illustrating this:

Detecting Hand Poses with Vision

For ARKit feature requests, we encourage you to send us a report in Feedback Assistant

Throwing machine learning at it sounds like a super fun project, I would recommend to start a bit simpler.

So as a small experiment you can take the four corners of a building from the Maps app and then create four Location Anchors based on these coordinates. As soon as those are tracked you can look at the local coordinates (in x,y,z) and then build a polygon based on it, you can extrude it towards the sky (y up) to get a nice collision/occlusion mesh.

ARKit allows streaming video from only one camera at a time. Which camera is used is determined by your configuration (e.g. ARFaceTrackingConfiguration will use the front facing camera, ARWorldTrackingConfiguration will use the back wide camera).

You can, however, enable face anchors detected by the front camera in an ARWorldTrackingConfiguration with userFaceTrackingEnabled. Vice versa, you can enable isWorldTrackingEnabled in an ARFaceTrackingConfiguration to benefit from 6DOF world tracking.

Check out this developer sample:

Combining User Face-Tracking and World Tracking

We introduced new API to capture frames in higher resolution than your configuration’s video format:

captureHighResolutionFrame(completion:)

Those frames are captured from the same camera, though.

Setting up additional cameras is not supported. We encourage you to file a feature request in Feedback Assistant

ARWorldMaps are optimized for room sized scenarios. If you exceed that limit then relocalization will stop working in certain areas as the map is not big enough any more to cover the whole area.

The frame drops sound related to the amount of content being displayed though. For that, feel free to provide more details through Feedback Assistant

This is not supported out of the box. What you could do is compute the offset between your current location (before relocalization) and after relocalization and apply that accordingly.

3D body tracking is only supported using the ARBodyTrackingConfiguration. However, we support the detection of 2D bodies on multiple configurations; the ARWorldTrackingConfiguration is one of them.

In order to check which configuration supports this you may use the supportsFrameSemantics(_:) function.

Body tracking runs at 60 Hz at the same cadence of the camera. We cannot compute this faster than this. However, by changing the ARVideoFormat you may change this to 30 Hz or other supported frame rates.

We do not offer a functionality to write the motion capture data to file. However, our data is very well compatible with a format called BVH. By following the topology of the skeleton given by the ARSkeletonDefinition and the data coming from an ARBodyAnchor one could generate such a file output.

ARKit detects one body at a time. If multiple people are in the scene, the most prominent one is returned.

Capturing images with LiDAR devices will give you automatic scale estimation and gravity vector information on your final usdz output

ARMeshAnchor transforms are already aligned with the wide camera, which is also what the camera transform is relative to.

The depth map surfaced through the Scene Depth API does align with the wide angle camera and should correspond to the camera transform available through the ARFrame.

Here is a sample code that generates a colored point cloud by combining the wide angle camera image and depth map:

Displaying a Point Cloud Using Scene Depth

If you still see some issues, I recommend filing a bug through the feedback assistant at Bug Reporting

No, any modifications to ARConfiguration object does not affect a running session. You need to call run(_:options:) after modifying configuration for it to be used.

You can change capture device settings such as exposure, white balance etc, that will be reflected in the output of ARSession. However, you cannot change the input/output configurations on capture session.

We do not have any resources for this.

You should be able to use the wide angle camera and camera transform to generate texture maps for the meshes but unfortunately we do not have any resources or sample code showing that.

We do have this sample code showing how to generate colored point clouds using the scene depth API, hope it is of some help.

Displaying a Point Cloud Using Scene Depth

ARKit returns discrete tracking state for every frame update. You can read more about it here:

Managing Session Life Cycle and Tracking Quality

It is highly recommended to integrate standard coaching view in your app to guide users when tracking is limited. More details at:

ARCoachingOverlayView

ARKit only supports wide camera as the primary camera. It is not possible use other cameras for rendering.

ARFaceTrackingConfiguration, however, uses the front facing camera.

If you have a need to use a different camera, please file a feature request through the feedback assistant at Bug Reporting

We are not exposing the heading directly.

You can create any ARGeoAnchor in your vicinity and then compare its transform with your camera’s transform. Since ARGeoAnchors are always aligned to East-Up-South, you can derive any global camera orientation by comparing the camera’s transform to the ARGeoAnchor transform.

To capture and replay an ARKit session, see an example here:

Recording and Replaying AR Session Data

If you want to capture video in your app in order to do post processing later, you could use and configure an AVAssetWriter to capture a video.

We also provide a camera frame with every ARFrame, see:

ARFrame.capturedImage

ARFrame.capturedImage is just the ‘clean slate’, it doesn’t contain any virtual content rendered on top of it. If you are doing your own rendering and your Metal textures are backed by IOSurfaces. then you can easily create CVPixelBuffers using the IOSurfaces and then pass those to AVFoundation for recording.

Body tracking runs at 60 Hz at the same cadence of the camera. We cannot compute this faster than this. However, by changing the ARVideoFormat you may change this to 30 Hz or other supported frame rates.

We do not offer a functionality to write the motion capture data to file. However, our data is very well compatible with a format called BVH. By following the topology of the skeleton given by the ARSkeletonDefinition and the data coming from an ARBodyAnchor one could generate such a file output.

Unfortunately not. ARKit consumes the UW camera internally for certain processing tasks in a specific configuration.

Though I encourage you to file a feature request. Feedback Assistant

ARKit configures the cameras according to the selected configuration. Capturing from another camera while an ARKit session is running is not supported.

ARKit allows streaming video from only one camera at a time. Which camera is used is determined by your configuration (e.g. ARFaceTrackingConfiguration will use the front facing camera, ARWorldTrackingConfiguration will use the back wide camera).

You can, however, enable face anchors detected by the front camera in an ARWorldTrackingConfiguration with userFaceTrackingEnabled. Vice versa, you can enable isWorldTrackingEnabled in an ARFaceTrackingConfiguration to benefit from 6DOF world tracking.

Check out this developer sample:

Combining User Face-Tracking and World Tracking

We introduced new API to capture frames in higher resolution than your configuration’s video format:

captureHighResolutionFrame(completion:)

Those frames are captured from the same camera, though.

Setting up additional cameras is not supported. We encourage you to file a feature request in Feedback Assistant

I’d recommend this year’s WWDC Xcode session, What's new in Xcode, for what’s new in debugging. And there have been many other excellent sessions over the years on debugging.

That said if you believe it may be RealityKit or another system framework responsible for leaking the entities we’d ask you to file a Feedback Item on http://feedbackassistant.apple.com if you haven’t done so already.

It depends which approach is best — each have a set of pros and cons based on what you want out of the experience.

It can be compelling to run through AR blocks if you want to emphasize lots of user motion in an experience and the scale of the experience is quite large — good for apps that can take advantage of wide open spaces.

Tapping them is more immediate and indirect so if you wanted to destroy a tower quickly or something like that then that would be the way to go — and I could see that being very satisfying to trigger many physics objects to react at once.

I think the same would apply to a button press, it’s an indirect way to trigger it if the experience requires rapidly knocking them down.

Overall I think it’s up to what you want the experience to be, and maintaining internal consistency with other interactions within the app.

Swiftstrike and Swiftshot are great example apps that use similar techniques.

We recommend adding an ARAnchor in the position where you want to place and object and then associate your node/entity with that anchor. This should help prevent drifting.

ARKit offers functionality to add custom anchors which is the preferred and recommended way to place content.

See the add(anchor:) method.

Custom anchors are used internally for drift correction. We cannot guarantee absolutely no drift. However, using your own anchors will use the system's best knowledge to correct for any drift.

We recommend adding ARAnchors and associating your virtual content with ARAnchors. In case there is a drift, the session delegate session(_didUpdate:) would update the anchor such that the virtual content stays in the same location in the real world.

In general we recommend to use/add anchors independent from the device you are using.

We are aware of a bug in macOS Ventura/iOS 16 that is causing the lighting to appear darker. Please feel free to file a bug report on Feedback Assistant about this.

Post processing effects are not supported in RC right now, only in a RealityKit app. However, feel free to file a feature request on Feedback Assistant about this.

Please file a bug report for this. Any additional information you can provide, such as a video reproducing the issue would be hugely helpful.

https://developer.apple.com/bug-reporting/

I don’t think we have any guidelines about this, since exporting/saving a scene is not supported by the current APIs. ModelIO seems like a reasonable solution to me, but you might also want to file a feature request for this on Feedback Assistant.

Unfortunately, we don’t discuss our future plans. However, we are aware that our tools haven’t been updated in a few years and could use some new features. Could you share what features you are looking for us to add?

I’d recommend this year’s WWDC Xcode session, What's new in Xcode, for what’s new in debugging. And there have been many other excellent sessions over the years on debugging.

That said if you believe it may be RealityKit or another system framework responsible for leaking the entities we’d ask you to file a Feedback Item on http://feedbackassistant.apple.com if you haven’t done so already.

You might want to ask this to the ARKit team, but I’m not aware of any plans.

A feedback item would be good though!

We don’t currently have a recommend method for doing this and as such would love to see a feedback item explaining what you need and a use case explaining it. That would be wonderful.

That said your method should in theory work and we’d also love to see feedback item describing the issues you’re seeing.

Bug Reporting

We don’t discuss future plans, but gathering developer feedback is important to us so we’d ask you to post your request to

Bug Reporting

While we don’t discuss specific future plans, we always want to make RealityKit as easy to use for everyone as we can.

We’d ask you to post your issues and/or suggestions to Bug Reporting

I’d love to find out more about what you find too cumbersome. Thanks!

While we don't discuss future plans, we always hope to gather this sort of feedback during WWDC. Thanks for taking the time to share 🙏

We do support collaborative sessions over the same network, more details and sample code can be found here:

Creating a Collaborative Session

Is this what you were looking for?

You can host RealityKit content inside SwiftUI views with UIViewRepresentable

If you’re asking if you can use SwiftUI content within RealityKit - there is no direct support for that at present and we’d ask you to file a feedback item explaining your use-case for that feature.

Bug Reporting

You may want to check with the SF Symbols team to confirm this is not possible yet, and also file a feature request on feedback assistant.

A bit of a hack solution but you may be able to get this work via drawing the Symbol to a CGImage and passing that image in as a texture.

I would suggest filing an enhancement request on feedback assistant for this

Hey there, we don’t discuss future releases of Apple products. But we’d love to hear your feedback and suggestions. Please file your feedback here to get it into our system.

Bug Reporting

No.

We don't discuss details about unreleased updates, but one of the things that’s most helpful to us as we continue to build out our suite of augmented reality developer tools is feedback

Please continue to submit ideas or suggestions in Feedback Assistant 🙂

I don't think that it's possible. But you should submit the idea for supporting virtual lights on:

https://feedbackassistant.apple.com

Hey there, we’d definitely be interested in hearing more about your idea.

I’d suggest submitting the suggestion at Bug Reporting

If you have your USDZ content on the web you can check out the AR Quick Look functionality for such things at:

Adding an Apple Pay Button or a Custom Action in AR Quick Look

As far as I know there isn’t currently a way to do such a thing directly from a USDZ sent from iMessage, but I can pass that request along.

While Reality Composer is available only for iOS and iPadOS on the App Store, we'll pass this feedback along. Thanks 🙏

Reality Composer is available on macOS as part of Xcode as a Developer Tool, though.

There’s no support for that in Reality Composer currently. As always a feature request filed on Bug Reporting would be most appreciated.

There’s also no method to export USDZ from RealityKit and again feature requests appreciated. Thank you!

Currently Reality Composer does not support this. This sounds like a great feature request and we would appreciate if you can file feedback through Feedback Assistant.

If you are willing to jump into code…

You can use the AudioPlaybackController returned from the playAudio API to play, pause, etc. You can also use AudioFileResource to to add / replace audio on entities.

Instancing is mostly abstracted away behind the Entity.clone() method.

Level of detail is not currently exposed as API and we’d recommend filing a feature suggestion on Bug Reporting

That said you can implement Level of Detail yourself (probably using custom Systems and Components) although we understand that may not be ideal. Please file feature suggestions regardless!

While we do not currently support custom render passes, we have support for post process effects. Please file a feature request through Feedback Assistant if your use case requires more customization 🙏

This may be achievable if embedded in Xcode with some code.

I recommend signing up for a Reality Composer lab if you would like to explore that further.

But yes, being able to observe live parameters sounds like a great feature in Reality Composer. Please file a feature request using Feedback Assistant with your use case 🙂

We really appreciate the feedback and are glad that you are already starting to put these API changes to good use! At the moment, we do not expose the ability to pass in AVCapturePhotoSettings through our API, but this would be a great feature request to submit via Bug Reporting

The depth map surfaced through the Scene Depth API does align with the wide angle camera and should correspond to the camera transform available through the ARFrame.

Here is a sample code that generates a colored point cloud by combining the wide angle camera image and depth map:

Displaying a Point Cloud Using Scene Depth

If you still see some issues, I recommend filing a bug through the feedback assistant at Bug Reporting

ARKit only supports wide camera as the primary camera. It is not possible use other cameras for rendering.

ARFaceTrackingConfiguration, however, uses the front facing camera.

If you have a need to use a different camera, please file a feature request through the feedback assistant at Bug Reporting

We do not have a sample code that uses ARKit together with spatial audio (PHASE). However, this is a great question, can you please send us a request through Bug Reporting

Creating a model for Object Detection and creating a textured mesh with Object Capture are two different use cases with separate workflows, we do not offer a tool to convert from one to another. That sounds like a great use case though, I encourage you to file a feature request.

Feedback Assistant

Unfortunately not. ARKit consumes the UW camera internally for certain processing tasks in a specific configuration.

Though I encourage you to file a feature request. Feedback Assistant

ARKit has no built-in hand or finger detection, but you can use Vision to track hands or detect hand poses. Here is a developer sample illustrating this:

Detecting Hand Poses with Vision

For ARKit feature requests, we encourage you to send us a report in Feedback Assistant

We introduced new API to capture frames in higher resolution than your configuration’s video format:

captureHighResolutionFrame(completion:)

Those frames are captured from the same camera, though.

Setting up additional cameras is not supported. We encourage you to file a feature request in Feedback Assistant

You can use the installGestures function on ARView. Keep in mind that the entity will need to conform to HasCollision.

To do this you could create your own CollisionComponent with a custom mesh and add it to your entity or you could simply call generateCollisionShapes(recursive: Bool) on your entity. Putting it all together, you can use .loadModel/.loadModelAsync, which will flatten the USDZ into a single entity. Then call generateCollisionShapes and pass that entity to the installGestures function. This will make your USDZ one single entity that you can interact with.

A way to get the entity gestures working on an entire scene is to use visualBounds(…) and create a CollisionComponent on the root entity. You can then use CollisionGroup to make sure it doesn’t interfere with any physics.

If you’re using ARView.installGestures(…) you’ll need the entity to conform to HasCollision, which may require you to create a new entity type for the root. Quick example:

// New Entity type which conforms for `HasCollision`
class CollisionAnchorEntity: Entity, HasAnchoring, HasCollision { }

// Transfer scene contents
let collisionAnchor = CollisionAnchorEntity()
collisionAnchor
    .children.append(contentsOf: originalAnchor.children)
collisionAnchor.anchoring = originalAnchor.anchoring

// Create CollisionComponent for bounds of scene
let sceneBounds = collisionAnchor
    .visualBounds(recursive: true, relativeTo: collisionAnchor)
let collisionShape = ShapeResource
    .generateBox(size: sceneBounds.extents)
    .offsetBy(translation: sceneBounds.center)
collisionAnchor.collision = CollisionComponent(
                                shapes: [collisionShape]
                            )

// Install gesture on new anchor
arView.installGestures(for: collisionAnchor)

Check out this page:

Augmented Reality

To learn more about RealityKit and ARKit, I would recommend starting with our documentation and videos. Here are a few links to help you get started:

• RealityKit
• More to explore with ARKit 6
• AR Creation Tools
• Videos > Augmented Reality

You can also always ask questions on Developer Forums 💬

Suggestion from non-Apple developer:

Reality School on YouTube

Hi! We have a number of WWDC sessions covering RealityKit and Reality Composer which is a great place to start.

• Building AR Experiences with Reality Composer
  • Sample Code: Creating a Game with Reality Composer

• Building Apps with RealityKit
• What’s new in RealityKit
• Dive into RealityKit 2
  • Sample Code: Building an Immersive Experience with RealityKit
  • Sample Code: Creating an App for Face-Painting in AR

• Explore advanced rendering with RealityKit 2

There’s also a great guide on building a SwiftStrike game: SwiftStrike: Creating a Game with RealityKit

Regardless of your educational background, anyone can learn how to code if you put in the effort and are passionate about it. There are tons of resources online, many of which have been produced by Apple in the form of documentation, example projects, and WWDC videos, that can help you to learn a programming language, such as Swift.

I would suggest doing some tutorials, watching videos, maybe find a highly rated book on iOS programming, etc to learn how to begin building iOS apps.

Once you are comfortable with that, then you can start to dive into AR specifically. Finding a good book on linear algebra would be useful if you are going to get into AR and graphics programming, but start with the basics first!

For ARKit, we have all sorts of documentation and examples that you can take a look at:

https://developer.apple.com/documentation/arkit/

From a non-Apple developer:

Apple’s documentation is great.

I also found the site RayWenderlich to be super helpful. They even have a book specifically for AR:

Apple Augmented Reality by Tutorials

ARKit does not have any hand tracking feature. The Vision framework offers functionality for hand gesture detection.

Detect Body and Hand Pose with Vision - WWDC20

You may find the camera's captured images on the ARFrame and can inject this into Vision. So by combining multiple frameworks you could achieve something close to the requested feature.

ARKit has no built-in hand or finger detection, but you can use Vision to track hands or detect hand poses. Here is a developer sample illustrating this:

Detecting Hand Poses with Vision

For ARKit feature requests, we encourage you to send us a report in Feedback Assistant

For creating 3D models from images captured on device, this should be a helpful resource to get more inspiration and help:

Bring your world into augmented reality

Capturing images with LiDAR devices will give you automatic scale estimation and gravity vector information on your final usdz output

You might want to ask this to the ARKit team, but I’m not aware of any plans.

A feedback item would be good though!

This should be fixed in iOS 16.

Yes this is something that is supported in AR Quick Look. You can place objects on the ceiling by dragging them there. This can be done using a regular horizontal (or vertical) anchor.

However, there are potential challenges to be aware, the biggest is that ceilings usually lack a lot of feature points, which makes it difficult to detect a proper plane. Using a device with LiDAR can improve the results that you get.

ARMeshAnchor transforms are already aligned with the wide camera, which is also what the camera transform is relative to.

The depth map surfaced through the Scene Depth API does align with the wide angle camera and should correspond to the camera transform available through the ARFrame.

Here is a sample code that generates a colored point cloud by combining the wide angle camera image and depth map:

Displaying a Point Cloud Using Scene Depth

If you still see some issues, I recommend filing a bug through the feedback assistant at Bug Reporting

There is no integration of MapKit into the ARView. If you know the building footprint (i.e. the polygon in lat/lon coordinates) or even exact geometry anchored to a lat/lon coordinate you can transform these coordinates by placing ARGeoAnchors at a that location. If they are tracked in ARKit you get the local coordinates and can build an occlusion/collision mesh.

In general we recommend to use/add anchors independent from the device you are using.

I don't think that it's possible. But you should submit the idea for supporting virtual lights on:

https://feedbackassistant.apple.com

There are various sorts of lighting in RealityKit - you might want to start here perhaps?

PointLight

(see the Cameras and Lighting section in the docs)

But looks like we don’t support lighting in Reality Composer unfortunately so I’d suggest filing a feature suggestion:

Bug Reporting

No, there haven’t been changes to light estimation in ARKit this year.

There is no integration of MapKit into the ARView. If you know the building footprint (i.e. the polygon in lat/lon coordinates) or even exact geometry anchored to a lat/lon coordinate you can transform these coordinates by placing ARGeoAnchors at a that location. If they are tracked in ARKit you get the local coordinates and can build an occlusion/collision mesh.

Throwing machine learning at it sounds like a super fun project, I would recommend to start a bit simpler.

So as a small experiment you can take the four corners of a building from the Maps app and then create four Location Anchors based on these coordinates. As soon as those are tracked you can look at the local coordinates (in x,y,z) and then build a polygon based on it, you can extrude it towards the sky (y up) to get a nice collision/occlusion mesh.

You can apply CustomMaterials & CustomMaterial.SurfaceShader to achieve certain cool effects for entities!

From the metal side you can call discard_fragment()

We don’t have any materials you can apply to objects to make them participate in the same shadows as ground planes. However, you can enable shadow casting from directional and spot lights via DirectionalLightComponent.Shadows and SpotLightComponent.Shadows. This may alter the overall lighting of your scene though.

Alternatively, we do have CustomMaterial, which allows you to create custom materials via Metal, but for this use-case may not be able to get you the desired effect.

We’re always looking to improve RealityKit, so would appreciate if you submitted a request for this via https://feedbackassistant.apple.com/

RealityKit has a CustomMaterial API which allows you to create custom Metal-based materials. I’d recommend our Explore advanced rendering with RealityKit 2 WWDC talk to learn more.

There is also a great resource on Custom Shader API that gives more details on the APIs available in Metal.

ARKit offers several ways to measure distances. You can either evaluate distances from the device to its environment or between ARAnchors.

Please see this documentation to get an overview:

Environmental Analysis

I recommend checking out our documentation of our SceneGeometry API that we presented in ARKit 3.5. A good overview is given in this tech talk:

Advanced Scene Understanding in AR

After getting a geometry that is good enough you still have to solve the task of isolating your object of choice and computing its volume. There would be several ways of doing it. For example cutting everything off above the ground level, or letting the user create a cube objects and then intersect it with the scene geometry, we do not have any code sample for these tasks, though.

Are you referring to https://developer.apple.com/documentation/metal/resource_loading?

We don’t have any materials you can apply to objects to make them participate in the same shadows as ground planes. However, you can enable shadow casting from directional and spot lights via DirectionalLightComponent.Shadows and SpotLightComponent.Shadows. This may alter the overall lighting of your scene though.

Alternatively, we do have CustomMaterial, which allows you to create custom materials via Metal, but for this use-case may not be able to get you the desired effect.

We’re always looking to improve RealityKit, so would appreciate if you submitted a request for this via https://feedbackassistant.apple.com/

RealityKit has a CustomMaterial API which allows you to create custom Metal-based materials. I’d recommend our Explore advanced rendering with RealityKit 2 WWDC talk to learn more.

There is also a great resource on Custom Shader API that gives more details on the APIs available in Metal.

“Maybe” 🙂

So you can certainly take MTLTextures and convert them (if they’re configured correctly) into CVPixelBuffers for AVFoundation to consume.

That said it’s really not the intended use case of RealityKit's post processing functionality and I wouldn’t be surprised if either it doesn’t work as you’d expect or if we break you in the future.

Sounds like a great feature request though - Bug Reporting

Thanks for the suggestion. Here are some existing sample code that uses Metal with ARKit:

• Displaying a Point Cloud Using Scene Depth
• Creating a Fog Effect Using Scene Depth
• Displaying an AR Experience with Metal

Throwing machine learning at it sounds like a super fun project, I would recommend to start a bit simpler.

So as a small experiment you can take the four corners of a building from the Maps app and then create four Location Anchors based on these coordinates. As soon as those are tracked you can look at the local coordinates (in x,y,z) and then build a polygon based on it, you can extrude it towards the sky (y up) to get a nice collision/occlusion mesh.

I don’t think we have any guidelines about this, since exporting/saving a scene is not supported by the current APIs. ModelIO seems like a reasonable solution to me, but you might also want to file a feature request for this on Feedback Assistant.

Multiuser AR experiences can be created using the SynchronizationComponent

This is a good tutorial (along with sample code) on building collaborative AR sessions between devices:

https://developer.apple.com/documentation/arkit/creatingacollaborative_session

For creating 3D models from images captured on device, this should be a helpful resource to get more inspiration and help:

Bring your world into augmented reality

Capturing images with LiDAR devices will give you automatic scale estimation and gravity vector information on your final usdz output

Creating a model for Object Detection and creating a textured mesh with Object Capture are two different use cases with separate workflows, we do not offer a tool to convert from one to another. That sounds like a great use case though, I encourage you to file a feature request.

Feedback Assistant

Instancing is mostly abstracted away behind the Entity.clone() method.

Level of detail is not currently exposed as API and we’d recommend filing a feature suggestion on Bug Reporting

That said you can implement Level of Detail yourself (probably using custom Systems and Components) although we understand that may not be ideal. Please file feature suggestions regardless!

We have a limited pool of resources for our ARFrames and in order to keep some available for ARKit to process, we recommend processing the frames as quickly as possible. If you need to perform longer computations, you can copy an ARFrame and release the ARFrame from the delegate method.

ARWorldMaps are optimized for room sized scenarios. If you exceed that limit then relocalization will stop working in certain areas as the map is not big enough any more to cover the whole area.

The frame drops sound related to the amount of content being displayed though. For that, feel free to provide more details through Feedback Assistant

Yes, using 4k resolution may result in more power being consumed. It may also result in thermal mitigation engaging to keep the device from getting too hot, which may impact performance. At the moment, we are only supporting 4k @ 30hz.

You can take the next photo right after the completion handler of your previous captureHighResolutionFrame call - or even from within the completion handler.

If you try taking a new photo before the previous call completed, you will receive an ARError.Code.highResolutionFrameCaptureInProgress error in the completion handler/

We have a limited pool of resources for our ARFrames and in order to keep some available for ARKit to process, we recommend processing the frames as quickly as possible. If you need to perform longer computations, you can copy an ARFrame and release the ARFrame from the delegate method.

ARWorldMaps are optimized for room sized scenarios. If you exceed that limit then relocalization will stop working in certain areas as the map is not big enough any more to cover the whole area.

The frame drops sound related to the amount of content being displayed though. For that, feel free to provide more details through Feedback Assistant

Reality Converter makes it easy to convert, view, and customize USDZ 3D objects on Mac. For more information, visit:

https://developer.apple.com/augmented-reality/tools/

We are aware of a bug in macOS Ventura/iOS 16 that is causing the lighting to appear darker. Please feel free to file a bug report on Feedback Assistant about this.

Yes, on macOS, you can open [from the menu bar] Reality Composer ▸ Settings [or Prefences on older versions of macOS] and check Enable USDZ export

Post processing effects are not supported in RC right now, only in a RealityKit app. However, feel free to file a feature request on Feedback Assistant about this.

Please file a bug report for this. Any additional information you can provide, such as a video reproducing the issue would be hugely helpful.

https://developer.apple.com/bug-reporting/

Unfortunately, we don’t discuss our future plans. However, we are aware that our tools haven’t been updated in a few years and could use some new features. Could you share what features you are looking for us to add?

Video textures are currently not supported through Reality Composer UI. However, if your .rcproj is part of an Xcode project, you can use the RealityKit VideoMaterial api to change the material of your object in the scene at runtime.

You could use Reality Composer's interactions to make an interactive USD where you can tap on different colors to change the model

This would need to be done in 3D. There’s a previous session, Building AR Experiences with Reality Composer, that has some examples.

There are various third-party DCCs with great USD support that let you create complex 3D object with textures and export as USD. You can then use Reality Converter to convert those to USDZ to import into Motion.

Another approach: three.js (web render engine) can actually create USDZs on the fly from 3D scenes. A colleague used that recently for USDZ AR files with changeable textures on https://webweb.jetzt/ar-gallery/ar-gallery.html

Also take a look at the Explore USD tools and rendering session tomorrow. You can now change materials properties in RealityConverter! (edited)

Another thing that might help for making quick adjustments: the browser-based three.js editor at https://threejs.org/editor.

You can assemble different USDZ assets together to build out a larger scene in Reality Composer and add triggers and actions to individual assets within the project

No.

We don't discuss details about unreleased updates, but one of the things that’s most helpful to us as we continue to build out our suite of augmented reality developer tools is feedback

Please continue to submit ideas or suggestions in Feedback Assistant 🙂

Hey there, we’d definitely be interested in hearing more about your idea.

I’d suggest submitting the suggestion at Bug Reporting

We don’t have any materials you can apply to objects to make them participate in the same shadows as ground planes. However, you can enable shadow casting from directional and spot lights via DirectionalLightComponent.Shadows and SpotLightComponent.Shadows. This may alter the overall lighting of your scene though.

Alternatively, we do have CustomMaterial, which allows you to create custom materials via Metal, but for this use-case may not be able to get you the desired effect.

We’re always looking to improve RealityKit, so would appreciate if you submitted a request for this via https://feedbackassistant.apple.com/

While Reality Composer is available only for iOS and iPadOS on the App Store, we'll pass this feedback along. Thanks 🙏

Reality Composer is available on macOS as part of Xcode as a Developer Tool, though.

Hi! We have a number of WWDC sessions covering RealityKit and Reality Composer which is a great place to start.

• Building AR Experiences with Reality Composer
  • Sample Code: Creating a Game with Reality Composer

• Building Apps with RealityKit
• What’s new in RealityKit
• Dive into RealityKit 2
  • Sample Code: Building an Immersive Experience with RealityKit
  • Sample Code: Creating an App for Face-Painting in AR

• Explore advanced rendering with RealityKit 2

There’s also a great guide on building a SwiftStrike game: SwiftStrike: Creating a Game with RealityKit

There’s no support for that in Reality Composer currently. As always a feature request filed on Bug Reporting would be most appreciated.

There’s also no method to export USDZ from RealityKit and again feature requests appreciated. Thank you!

You can use Reality Composer to create a scene attached to an image anchor. You can then export the scene to a USDZ or a Reality File.

See Selecting an Anchor for a Reality Composer Scene

Currently Reality Composer does not support this. This sounds like a great feature request and we would appreciate if you can file feedback through Feedback Assistant.

If you are willing to jump into code…

You can use the AudioPlaybackController returned from the playAudio API to play, pause, etc. You can also use AudioFileResource to to add / replace audio on entities.

There are various sorts of lighting in RealityKit - you might want to start here perhaps?

PointLight

(see the Cameras and Lighting section in the docs)

But looks like we don’t support lighting in Reality Composer unfortunately so I’d suggest filing a feature suggestion:

Bug Reporting

This may be achievable if embedded in Xcode with some code.

I recommend signing up for a Reality Composer lab if you would like to explore that further.

But yes, being able to observe live parameters sounds like a great feature in Reality Composer. Please file a feature request using Feedback Assistant with your use case 🙂

A way to get the entity gestures working on an entire scene is to use visualBounds(…) and create a CollisionComponent on the root entity. You can then use CollisionGroup to make sure it doesn’t interfere with any physics.

If you’re using ARView.installGestures(…) you’ll need the entity to conform to HasCollision, which may require you to create a new entity type for the root. Quick example:

// New Entity type which conforms for `HasCollision`
class CollisionAnchorEntity: Entity, HasAnchoring, HasCollision { }

// Transfer scene contents
let collisionAnchor = CollisionAnchorEntity()
collisionAnchor
    .children.append(contentsOf: originalAnchor.children)
collisionAnchor.anchoring = originalAnchor.anchoring

// Create CollisionComponent for bounds of scene
let sceneBounds = collisionAnchor
    .visualBounds(recursive: true, relativeTo: collisionAnchor)
let collisionShape = ShapeResource
    .generateBox(size: sceneBounds.extents)
    .offsetBy(translation: sceneBounds.center)
collisionAnchor.collision = CollisionComponent(
                                shapes: [collisionShape]
                            )

// Install gesture on new anchor
arView.installGestures(for: collisionAnchor)

Unfortunately, we don’t discuss our future plans. However, we are aware that our tools haven’t been updated in a few years and could use some new features. Could you share what features you are looking for us to add?

There are various third-party DCCs with great USD support that let you create complex 3D object with textures and export as USD. You can then use Reality Converter to convert those to USDZ to import into Motion.

Another approach: three.js (web render engine) can actually create USDZs on the fly from 3D scenes. A colleague used that recently for USDZ AR files with changeable textures on https://webweb.jetzt/ar-gallery/ar-gallery.html

Also take a look at the Explore USD tools and rendering session tomorrow. You can now change materials properties in RealityConverter! (edited)

Another thing that might help for making quick adjustments: the browser-based three.js editor at https://threejs.org/editor.

Yes, if you can export your assets to FBX, glTF or OBJ, you can convert them to USDZ using Reality Converter, which is compatible with ARKit and RealityKit

We are aware of a bug in macOS Ventura/iOS 16 that is causing the lighting to appear darker. Please feel free to file a bug report on Feedback Assistant about this.

Post processing effects are not supported in RC right now, only in a RealityKit app. However, feel free to file a feature request on Feedback Assistant about this.

I don’t think we have any guidelines about this, since exporting/saving a scene is not supported by the current APIs. ModelIO seems like a reasonable solution to me, but you might also want to file a feature request for this on Feedback Assistant.

Multiuser AR experiences can be created using the SynchronizationComponent

This is a good tutorial (along with sample code) on building collaborative AR sessions between devices:

https://developer.apple.com/documentation/arkit/creatingacollaborative_session

Generally Systems are designed to operate on Entities (within Scenes) and their Components. Each System can be updated against multiple Scenes (and the Scene’s entities).

If you have state that you want to be represented with a System, one method to do that is to have a root entity that holds a “System component”.

There’s more information on System’s in last years WWDC session and on the developer documentation website:

• Dive into RealityKit 2
• https://developer.apple.com/documentation/realitykit/system/update(context:)-69f86

I’d recommend this year’s WWDC Xcode session, What's new in Xcode, for what’s new in debugging. And there have been many other excellent sessions over the years on debugging.

That said if you believe it may be RealityKit or another system framework responsible for leaking the entities we’d ask you to file a Feedback Item on http://feedbackassistant.apple.com if you haven’t done so already.

We don’t currently have a recommend method for doing this and as such would love to see a feedback item explaining what you need and a use case explaining it. That would be wonderful.

That said your method should in theory work and we’d also love to see feedback item describing the issues you’re seeing.

Bug Reporting

Video textures are currently not supported through Reality Composer UI. However, if your .rcproj is part of an Xcode project, you can use the RealityKit VideoMaterial api to change the material of your object in the scene at runtime.

I don’t believe there is a built in way, but loading a USDZ into a RealityKit scene as a ModelEntity is very simple

While we don’t discuss specific future plans, we always want to make RealityKit as easy to use for everyone as we can.

We’d ask you to post your issues and/or suggestions to Bug Reporting

I’d love to find out more about what you find too cumbersome. Thanks!

This should be fixed in iOS 16.

You can host RealityKit content inside SwiftUI views with UIViewRepresentable

If you’re asking if you can use SwiftUI content within RealityKit - there is no direct support for that at present and we’d ask you to file a feedback item explaining your use-case for that feature.

Bug Reporting

You may want to check with the SF Symbols team to confirm this is not possible yet, and also file a feature request on feedback assistant.

A bit of a hack solution but you may be able to get this work via drawing the Symbol to a CGImage and passing that image in as a texture.

Check out this page:

Augmented Reality

To learn more about RealityKit and ARKit, I would recommend starting with our documentation and videos. Here are a few links to help you get started:

• RealityKit
• More to explore with ARKit 6
• AR Creation Tools
• Videos > Augmented Reality

You can also always ask questions on Developer Forums 💬

Suggestion from non-Apple developer:

Reality School on YouTube

I would recommend using ReplayKit or ScreenCaptureKit to record your app screen to stream / share

This is not currently possible to do in a USD, but you should submit the idea to https://feedbackassistant.apple.com.

You can however do some particle effects in an app by using RealityKit's CustomShaders.

Depending on how complex your effect is, you can also bake your particle effects to regular mesh + bones animation :sparkles:

In many cases you can also create a pretty convincing effect just by scaling/rotating a few planes. Example link (no USDZ behind that right now, but you get the idea - this is just two simple meshes for the particles)

Yes, if you can export your assets to FBX, glTF or OBJ, you can convert them to USDZ using Reality Converter, which is compatible with ARKit and RealityKit

You can use the installGestures function on ARView. Keep in mind that the entity will need to conform to HasCollision.

To do this you could create your own CollisionComponent with a custom mesh and add it to your entity or you could simply call generateCollisionShapes(recursive: Bool) on your entity. Putting it all together, you can use .loadModel/.loadModelAsync, which will flatten the USDZ into a single entity. Then call generateCollisionShapes and pass that entity to the installGestures function. This will make your USDZ one single entity that you can interact with.

Yes, you can use ARView.snapshot(...)

If you want, you can change the background of the ARView there:

ARView.Environment.Background

If you call .clone(...) on an Entity, the clone will re-use the same meshes.

You can apply CustomMaterials & CustomMaterial.SurfaceShader to achieve certain cool effects for entities!

From the metal side you can call discard_fragment()

We don’t have any materials you can apply to objects to make them participate in the same shadows as ground planes. However, you can enable shadow casting from directional and spot lights via DirectionalLightComponent.Shadows and SpotLightComponent.Shadows. This may alter the overall lighting of your scene though.

Alternatively, we do have CustomMaterial, which allows you to create custom materials via Metal, but for this use-case may not be able to get you the desired effect.

We’re always looking to improve RealityKit, so would appreciate if you submitted a request for this via https://feedbackassistant.apple.com/

There’s no API in RealityKit to capture video. That said there are system level APIs to capture screen recordings and I wonder if that would be useful for you:

ScreenCaptureKit

I’d suggest filing a feature request with your use-case. Thanks!

Bug Reporting

Hi! We have a number of WWDC sessions covering RealityKit and Reality Composer which is a great place to start.

• Building AR Experiences with Reality Composer
  • Sample Code: Creating a Game with Reality Composer

• Building Apps with RealityKit
• What’s new in RealityKit
• Dive into RealityKit 2
  • Sample Code: Building an Immersive Experience with RealityKit
  • Sample Code: Creating an App for Face-Painting in AR

• Explore advanced rendering with RealityKit 2

There’s also a great guide on building a SwiftStrike game: SwiftStrike: Creating a Game with RealityKit

RealityKit has a CustomMaterial API which allows you to create custom Metal-based materials. I’d recommend our Explore advanced rendering with RealityKit 2 WWDC talk to learn more.

There is also a great resource on Custom Shader API that gives more details on the APIs available in Metal.

There’s no support for that in Reality Composer currently. As always a feature request filed on Bug Reporting would be most appreciated.

There’s also no method to export USDZ from RealityKit and again feature requests appreciated. Thank you!

You can certainly load a model and preserve your hierarchy, then use the entity name or another attribute to find an entity, then hide/show it with Entity.isEnabled

Look at EntityQuery for finding entities efficiently.

“Maybe” 🙂

So you can certainly take MTLTextures and convert them (if they’re configured correctly) into CVPixelBuffers for AVFoundation to consume.

That said it’s really not the intended use case of RealityKit's post processing functionality and I wouldn’t be surprised if either it doesn’t work as you’d expect or if we break you in the future.

Sounds like a great feature request though - Bug Reporting

Currently Reality Composer does not support this. This sounds like a great feature request and we would appreciate if you can file feedback through Feedback Assistant.

If you are willing to jump into code…

You can use the AudioPlaybackController returned from the playAudio API to play, pause, etc. You can also use AudioFileResource to to add / replace audio on entities.

Instancing is mostly abstracted away behind the Entity.clone() method.

Level of detail is not currently exposed as API and we’d recommend filing a feature suggestion on Bug Reporting

That said you can implement Level of Detail yourself (probably using custom Systems and Components) although we understand that may not be ideal. Please file feature suggestions regardless!

While we do not currently support custom render passes, we have support for post process effects. Please file a feature request through Feedback Assistant if your use case requires more customization 🙏

There are various sorts of lighting in RealityKit - you might want to start here perhaps?

PointLight

(see the Cameras and Lighting section in the docs)

But looks like we don’t support lighting in Reality Composer unfortunately so I’d suggest filing a feature suggestion:

Bug Reporting

You can use ARKit inside a SwiftUI app. You can also use RealityKit to build ARKit apps in a declarative way.

Here are the links to resources and sample code to help you get started:

• ARKit
• RealityKit

RealityKit exposes the ARSession through this API:

https://developer.apple.com/documentation/realitykit/arview/session

You can set the delegate on it to listen to ARKit delegate events.

We understand that the number of frameworks can be daunting sometimes. However as you alluded to, we try and offer "high level" frameworks to try and meet developers' needs out of the box, for example, being able to use RealityKit for rendering instead of the lower level Metal.

That said, Apple provides several tutorials and code samples to introduce developers into the various frameworks, e.g.:

Building an Immersive Experience with RealityKit

Another great resource are WWDC videos, which go back several years in order to build a solid understanding of a particular framework or technology.

ARKit doesn’t support physics but rather detects the surrounding scene to allow RealityKit to handle virtual objects. RealityKit does support rigid body physics and a good place to start looking is at the physics APIs here:

Preliminary_PhysicsRigidBodyAPI

Are you referring to https://developer.apple.com/documentation/metal/resource_loading?

No there is no re-localization in RoomPlan. But we expose the ARSession so you could fallback to ARKit for re-localization.

I don’t believe there is a built in way, but loading a USDZ into a RealityKit scene as a ModelEntity is very simple

RoomPlan is the name of the SDK. You’ll want to refer to those APIs as RoomPlan instead of ScanKit.

Yes, please ask this during the object capture lounge tomorrow. But you should be able to modify after export and re-render.

You would need to use the RoomCaptureSession API and subscribe to a delegate to get those updates which contain the Surfaces and Objects. You can then process that data and render it as per your liking.

The recommended maximum size of the room is 30 x 30 feet.

Yono's Note: This is about 9 x 9 meters.

RoomPlan is not an ARAnchor in current design. Thanks for suggestion. We will take into consideration.

From a non-Apple developer:

I created a demo where custom ARAnchors are created for RoomPlan objects. The same could be done for surfaces and then saved to a world map:

https://github.com/jmousseau/RoomObjectReplicatorDemo

Multiuser AR experiences can be created using the SynchronizationComponent

This is a good tutorial (along with sample code) on building collaborative AR sessions between devices:

https://developer.apple.com/documentation/arkit/creatingacollaborative_session

Hi! We have a number of WWDC sessions covering RealityKit and Reality Composer which is a great place to start.

• Building AR Experiences with Reality Composer
  • Sample Code: Creating a Game with Reality Composer

• Building Apps with RealityKit
• What’s new in RealityKit
• Dive into RealityKit 2
  • Sample Code: Building an Immersive Experience with RealityKit
  • Sample Code: Creating an App for Face-Painting in AR

• Explore advanced rendering with RealityKit 2

There’s also a great guide on building a SwiftStrike game: SwiftStrike: Creating a Game with RealityKit

It depends which approach is best — each have a set of pros and cons based on what you want out of the experience.

It can be compelling to run through AR blocks if you want to emphasize lots of user motion in an experience and the scale of the experience is quite large — good for apps that can take advantage of wide open spaces.

Tapping them is more immediate and indirect so if you wanted to destroy a tower quickly or something like that then that would be the way to go — and I could see that being very satisfying to trigger many physics objects to react at once.

I think the same would apply to a button press, it’s an indirect way to trigger it if the experience requires rapidly knocking them down.

Overall I think it’s up to what you want the experience to be, and maintaining internal consistency with other interactions within the app.

Swiftstrike and Swiftshot are great example apps that use similar techniques.

The depth map surfaced through the Scene Depth API does align with the wide angle camera and should correspond to the camera transform available through the ARFrame.

Here is a sample code that generates a colored point cloud by combining the wide angle camera image and depth map:

Displaying a Point Cloud Using Scene Depth

If you still see some issues, I recommend filing a bug through the feedback assistant at Bug Reporting

We do not have any resources for this.

You should be able to use the wide angle camera and camera transform to generate texture maps for the meshes but unfortunately we do not have any resources or sample code showing that.

We do have this sample code showing how to generate colored point clouds using the scene depth API, hope it is of some help.

Displaying a Point Cloud Using Scene Depth

RoomPlan is not an ARAnchor in current design. Thanks for suggestion. We will take into consideration.

From a non-Apple developer:

I created a demo where custom ARAnchors are created for RoomPlan objects. The same could be done for surfaces and then saved to a world map:

https://github.com/jmousseau/RoomObjectReplicatorDemo

Thanks for the suggestion. Here are some existing sample code that uses Metal with ARKit:

• Displaying a Point Cloud Using Scene Depth
• Creating a Fog Effect Using Scene Depth
• Displaying an AR Experience with Metal

We understand that the number of frameworks can be daunting sometimes. However as you alluded to, we try and offer "high level" frameworks to try and meet developers' needs out of the box, for example, being able to use RealityKit for rendering instead of the lower level Metal.

That said, Apple provides several tutorials and code samples to introduce developers into the various frameworks, e.g.:

Building an Immersive Experience with RealityKit

Another great resource are WWDC videos, which go back several years in order to build a solid understanding of a particular framework or technology.

Glad you’re also a fan of the new desk view feature. There are potentially two solutions to this:

1. Do a single perspective projections for the whole image
2. Use a per-pixel correction like you suggested

Both come with their own benefits and drawbacks. Please check out our documentation for implementing the second approach:

Displaying a Point Cloud Using Scene Depth

ARKit allows streaming video from only one camera at a time. Which camera is used is determined by your configuration (e.g. ARFaceTrackingConfiguration will use the front facing camera, ARWorldTrackingConfiguration will use the back wide camera).

You can, however, enable face anchors detected by the front camera in an ARWorldTrackingConfiguration with userFaceTrackingEnabled. Vice versa, you can enable isWorldTrackingEnabled in an ARFaceTrackingConfiguration to benefit from 6DOF world tracking.

Check out this developer sample:

Combining User Face-Tracking and World Tracking

RoomPlan is the name of the SDK. You’ll want to refer to those APIs as RoomPlan instead of ScanKit.

You may want to check with the SF Symbols team to confirm this is not possible yet, and also file a feature request on feedback assistant.

A bit of a hack solution but you may be able to get this work via drawing the Symbol to a CGImage and passing that image in as a texture.

ARKit configures the cameras according to the selected configuration. Capturing from another camera while an ARKit session is running is not supported.

Yes, there are various third-party Digital Content Creators (DCC) that let you create skeletons and Reality Converter lets you convert other file formats with skeletons to USD.

• Autodesk Maya - USD
• Blender - USD

Are some example Digital Content Creation tools that can help you create rigged skeletons for characters exported to USD.

Yes, you can use ARView.snapshot(...)

If you want, you can change the background of the ARView there:

ARView.Environment.Background

That’s a good question.

I think you can get some of the way there via the ARKit APIs to get the current frame.

You can also toggle the mode of an ARView to switch it to .nonAR view - then use ARView.snapshot() to grab a snapshot of the virtual content. And then switch it back.

However, I don’t believe that would give you exactly what you want - I think the ARView snapshot would not necessarily have a transparent background (if that’s what you need). And even then the performance of this may not be great.

You could also try setting the Environment background color to something with 100% alpha.

I’d suggest filing a feature request for this with Bug Reporting

There is no dedicated talk about Spatial framework. It provides core functions that can be used with any 2D/3D primitive data.

We do not have a sample code that uses ARKit together with spatial audio (PHASE). However, this is a great question, can you please send us a request through Bug Reporting

I would recommend using ReplayKit or ScreenCaptureKit to record your app screen to stream / share

We don’t discuss future plans, but gathering developer feedback is important to us so we’d ask you to post your request to

Bug Reporting

You can host RealityKit content inside SwiftUI views with UIViewRepresentable

If you’re asking if you can use SwiftUI content within RealityKit - there is no direct support for that at present and we’d ask you to file a feedback item explaining your use-case for that feature.

Bug Reporting

You can use ARKit inside a SwiftUI app. You can also use RealityKit to build ARKit apps in a declarative way.

Here are the links to resources and sample code to help you get started:

• ARKit
• RealityKit

Reality Converter makes it easy to convert, view, and customize USDZ 3D objects on Mac. For more information, visit:

https://developer.apple.com/augmented-reality/tools/

Yes, on macOS, you can open [from the menu bar] Reality Composer ▸ Settings [or Prefences on older versions of macOS] and check Enable USDZ export

I don’t think we have any guidelines about this, since exporting/saving a scene is not supported by the current APIs. ModelIO seems like a reasonable solution to me, but you might also want to file a feature request for this on Feedback Assistant.

There are great USDZ examples on Quick Look Gallery

For example you have the Lunar Rover from For All Mankind

We've also added new documentation to help you generate better USD assets here:

Creating USD files for Apple devices

I don’t believe there is a built in way, but loading a USDZ into a RealityKit scene as a ModelEntity is very simple

Yes, please ask this during the object capture lounge tomorrow. But you should be able to modify after export and re-render.

You would need to use the RoomCaptureSession API and subscribe to a delegate to get those updates which contain the Surfaces and Objects. You can then process that data and render it as per your liking.

I would suggest filing an enhancement request on feedback assistant for this

There are different USD renderers across our platforms but each serve a different purpose.

Here is a developer document that explains these different USD renderers and what their feature sets are

Creating USD files for Apple devices

This is not currently possible to do in a USD, but you should submit the idea to https://feedbackassistant.apple.com.

You can however do some particle effects in an app by using RealityKit's CustomShaders.

Depending on how complex your effect is, you can also bake your particle effects to regular mesh + bones animation :sparkles:

In many cases you can also create a pretty convincing effect just by scaling/rotating a few planes. Example link (no USDZ behind that right now, but you get the idea - this is just two simple meshes for the particles)

There are various third-party DCCs with great USD support that let you create complex 3D object with textures and export as USD. You can then use Reality Converter to convert those to USDZ to import into Motion.

Another approach: three.js (web render engine) can actually create USDZs on the fly from 3D scenes. A colleague used that recently for USDZ AR files with changeable textures on https://webweb.jetzt/ar-gallery/ar-gallery.html

Also take a look at the Explore USD tools and rendering session tomorrow. You can now change materials properties in RealityConverter! (edited)

Another thing that might help for making quick adjustments: the browser-based three.js editor at https://threejs.org/editor.

Yes, if you can export your assets to FBX, glTF or OBJ, you can convert them to USDZ using Reality Converter, which is compatible with ARKit and RealityKit

Yes, there are various third-party Digital Content Creators (DCC) that let you create skeletons and Reality Converter lets you convert other file formats with skeletons to USD.

• Autodesk Maya - USD
• Blender - USD

Are some example Digital Content Creation tools that can help you create rigged skeletons for characters exported to USD.

You can assemble different USDZ assets together to build out a larger scene in Reality Composer and add triggers and actions to individual assets within the project

Yes, if you load the USDZ with Entity.load(...) or Entity.loadAsync(...) you can traverse the hierarchy and modify the individual entities.

You’d want to use Entity.isEnabled in this instance to hide/show a node.

Note that .loadModel will flatten the hierarchy whereas .load will show all entities

You can use the installGestures function on ARView. Keep in mind that the entity will need to conform to HasCollision.

To do this you could create your own CollisionComponent with a custom mesh and add it to your entity or you could simply call generateCollisionShapes(recursive: Bool) on your entity. Putting it all together, you can use .loadModel/.loadModelAsync, which will flatten the USDZ into a single entity. Then call generateCollisionShapes and pass that entity to the installGestures function. This will make your USDZ one single entity that you can interact with.

Hey there, we’d definitely be interested in hearing more about your idea.

I’d suggest submitting the suggestion at Bug Reporting

If you have your USDZ content on the web you can check out the AR Quick Look functionality for such things at:

Adding an Apple Pay Button or a Custom Action in AR Quick Look

As far as I know there isn’t currently a way to do such a thing directly from a USDZ sent from iMessage, but I can pass that request along.

There’s no support for that in Reality Composer currently. As always a feature request filed on Bug Reporting would be most appreciated.

There’s also no method to export USDZ from RealityKit and again feature requests appreciated. Thank you!

You can use Reality Composer to create a scene attached to an image anchor. You can then export the scene to a USDZ or a Reality File.

See Selecting an Anchor for a Reality Composer Scene

Currently Reality Composer does not support this. This sounds like a great feature request and we would appreciate if you can file feedback through Feedback Assistant.

If you are willing to jump into code…

You can use the AudioPlaybackController returned from the playAudio API to play, pause, etc. You can also use AudioFileResource to to add / replace audio on entities.

ARKit itself has no notion of rendered content. Content (USDZ) is commonly handled by the rendering engine on top of ARKit like RealityKit, SceneKit, Metal, etc.

In order to learn more about USDZ and how to efficiently use it we recommend this talk.

Understand USD fundamentals

Creating a model for Object Detection and creating a textured mesh with Object Capture are two different use cases with separate workflows, we do not offer a tool to convert from one to another. That sounds like a great use case though, I encourage you to file a feature request.

Feedback Assistant

ARKit doesn’t support physics but rather detects the surrounding scene to allow RealityKit to handle virtual objects. RealityKit does support rigid body physics and a good place to start looking is at the physics APIs here:

Preliminary_PhysicsRigidBodyAPI

We don’t currently have a recommend method for doing this and as such would love to see a feedback item explaining what you need and a use case explaining it. That would be wonderful.

That said your method should in theory work and we’d also love to see feedback item describing the issues you’re seeing.

Bug Reporting

Video textures are currently not supported through Reality Composer UI. However, if your .rcproj is part of an Xcode project, you can use the RealityKit VideoMaterial api to change the material of your object in the scene at runtime.

I would recommend using ReplayKit or ScreenCaptureKit to record your app screen to stream / share

There’s no API in RealityKit to capture video. That said there are system level APIs to capture screen recordings and I wonder if that would be useful for you:

ScreenCaptureKit

I’d suggest filing a feature request with your use-case. Thanks!

Bug Reporting

“Maybe” 🙂

So you can certainly take MTLTextures and convert them (if they’re configured correctly) into CVPixelBuffers for AVFoundation to consume.

That said it’s really not the intended use case of RealityKit's post processing functionality and I wouldn’t be surprised if either it doesn’t work as you’d expect or if we break you in the future.

Sounds like a great feature request though - Bug Reporting

recommendedVideoFormatForHighResolutionFrameCapturing is used for capturing high resolution still images while the session is running.

For 4K video, you should use recommendedVideoFormatFor4KResolution

Note that this feature is only supported on iPad with M1

Yes, using 4k resolution may result in more power being consumed. It may also result in thermal mitigation engaging to keep the device from getting too hot, which may impact performance. At the moment, we are only supporting 4k @ 30hz.

To capture and replay an ARKit session, see an example here:

Recording and Replaying AR Session Data

If you want to capture video in your app in order to do post processing later, you could use and configure an AVAssetWriter to capture a video.

We also provide a camera frame with every ARFrame, see:

ARFrame.capturedImage

ARFrame.capturedImage is just the ‘clean slate’, it doesn’t contain any virtual content rendered on top of it. If you are doing your own rendering and your Metal textures are backed by IOSurfaces. then you can easily create CVPixelBuffers using the IOSurfaces and then pass those to AVFoundation for recording.

Setting videoHDRAllowed means that HDR will be enabled on the formats supporting it; however this is not the case for all video formats.

In iOS 16, ARVideoFormat has a new property isVideoHDRSupported. You can filter the list of the configuration’s supportedVideoFormat to find one where videoHDRSupported is true, and set this format as the configuration’s videoFormat before running the session.

ARKit does not have any hand tracking feature. The Vision framework offers functionality for hand gesture detection.

Detect Body and Hand Pose with Vision - WWDC20

You may find the camera's captured images on the ARFrame and can inject this into Vision. So by combining multiple frameworks you could achieve something close to the requested feature.

We don’t have QR code detection in ARKit. However, you can use the Vision APIs to do QR code detection on the captured image. This VisionKit talk and article might be of interest to you:

• Capture machine-readable codes and text with VisionKit
• Scanning data with the camera

ARKit has no built-in hand or finger detection, but you can use Vision to track hands or detect hand poses. Here is a developer sample illustrating this:

Detecting Hand Poses with Vision

For ARKit feature requests, we encourage you to send us a report in Feedback Assistant

Using data scanner via VisionKit is possible using ARKit. ARKit provides the captured image on the ARFrame. One can inject the ARFrame's captured image into data scanner and obtain information about text.

However, the result will be two-dimensional. If the use-case is to bring the detected text into the AR world in three dimensions one needs to estimate a transform for the 2D text. ARKit does not support this natively but does support custom anchoring.

We don’t have QR code detection in ARKit. However, you can use the Vision APIs to do QR code detection on the captured image. This VisionKit talk and article might be of interest to you:

• Capture machine-readable codes and text with VisionKit
• Scanning data with the camera