Show/Hide Mobile Menu

Shaders in RealityKit

Jan 14, 2022

In RealityKit 2 Apple now allows you to create vertex and fragment shaders for materials. You specify the shaders using CustomMaterial, similar to how you add shaders to ShaderMaterial in Three.js. But one major difference with RealityKit's approach is that you modify properties of the existing material types rather than having to start from scratch. You can create a custom material from an existing base material:

let mtlLibrary = MTLCreateSystemDefaultDevice()!.makeDefaultLibrary()!

let surfaceShader = CustomMaterial.SurfaceShader(named: "mySurfaceShader", in: mtlLibrary)
let geometryModifer = CustomMaterial.GeometryModifier(named: "myGeometryModifer", in: mtlLibrary)

var baseMaterial = PhysicallyBasedMaterial()
baseMaterial.baseColor = PhysicallyBasedMaterial.BaseColor(tint: UIColor.blue)

var customMaterial = try! CustomMaterial(from: baseMaterial, surfaceShader: surfaceShader, geometryModifier: geometryModifer)

The above code doesn't actually create a blue custom material - you would still have to set the base color in the shader but you can access the properties of the base material in the shader. Because of this it's probably clearer to use the constructor where you specify the lighting model:

var customMaterial = try! CustomMaterial(surfaceShader: surfaceShader, geometryModifier: geometryModifer, lightingModel: .lit)

The lighting model options are: .lit: PBR (physically based rendering) without clearcoat, .clearcoat: PBR with clearcoat, .unlit: renders without shading or lighting, similar to UnlitMaterial.

RealityKit's name for fragment shader is surface shader, and for vertex shader is geometry modifier. RealityKit is quite restrictive in what parameters you can pass from Swift to the shaders. You can only pass a vector with 4 components and a texture. The vector can be treated as 4 floats if you want. If that's not enough you could set the standard material properties and access them inside the shader. To pass a float you would do this:

customMaterial.custom.value[0] = 0.1

To pass a 4 component vector (SIMD4<Float>) you would do this:

customMaterial.custom.value = [0, 0, 0, 0]

To pass a texture, you would first create a texture set in the asset catalogue and then do this:

if let resource = try? TextureResource.load(named: "MyTexture") {
  let texture = CustomMaterial.Texture(resource)
  customMaterial.custom.texture = .init(texture)
}

Next you need to create the shaders. You do this by creating a Metal file (File > New > Metal File). Metal Shader Language (MSL) is written in C++ 14. The surface shader and geometry modifier can be in the same file but the functions need to have a visible attribute. Attributes in C++ are written like this: [[visible]]. The name of the function needs to match what you specified above when creating CustomMaterial.SurfaceShader() and CustomMaterial.GeometryModifier() and each function needs to be passed a particular parameter as you can see below. The details of these parameters are specified here.

#include <metal_stdlib>
#include <RealityKit/RealityKit.h>

using namespace metal;

[[visible]]
void mySurfaceShader(realitykit::surface_parameters params) {
  // Get the tint of the base material
  half3 color = (half3)params.material_constants().base_color_tint();
  // Set the base color to the tint
  params.surface().set_base_color(color)
  // Set other material parameters
  params.surface().set_metallic(1);
  params.surface().set_roughness(0.5);
}

[[visible]]
void myGeometryModifer(realitykit::geometry_parameters params) {
  // Move each vertex out by 0.1
  params.geometry().set_model_position_offset(params.geometry().normal() * 0.1)
}

You don't have the type half in GLSL - it's basically a half-precision float. Instead of vec2, vec3, vec4 that you have in GLSL, you have float2, float3, float4 or half2, half3, half4. MSL isn't fussy about always adding decimal points to float literals.

To access a custom parameter in a shader you would do this:

float customParameter = params.uniforms().custom_parameter()[0];

// or

float4 customParameter = params.uniforms().custom_parameter();

To access a custom texture you have to define a sampler and then use it to grab a pixel from it:

constexpr sampler textureSampler(coord::normalized, address::repeat, filter::linear, mip_filter::linear);

[[visible]]
void mySurfaceShader(realitykit::surface_parameters params) {
  float2 uv = params.geometry().uv0();
  half3 color = params.textures().custom().sample(textureSampler, uv).rgb;
  params.surface().set_base_color(color);
}

You can access time (the number of seconds since RealityKit began rendering the current scene) inside a shader with params.uniforms().time() so you don't have to pass it in from Swift.

Related links

Tags