[Showcase] An Approach to Photogrammetry, The new generation of Photorealism - Unreal Engine 5

Revised on June 2023, Original Post in Arstation, 2021

Photogrammetry in videogames is not a new concept, we were using it for years. How is this work? Basically we take pictures/scan objects or textures from the real world and we process and clean those objects digitally to use them in our scenarios. With it we can put real objects in our game/project, and what's more realistic than real-world objects? Most of the AAA games are using this technique, some of the big studios creates their own models but now is more and more common the use of libraries. Megascans is the most popular library and now it is integrated with Unreal Engine 5 (Quixel Bridge plugin).

With my past as a photographer and my interest in the new generation of videogames I was very interested in Photogrammetry and its creation from scratch, so I decided to do a small scene to learn more about the workflow and start to use them in my scenes or future work, this is an overview about the process.

The first thing I would like to say is that this process takes time, and there is a lot of cleaning and optimization work, but hopefully, with Unreal 5 and the new Nanite system we will no longer worry about the millions of polys of these objects (or maybe yes?) For this scene I didn't optimize these meshes, each object has millions of polys. Nevertheless, if you need to reduce the polys the photogrammetry software has options to decimate the mesh easily. I will not go into depth about the whole process since it is not a tutorial, but I will do a brief to show how it works.

I started planning the route to scan some objects back in Ely, Cambridgeshire where I lived for a while during the pandemic. At the time of planning, we need to have in mind to avoid strong lights or shadows on the object that we are going to scan, a homogeneous light is ideal because we can have issues if we want to change the lighting in the render or engine later.

There are apps where you can see at what time there are shadows or not on the streets. It is really useful to do the planning before going outside, especially in countries like Spain where we got a strong sun and hard shadows. I planned the day and the time to do it but since I'm living in the UK most of the days are cloudy... so almost every day is good to do photogrammetry here! :D

We need at least 100 pictures per object to have enough references. To take these pictures we have to rotate around the object, from left to right and down to up or vice-versa, making sure that from one picture to another, you have a reference from the previous one, so the program can read it and create the 3D model later. You can also set up a studio at home for individual and small objects and rotate them with studio light. In my case, I took between 180-300 pictures depending on the object, with a 50mm full-frame lens with my Sony A7s II. As you can see in the picture above, once you put the images in the software, you'll see the location of each picture and how it creates the 3D model.

The best practice is to use a chart to make sure that the colors are right and accurate and that they don't depend on the camera settings. I didn't use a chart since they are quite expensive and I didn't buy it just for this test, but I neutralized the color in the settings of the camera and later on Photoshop too. I'll buy one in the future, I used them in my previous work and they are really necessary to work on photogrammetry professionally. I highly recommend the X-Rite ColorChecker Passport Photo

I used Agisoft to create the meshes, it is really great software for this task! Another software probably even more popular is RealityCapture . The process can take hours and even days. The creation of the model in 3D consists mainly of 3 steps:

1. Align the pictures, removing the pictures that are not valid, blurry, etc.

2. Create the tile cloud and density cloud, a preview where we can remove parts that we don't need, make adjustments, etc.

3. Create the Mesh and bake the textures

In between these processes and after building the mesh there is a lot of work on optimization, decimation, and cleanup. Each one of these steps takes hours, depending on your PC and the final resolution of the objects (it will vary depending on the number of pictures/detail and camera resolution)

It can be tedious, but after all the process, the first time you see an object from real life converted into a 3D model with all that detail... is amazing! :D. The renders below were made with Marmoset. It was a great way to show the final models, it was my first time using it since I always do the final renders in Unreal.

After preparing these assets, I wanted to try to use some of them in realistic environments to see how they can look, I attach here some of those tests:

And that's all! I'll continue exploring photogrammetry, as a Filmmaker, the possibility of mixing the real world and videogames sounds like a great opportunity to dig in :)