This talk was first presented at DigiPro 2019 by Mill Film Lead Software Developer Victor Yudin.

Authors

Victor Yudin(Mill Film)
Gregory Ducatel(Mill Film)

Abstract

Millefiori is a visual effects application designed to allow users to view and edit a series of large scenes, leveraging Pixar Universal Scene Description in its core, and Qt/QML for the UI components.
While Millefiori was initiated as a sequence editor, its USD-based core has since formed the basis of the entire pipeline at Mill Film, from editing a USD stage to generating a review QuickTime. Development of the technology was a successful collaboration of developers led by Mill Film, MPC R&D, and Technicolor Research and Innovation.

Introduction

The primary purpose of Millefiori is to provide the ability for supervisors to review and edit multiple shots at once. It gives the ability to switch between scenes, navigate them, inspect the full history based on a composition of layers, and apply minimal modifications (such as changing shaders, lights, various attributes, transformations, flags) and comments to objects, all using an artist-friendly interface as shown in Figure 1. The modifications can be applied to individual shots or holistically across all the shots at once. It’s built on top of the USD-based core and provides the interfaces to the external modules for accessing the USD Stage.

One of the aims of our development process is to reuse as much common code as possible, sharing development between multiple business units of Technicolor Film and TV. As we had strong requirements for the core of the pipeline to capitalize on Open Source projects, Millefiori is using other industry standards such as Open Shading Language and Qt/QML. With this in mind, modules created for our application are compatible with the other DCCs across our pipeline such as Maya, Houdini, Mari, Nuke and Katana.

To display large scenes efficiently, we are using Pixar’s Hydra. Hydra is a highly-scalable, multi-pass rendering architecture that ships with the USD distribution. Since OpenGL slows down with the increase of our dataset complexity, we developed an NVIDIA RTX backend called Prism [Mill Film 2019]. It is a real-time, highquality viewport that leverages NVIDIA OptiX and Open Shading Language to support exceptionally complex scenes with shading and textures.

We are presenting our work in which Millefiori and its components are integrated throughout the Mill Film pipeline, driving a variety of modules continuously used on our productions.

Background

Before the launch of Mill Film, we embarked on building a pipeline almost from scratch, starting with the foundation/core of it. We needed a pipeline able to produce standard computer graphic data, store it, and transmit high volumes of “scene description” between our tools, our departments, and our sites in different parts of the world. We decided to use USD as the core of this system because it provides an excellent toolset for the interchange of models and animations.

However, the initial integration of USD into Maya had some weak points that we needed to address, and we wanted to enhance the capabilities of the Hydra viewport. This led to us creating two USD-related modules for Maya before we even started working on Millefiori itself. Despite being initially hosted in Maya, we designed
these modules to be application-independent, interacting directly with USD Stage data using the Python API and the notification system of USD. The components were ultimately reused in the Millefiori project.
1. To allow multiple artists to collaborate on the same assets and scenes, USD has an interesting layering system that allows several artists in different departments to work on the same asset in their own file (or “layer”), all of which are in the end combined. This ability enables each artist to work independently without erasing or editing any other artist’s work and helps to provide a transparent review of changes. Unfortunately, the USD integration in Maya didn’t provide support for robust layer manipulation. Layers in USD can contain nested sublayers, and there was no way to edit or even inspect the layer hierarchy (in Maya). To solve this point we developed a Layer Editor that later became a part of Millefiori. With it, the user can control the order of sub-layers and some of their
properties like mute/unmute, add existing layers to the sub-layer stack, or create new layers in memory. We provided methods for selecting any sub-layer in the stage for editing. Thus it’s possible to control which sub-layer contains a specific modification. For example, the user can create a different set of lights in different layers in memory and save them to the disk or send them to the asset management system.

2. USD provides a high-speed data model and rich APIs that organize concepts such as prim-vars or variants. Despite such a great technical foundation, inspecting a stage using the vanilla Maya pxrUsdProxyShape transparently wasn’t possible in the native integration. To solve this point we developed Armilla. Armilla is a Python/Qt tree view component that allows users to properly inspect and edit the hierarchy of the primitives in a USD Stage.
Hierarchy-based scenegraphs (with parent-child relationships and properties) are very familiar to VFX artists, making Armilla an immediately-intuitive component. To manage the complexity found in a typical VFX production, it has advanced filtering to focus an artist’s attention and effort. Lastly, in addition to providing information about the stage, it also allows authoring via editing of USD attributes and variants. As shown on Figure 3, we naturally reused it as one of the modules of Millefiori.

Sequence Editing

In a pipeline, a sequence is a series of shots that work together. With the wish to provide a consistent workflow for sequences/episodes, we needed to provide a tool that would help leads and supervisors validate and modify a complete sequence and publish to their teams a good representation of the changes that need to be applied to the project.

The vast majority of industry toolsets for editing or commenting on sequences are typically video-based. When using video editing and playback software, artists need to pre-render their work to get it reviewed. In order to adjust or modify their scenes once the review is done, the artist would have to open all their retakes in a 3D editing software like Maya to perform even the most straightforward feedback, and then release only the changes. This is an involved and time-consuming process, even if publishing time is fast and a modern pipeline would automatically push the modification to the 3D assembly of each shot.

The timeline module of Millefiori, which we callled Millisecond, is a one track non-linear editor, developed to allow changing the playback timing of single shots in the overall motion sequence. Despite USD-level support for references and sublayers to include an offset and scaling of time, USD can’t ultimately control the time of the root layer. We therefore use our own very simple retiming in the Millisecond module, and store the final timing data in the metadata of the root layer of the shot.

A 3D interactive sequence editor provides the ability to modify and validate the changes in the context and continuity of the project with a comfortable feel and look, including navigating around the scene to better understand the resulting image. With Millefiori we have been exploring that aspect of the pipeline that is not covered by any industry tools.

CHARACTERISTICS

USD is at the core of the Mill Film pipeline, from modelling through to the delivery of the shot. To manage our assets and connect the entire pipeline together, we are also using a proprietary Asset Management System, written by the Core Engineering group of MPC R&D, called Tessa [MPC R&D 2019]. Since every asset or piece of shot work which we publish to Tessa has its own USD representation, the user can instantly inspect any point of the pipeline with Millefiori, without any data conversion of the scene between different formats.

Millefiori is represented by the core and some modules attached to the factory, as shown in Figure 3. The factory system supports the ability to have modules written in C++ while others are written in Python. To register modules, we use the USD mechanism for discovering and registering plugins. We found it to be very flexible, containing methods that indicate whether or not a plug-in is currently loaded and whether a plugin is a Python module or a C++ library. It is cross platform and supports both shared and static libraries. Thus, it is possible to compile Millefiori either as a monolithic application or as a set of dynamic libraries.

For the user experience, we use a hybrid Qt and QML solution. We use Qt due to its status as the industry standard, compatible with every DCC. The primary goal of using QML is to explore Qt Quick WebGL streaming, the new Qt Quick back-end that uses a JavaScript API for rendering 2D and 3D graphics within any compatible web
browser without the use of plugins (therefore, using C++ code for a web-based application).

We also have a variety of command modules which are able to perform modifications to the current stage. A simple example of a command module would one which deforms a mesh and adds normals. All such modules have full access to the USD Stage.
USD has a flexible notification subsystem that allows external systems to be triggered when significant events have occurred. We use this as the primary mechanism for the communication between modules inside Millefiori, allowing the modules to be aware that a specific event has occurred somewhere in the application. Furthermore, we extended the notification subsystem with time changes, selection, and multi-stage events.

Since the modules we developed are operating with USD directly, they can work outside of the Millefiori core. As an example, our USD scene graph editor (Armilla) is available in Maya, as shown in Figure 2. We have embedded componentes of Millefiori within Maya and Katana to allow our artists to leverage their existing skills and experience, and to take advantage of familiar tools. Due to our precompiled modules, Millefiori can be efficiently scaled and used in different parts of the pipeline.

Millisecond is a QML based module that displays all the loaded USD stages on the timeline. Armilla is a Qt-based widget that displays the scene hierarchy and primitive properties. Milligraph is a QML module that allows inspecting the shader graphs. Millilambert is a QML based viewport that uses Pixar Hydra with Prism backend to render USD stage.

Rendering

To visualize the USD stages, we use Hydra with the Prism backend, mentioned earlier. Prism aims to be used from simple visualization of extremely complex scenes to the near finish results comparable to RenderMan, and we find it a natural fit as, with ever-increasing scene complexity, a raytracer is much faster than the standard Hydra OpenGL backend that uses rasterization.
Prism is highly configurable and provides performances that are fully scalable between speed and quality. Due to the render delegate plug-in for Hydra, Prism is natively compatible with software like Millefiori or commercial products such as Maya and Houdini. Prism can provide both single GPU and multiple GPU rendering solutions.
We also benefit from the excellent support of NVIDIA which frees us from worrying about low-level development and, instead, allows us to concentrate on features.

Workflow

At a production level, CG Supervisors load Millefiori with a sequence or master/child shots. The sequence is initialized with the latest versions available. Using our fast preview system, supervisors can inspect the full history of each shot based on a composition of layers. As mentioned earlier, modifications are applied either to all the shots at once or just to specific shots. When the modifications are completed, using a specific layer called “supervisor layer”, the complete set of changes can be published back to the Asset Management System.
Our pipeline gives artists an automated message when a “supervisor layer” is available to them. The artist can see the changes and the comments within the DCC without affecting the work done.
The artist can accept the changes and import them to the shot, or use the changes as reference and redo the work themselves. This approach allows validation in the context of shots (and in 3D) without exposing unnecessary complexity to the departments. Another exciting feature is the ability to see the GUI of Millefiori
in a web browser, with the bulk of the system running on a remote host. In practice, it means that Millefiori can run be exposed to artists and supervisors working on potentially light-weight client machines, while leveraging the maximum power of a remote workstation using, for example, an NVIDIA RTX video card.

Collaboration

Close collaboration between Mill Film, MPC R&D, and Technicolor Research and Innovation was essential to manage the robust, fluid development process, and to ensure that in the end Millefiori is artist-friendly and intuitive for technical and non-technical users. The result of such collaboration saw us producing the first only version three months after the initial development started.

Future Work

In addition to optimization, we intend to capitalize on its general purpose and extend it into new areas of our pipeline/workflow. One area of interest is
to use Millefiori as our primary tool for layout in general, giving us the ability to capitalize on a unified toolset and user interface that will help us build and expose tools and modules to better perform early-stage tasks (e.g., camera setup, in-context layout, retime of shots, etc) in order to potentially perform simple lighting tasks in the future.

Conclusion

We have shown Millefiori and how the original goals of the platform are still in sight. Although the system is still in early stages, it has already resulted in faster and newer workflows with much more responsive interaction cycles. It also validated our initial assessment of Pixar USD and our choice in using Open Source technologies as the foundation of a modern pipeline.

References

Mill Film. 2019. NVIDIA GTC 2019: S9197 – Prism & RTX . https://developer.nvidia.
com/gtc/2019/video/S9197. [Online; accessed 01-June-2019].

MPC R&D. 2019. Tessa: Asset Management System. https://www.mpc-rnd.com/
technology/tessa/. [Online; accessed 01-June-2019].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
DigiPro ’19, July 27, 2019, Los Angeles, CA, USA
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6799-8/19/07. . . $15.00
https://doi.org/10.1145/3329715.3338882