Study of areas likely to be procedural

(Polygon Pictures Inc. / Studio Phones)
This document is based on post-seminar discussions. We plan to continue making additions and corrections based on the situation at seminars hereafter.
translated by PPI Translation Team

■Overview


We believe there will be an increase in areas in which we can anticipate positive results from the proceduralization of work which has heretofore been done by hand using artisanal methods; examples of such areas include optimization of video production work, execution of server-side processing, and affinity with machine-learning.
In this document, we should like to give an overview of areas of artistic work now performed manually in which proceduralization is conceivable, while providing explanations of methods generally used at present, and touching on points seen as important in industrial terms as well, with the goal of fostering meaningful discussion during seminars.

Please note that, at this seminar, what we present may include features the moderators are already putting into practice; but for the moment, this document will cover points presently under consideration.


■Future prospects


We hear more and more that various studios are conducting myriad internal trials regarding possibilities such as deep learning or machine learning. For much of this learning, computational clusters are also made use of at the time of learning, but in order to move forward with proceduralization, it seems an increasing number of studios are investigating paradigms high in affinity with these forms of learning at the earliest stage.
For example, it seems that we now see studios designing things such as the operations used in the configuration of various procedures for implementing proceduralization with an awareness of such things as Protocol Buffers and GraphDef, which are contained in a software library called TensorFlow that is famous as a framework for deep learning.

It is true that the great majority of engineers who construct pipelines are familiar with json, but there also seems to be a tendency to attempt to implement acceleration by handling Protocol Buffers in binary, and to develop pipelines in a form suitable for the culture of the particular studio. Also, there appear to be studios exploring similar development using other frameworks such as Caffe or Chainer.

So in this document, we should like to make a brief survey of points that have come into view before the seminar, keeping in mind server-side microservice implementation; and then, by adding opinions that arise during the seminar, to connect this to future considerations.


・TensorFlow
https://www.tensorflow.org

・Protocol Buffers, GraphDef
https://www.tensorflow.org/extend/tool_developers/

・Caffe, Chainer
http://caffe.berkeleyvision.org
https://chainer.org



■Background modeling


Procedural modeling techniques are already in prevalent use for buildings and other structures, and for vegetation such as forests or plants, but demand is rising year by year for production of high-definition, high-density background assets, and we anticipate this continuing hereafter as well.
There are cases in which methods are used whereby, drawing on geographical information from actually existing buildings, houses, roads, street scenes, forests, etc., desired architectural structures are generated procedurally in their entirety; and there is also a growing tendency toward large background asset data size.
In recent years, steady improvement has been seen in the development of rendering software performance and solutions for real-time visualization utilizing GPUs, etc., but, while it has become easier to create large-scale assets through procedural processes, strategies and techniques for handling this data efficiently have also become necessary; and we believe that the importance of optimization of generated assets will continue to increase, of course via instancing and referencing, but also through things like subdivision or systems that do not generate cache files.
When processing is done on the server side (which is also the theme of this seminar), in order to be able to implement these feature with more of a view toward micro services, we anticipate that the issue of increasing data size will become more prominent.
Also, while it seems as though proceduralization which simply pertains to the formation of shapes is progressing, in terms of CG representation, it is desirable to be able to procedurally generate things which have a bearing on quality such as texture maps or displacement maps; and, although solutions also exist to procedurally generate discrete textures, there are few solutions which allow the concurrent generation of shapes along with the generation and mapping of textures; and, since it is conceivable that, by consolidating these forms of procedural processing, the effectiveness of procedurals can be raised, it may be possible to construct a workflow that allows continuous improvement of asset quality as well, so that this seems to be an area in which considerations of procedural processing of background sets can be taken deeper. In other words, it is possible to say that it would be difficult to satisfy the demands made on an actual pipeline if proceduralization is leveraged in only a single area, and that, as we advance with proceduralization in all processes existing in the production flow, an infrastructure design that takes into account a level of flexibility in pipeline architecture will become increasingly important.
Also, in order to depict backgrounds in 3DCG that are stylized in ways often seen in cel-look CG, while the adoption of methods such as projection-mapping hand-drawn textures onto 3D models is often seen, it seems possible, as a result of proceduralization advancing for processes which, based on things like style pattern images for the project, automatically map textures to models after their geometry is generated as 3DCG using procedurals, that the use of 3D background assets in cel-look CG will advance, and the expressive range of cel-look CG will be enlarged. For things like project style pattern images, it does seem possible to structure the necessary data as a library; but, by exploiting machine learning, further automation seems likely; and it also seems necessary to develop a sort of proceduralization for asset management linked with forum features that take into account interactivity between artists.


■Background modeling


Much character modeling features unique design or styles incorporating deformation, making it seem like an area into which procedural methods would be difficult to introduce. In particular, things like hero characters are often designed distinctively, making it likely that, even if a procedural structure were put into place, it would only be usable on that character.
For productions like series which span long terms, when design and style are established to a degree and there are multiple modeling objectives in order to create variations for the characters, procedural methods may be effective, but for short-term projects such as one-off movies, we do not anticipate that the benefits of procedurals will be easily reaped.
On the other hand, when it is desirable to create variations of mob characters and other more general-use characters, procedural methods seem more likely to bear fruit.
They can make it possible to select components from a large-scale asset library to procedurally generate characters, and to create a multitude of combinations of such elements as body type, clothing and hairstyle, allowing us to anticipate improvements in productivity.
Also, in large crowd scenes in which great numbers of mob characters appear, we see the same tendency as with background sets for data to grow to unwieldy sizes; but, by executing procedural processing, including mob character animation, at render time, it may become possible to lighten the load placed on storage by pre-cached data. Although the pre-production phase may be an area into which proceduralization will not make inroads, we do seem to see a tendency for the effort required during the trial-and-error undertaken in this phase to be reduced by proceduralization; and for special assets such as hero assets as well, even if not everything can be generated procedurally, the moderators feel the area in which proceduralization can develop here is in increasing their structural commonalities with other assets.
In cel-look CG, as when using 3DCG to recreate a cel-image visual style, a challenge that is seen to occur regularly during animation is that the shape or movement of hair becomes stiff. The common practice is to model hair beforehand as an asset shaped in clumps to a degree; and when animating these clumps of hair, in order to avoid the erroneous rendering of unintentional outlines, animators are required to animate these clump-shaped models so that they do not penetrate with one another, leading, in our view, to occasional restrictions in the available range of movement.
For example, rather than creating hair animation by changing the shape of the model, the hair could be animated using something like curve data, and then, by procedurally generating the shape of the hair using the curve data as a guide during animation checks or at time of rendering, it is possible that penetration between clump-shaped hair models can be avoided, and dynamic hair animation closer to the cel-look style can be achieved. What this means is, the moderators believe that what may be necessary in future is a more meaningful form of proceduralization than seen at present, one that takes UX into account, including UI design intended for animators.
We hope not only to take up the topic of considerations of character modeling proceduralization during this seminar, but also to continue to hear the opinions of all participants as we carry on this discussion.


■Matte painting


Some things, such as the matte paintings mainly used as distant backdrops, or the 2D backgrounds used in cel-look CG, are rendered using 2D drawing processes. These are often composed of combinations of photographic images, or are based on such images and then stylized to match the style of the project.
Also, a not uncommon method is to create a layout in 3DCG using simple models and then use drawings to polish them up to a high resolution.
A great number of artisanal techniques are included in this work progression, but we also believe there are a number of areas in which proceduralization can be considered.
For example, for things like stylizing a given image into a different artistic style or automated generation of layouts, methods have emerged in recent years such as implementation via linkage with machine learning functionality, and we believe that hereafter an increase will be seen in situations in which procedural processing plays a supporting role for artists. In other words, it can be said that, while proceduralization has heretofore been concentrated in given phases of production such as backgrounds or layout, a future necessary challenge will be to consider ways of connecting two such areas via proceduralization.
Also, things like the shapes of brushes for digital matte painting also seem likely candidates for proceduralization. In most cases, matte painters save, as image data, the shapes of the various brushes they use when painting using digital drawing processes. Depending on the artist, the number of brush shape images can range from a few to dozens, but by using different brush shapes for differing purposes, they lend variety to their visual representations.
When brush images are prepared ahead of time, there is a limit to the number that can be created, but by using procedural brush shape generation in which the base shape, size, scale, pattern, density and other parameters can be controlled, it may be possible to create an environment in which greater subtlety in shades of expression or strokes that reflect artistic intention can be depicted.
While we hope that research will proceed developmentally into such fields as coupling fluid or other simulation to brush strokes, or considering discretization of physical phenomena in order to enable physical simulation in accordance with the artist’s own expressivity, we also believe that the need to develop features with a view turned even more than at present toward the various artisanal and creative methods of artists awaits as a challenge in proceduralization.
Also, in most cases, all matte images are generated in a style that has been established to a degree for each project, so the opposite method, of creating brush patterns from those images, can also be taken into consideration. We also take interest in paradigms such as the mechanical generation of mattes according to defined styles or rules, and believe that production processes in which proceduralization is further promoted, including machine learning, is an area in which significant evolution will continue.


■Rigging


In modular rigging systems (rig systems which enable segmenting individual rigs into modular components and building them), which have been growing more common in recent years, techniques of assembling modules procedurally have become standard, making this another area in which proceduralization seems to be advancing.
The automation and proceduralization of rigging has a close connection to modeling, and a large number of systems are seen which make it possible to do things like automatically generate or reuse rigs by making the topology or structure of model data identical, but when it comes to things like the work of positioning bones or making fine adjustments to weights, it seems to us that these are areas for which integration with the above has not yet progressed.
Also, while it feels as though procedural methods for handling standard human shapes have advanced to a degree, for unusually shaped characters such as creatures or animals, rig work has to be done from scratch each time.
As we consider proceduralization of these tasks, rather than be limited by the requirement to unify topology, we believe a system will be necessary in which bones are positioned as the shape of the model is automatically recognized and understood, and adjustments are automatically made to weights for skinning in a way appropriate to the shape. Also, methods in which bone positions are automatically determined based on machine learning of things like prior set-up data may also be possible to contemplate.
In normal production, since a close relationship exists between rig and things like modeling shape and mesh division shape, rather than merely proceduralizing rig construction, it does not seem impossible to implement a more automated procedural rigging, by constructing an environment in which both modeling and rigging can be conducted more procedurally, with mesh shape and division number automatically optimized based on things like the range of possible movement of the rig.


■Animation


When animating characters and other things during film and television production, the commonest method is to key animation into rig controllers frame by frame. There are situations in which this can result in an enormous amount of working by hand, and can consume much time and effort.
On the other hand, in areas like game production, methods are also seen in which appropriate animation is loaded from an animation library and blended dynamically, adding variation the animation to correspond to the surrounding environment; while, in film and television production, the usual technique of animating each frame painstakingly by hand is due to the way the show itself is directed and the non-interactive, linear nature of the visual storytelling.
For things like the simulation of hair or cloth, procedural processing making use of physically-based calculation algorithms are also conceivable, but there are cases in which elements cannot be used without changes to the shapes resulting from calculations based on physical simulation, and it is often seen that animators must ultimately make modifications to the shape or timing of such elements.
In light of the enormous amount of work done by hand in the animation phase of production, a system which can lighten this load to any degree, however slight, may contribute to greater efficiency in the work of the animator, and to continuing improvements in productivity.
For example, a system which automatically generates supplementary data between key frames—by using machine learning to glean, from large data corpora of hand-generated animation, the habits of the animator, the timing peculiar to the style of that particular show, or the changes in shape brought on through the animation, and then polishing the blocking animation created during the layout stage—is also within the realm of consideration, and there seem to be possibilities to aim for greater efficiency by doing so.
In the proceduralization of animation, the incorporation of procedural techniques which also leave room for the artistic handiwork of animators clearly has practical application, and it seems possible that, by learning the quirks and unique characteristics of the animator, as well as the style of that show or film, via the integration of methods such as machine learning, and then applying these things to procedural processes, support for more efficient generation of animation may come into view.


■Topics around Compositing


In such areas as keying and matte painting, it appears that the techniques involved are being gradually deepened by forms of proceduralization which bundle processes together. As instruments used beforehand to acquire data to be worked with in compositing, such as LiDAR, become more prevalent, various forms of signal processing intended to prevent problems related to the sensitivity inherent in such sensors have tended to come into use, on the assumption that, as processes, they will be reused and refined; and because, in given productions, an increasing number of phases of production have come to rely on such methods as front projection mapping or rear projection, for purposes of coordination between successive phases within a film production pipeline, we anticipate significant increases in the tendency to find solutions within each studio.
For operations like these which cannot be completed only on a PC, although there are times when various sensors or other instruments are provisionally created in-house and employed in the photography phase, etc., it seems to us that there has also been a tendency in recent years toward increasing complexity in production flows in which, as graphics are handled interactively, compositing into photographed imagery of time-consuming processing done via CG is done after photography is complete, and then the final image is refined, etc.
For example, there is a framework known as Gobot, based primarily on the Go programming language, for which a robust development environment is available for such things as devices commonly employed in drones or the media art field, Bluetooth, WiFi, and various IoT and wearable devices, etc. When one considers such possibilities as creating things like tools needed for production without much fuss, procedural construction of filtering for the data captured via sensors during that process, setting up a PerCap system internally, incorporating things like motion graphics generated by user actions, or building rigs for use in animation which move in response to sound, the moderators feel that these fields, which have heretofore not been taken up much in CG production, may also see further developments in proceduralization going forward.


・Gobot
https://gobot.io
https://gobot.io/documentation/platforms/