Automatic 3D Shape Co-Segmentation Using Spectral Graph Method

Co-analyzing a set of 3D shapes is a challenging task considering a large geometrical variability of the shapes. To address this challenge, this paper proposes a new automatic 3D shape co-segmentation algorithm by using spectral graph method.Our method firstly represents input shapes as a set of weighted graphs and extracts multiple geometric features to measure the similarities of faces in each individual shape.Secondly all graphs are embedded into the spectral domain to find meaningful correspondences across the set.After that we build a joint weighted matrix for the graph set and then apply normalized cut criterion to find optimal co-segmentation of the input shapes.Finally we evaluate our approach on different categories of 3D shapes, and the experimental results demonstrate that our method can accurately co-segment a wide variety of shapes, which may have different poses and significant topology changes.     

Modal Locomotion: Animating Virtual Characters with Natural Vibrations

We present a general method to intuitively create a wide range of locomotion controllers for 3D legged characters. The key of our approach is the assumption that efficient locomotion can exploit the natural vibration modes of the body, where these modes are related to morphological parameters such as the shape, size, mass, and joint stiffness. The vibration modes are computed for a mechanical model of any 3D character with rigid bones, elastic joints, and additional constraints as desired. A small number of vibration modes can be selected with respect to their relevance to locomotion patterns and combined into a compact controller driven by very few parameters. We show that these controllers can be used in dynamic simulations of simple creatures, and for kinematic animations of more complex creatures of a variety of shapes and sizes.     

ExquiMo: An Exquisite Corpse Tool for Collaborative 3D Shape Design

We introduce ExquiMo, a collaborative modeling tool which enables novice users to work together to generate interesting, and even creative, 3D shapes. Inspired by an Exquisite Corpse gameplay, our tool allocates distinct parts of a shape to multiple players who model the assigned parts in a sequence. Our approach is motivated by the understanding that effective surprise leads to creative outcomes. Hence, to maintain the surprise factor of the output, we conceal the previously modeled parts from the most recent player. Part designs from individual players are fused together to produce an often unexpected and novel end result. We investigate the effectiveness of collaboration on the output designs by conducting a sequence of user studies to validate the hypotheses formed based on our research questions. Results of the user studies are supportive of our hypotheses that multi-user collaborative 3D modeling via ExquiMo tends to lead to more creative novice designs according to the commonly used criteria for creativity: novelty and surprise.     

Semi-Supervised 3D Shape Recognition via Multimodal Deep Co-training

3D shape recognition has been actively investigated in the field of computer graphics. With the rapid development of deep learning, various deep models have been introduced and achieved remarkable results. Most 3D shape recognition methods are supervised and learn only from the large amount of labeled shapes. However, it is expensive and time consuming to obtain such a large training set. In contrast to these methods, this paper studies a semi-supervised learning framework to train a deep model for 3D shape recognition by using both labeled and unlabeled shapes. Inspired by the co-training algorithm, our method iterates between model training and pseudo-label generation phases. In the model training phase, we train two deep networks based on the point cloud and multi-view representation simultaneously. In the pseudo-label generation phase, we generate the pseudo-labels of the unlabeled shapes using the joint prediction of two networks, which augments the labeled set for the next iteration. To extract more reliable consensus information from multiple representations, we propose an uncertainty-aware consistency loss function to combine the two networks into a multimodal network. This not only encourages the two networks to give similar predictions on the unlabeled set, but also eliminates the negative influence of the large performance gap between the two networks. Experiments on the benchmark ModelNet40 demonstrate that, with only 10% labeled training data, our approach achieves competitive performance to the results reported by supervised methods.     

ArchiDNA: An interactive system for creating 2D and 3D conceptual drawings in architectural design

We present an interactive system called ArchiDNA for creating 2D and 3D conceptual drawings in architectural design. We developed a novel principle of shape generation called match-and-attach by analyzing drawing styles of a contemporary architect, Peter Eisenman. The process consists of user interaction techniques and a set of rules that decide how one or more shapes attach to another shape. One key ingredient of our process is a unique concept for the interactive semi-automatic shape generation that uses the combination of algorithmic rules of a computer and designers’ manual inputs. These techniques enable designers to use CAD software in the early stages of architectural designs to explore conceptual building forms. ArchiDNA dynamically responds to drawing inputs, configures 2D shapes, and converts them to 3D shapes in a similar style. We intend to complement existing CAD software and computational drawing pipelines for intuitive 2D and 3D conceptual drawing creation.     

EM-GPA: Generalized Procrustes analysis with hidden variables for 3D shape modeling

Aligning shapes is essential in many computer vision problems and generalized Procrustes analysis (GPA) is one of the most popular algorithms to align shapes. However, if some of the shape data are missing, GPA cannot be applied. In this paper, we propose EM-GPA, which extends GPA to handle shapes with hidden (missing) variables by using the expectation-maximization (EM) algorithm. For example, 2D shapes can be considered as 3D shapes with missing depth information due to the projection of 3D shapes into the image plane. For a set of 2D shapes, EM-GPA finds scales, rotations and 3D shapes along with their mean and covariance matrix for 3D shape modeling. A distinctive characteristic of EM-GPA is that it does not enforce any rank constraint often appeared in other work and instead uses GPA constraints to resolve the ambiguity in finding scales, rotations, and 3D shapes. The experimental results show that EM-GPA can recover depth information accurately even when the noise level is high and there are a large number of missing variables. By using the images from the FRGC database, we show that EM-GPA can successfully align 2D shapes by taking the missing information into consideration. We also demonstrate that the 3D mean shape and its covariance matrix are accurately estimated. As an application of EM-GPA, we construct a 2D + 3D AAM (active appearance model) using the 3D shapes obtained by EM-GPA, and it gives a similar success rate in model fitting compared to the method using real 3D shapes. EM-GPA is not limited to the case of missing depth information, but it can be easily extended to more general cases.     

Estimating body shape of dressed humans

The paper presents a method to estimate the detailed 3D body shape of a person even if heavy or loose clothing is worn. The approach is based on a space of human shapes, learned from a large database of registered body scans. Together with this database we use as input a 3D scan or model of the person wearing clothes and apply a fitting method, based on ICP (iterated closest point) registration and Laplacian mesh deformation. The statistical model of human body shapes enforces that the model stays within the space of human shapes. The method therefore allows us to compute the most likely shape and pose of the subject, even if it is heavily occluded or body parts are not visible. Several experiments demonstrate the applicability and accuracy of our approach to recover occluded or missing body parts from 3D laser scans.     

Surface reconstruction with data-driven exemplar priors

In this paper, we propose a framework to reconstruct 3D models from raw scanned points by learning the prior knowledge of a specific class of objects. Unlike previous work that heuristically specifies particular regularities and defines parametric models, our shape priors are learned directly from existing 3D models under a framework based on affinity propagation. Given a database of 3D models within the same class of objects, we build a comprehensive library of 3D local shape priors. We then formulate the problem to select as-few-as-possible priors from the library, referred to as exemplar priors. These priors are sufficient to represent the 3D shapes of the whole class of objects from where they are generated. By manipulating these priors, we are able to reconstruct geometrically faithful models with the same class of objects from raw point clouds. Our framework can be easily generalized to reconstruct various categories of 3D objects that have more geometrically or topologically complex structures. Comprehensive experiments exhibit the power of our exemplar priors for gracefully solving several problems in 3D shape reconstruction such as preserving sharp features, recovering fine details and so on.     

A shape grammar interpreter for rectilinear forms

Shape grammars play an important role in a new generation of tools for the analysis and design of products. Up until now there has been numerous attempts to create a general shape grammar interpreter, but most of the existing tools are either very specific in their purpose, have only limited functionality or were programmed for one operating system. In this work, we present a tool named Shape Grammar Interpreter (SGI) for the automatic generation of designs. The developed shape grammar framework allows designers to automatically synthetize designs and to actively participate in the generation process. Great effort has been devoted to provide an interactive way of defining shapes and later using them in shape grammar rules and designs’ generation process. The tool implements two different types of algorithms for the generation of designs. First, Tree-search algorithms which store the state of the generation process in a tree structure and uses traditional tree-search algorithms to find the next rule to apply. Second, and most importantly, an optimized subshape detection algorithm. Hence, subshapes of the existing shapes can be detected and used in the generation process obtaining not only a wider set of designs but potentially more appealing ones. In this paper, we also describe the architecture of the framework and provide a performance evaluation of proposed algorithms, showing a significant gain in performance. Potential applications of our research can be found in the educational field (i.e. architecture and arts) and in the automatic generation of architectural, mechanical and product designs.     

Progressive 3D shape segmentation using online learning

In this article, we propose a progressive 3D shape segmentation method, which allows users to guide the segmentation with their interactions, and does segmentation gradually driven by their intents. More precisely, we establish an online framework for interactive 3D shape segmentation, without any boring collection preparation or training stages. That is, users can collect the 3D shapes while segment them, and the segmentation will become more and more precise as the accumulation of the shapes.Our framework uses Online Multi-Class LPBoost (OMCLP) to train/update a segmentation model progressively, which includes several Online Random forests (ORFs) as the weak learners. Then, it performs graph cuts optimization to segment the 3D shape by using the trained/updated segmentation model as the optimal data term. There exist three features of our framework. Firstly, the segmentation model can be trained gradually during the collection of the shapes. Secondly, the segmentation results can be refined progressively until users’ requirements are met. Thirdly, the segmentation model can be updated incrementally without retraining all shapes when users add new shapes. Experimental results demonstrate the effectiveness of our approach.     

Shape-It-Up: Hand gesture based creative expression of 3D shapes using intelligent generalized cylinders

We present a novel interaction system, “Shape-It-Up”, for creative expression of 3D shapes through the naturalistic integration of human hand gestures with a modeling scheme dubbed intelligent generalized cylinders (IGC). To achieve this naturalistic integration, we propose a novel paradigm of shape–gesture–context interplay (SGCI) wherein the interpretation of gestures in the spatial context of a 3D shape directly deduces the designer’s intent and the subsequent modeling operations. Our key contributions towards SGCI are threefold. Firstly, we introduce a novel representation (IGC) of generalized cylinders as a function of the spatial hand gestures (postures and motion) during the creation process. This representation allows for fast creation of shapes while retaining their aesthetic features like symmetry and smoothness. Secondly, we define the spatial contexts of IGCs as proximity functions of their representational components, namely cross-sections and the skeleton with respect to the hands. Finally, we define a natural association of modification and manipulation of the IGCs by combining the hand gestures with the spatial context. Using SGCI, we implement intuitive hand-driven shape modifications through skeletal bending, sectional deformation and sectional scaling schemes. The implemented prototype involves human skeletal tracking and hand posture classification using the depth data provided by a low-cost depth sensing camera (Kinect?). With Shape-It-Up, our goal is to make the designer an integral part of the shape modeling process during early design, in contrast to the case for current CAD tools which segregate 3D sweep geometries into procedural 2D inputs in a non-intuitive and onerous process requiring extensive training. We conclusively demonstrate the modeling of a wide variety of 3D shapes within a few seconds.     

Handwritten Chinese character recognition using nonlinear active shape models and the Viterbi algorithm

Since Chinese characters are composed from a small set of fundamental shapes (radicals) the problem of recognising large numbers of characters can be converted to that of extracting a small number of radicals and then finding their optimal combination. In this paper, radical extraction is carried out by nonlinear active shape models, in which kernel principal component analysis is employed to capture the nonlinear variation. Treating Chinese character composition as a discrete Markov process, we also propose an approach to recognition with the Viterbi algorithm. Our initial experiments are conducted on off-line recognition of 430,800 loosely-constrained characters, comprised of 200 radical categories covering 2154 character categories from 200 writers. The correct recognition rate is 93.5% characters correct (writer-independent). Consideration of published figures for existing radical approaches suggests that our method achieves superior performance.     

Reconstruction of non-rigid 3D shapes from stereo-motion

Several non-rigid structure from motion methods have been proposed so far in order to recover both the motion and the non-rigid structure of an object. However, these monocular algorithms fail to give reliable 3D shape estimates when the overall rigid motion of the sequence is small. Aiming to overcome this limitation, in this paper we propose a novel approach for the 3D Euclidean reconstruction of deformable objects observed by an uncalibrated stereo rig. Using a stereo setup drastically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach is based on the following steps. Firstly, the stereo system is automatically calibrated and used to compute metric rigid structures from pairs of views. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points which have remained rigid throughout the sequence. The selected rigid points are then used to compute frame-wise shape registration and to robustly extract the motion parameters from frame to frame. Finally, all this information is used as initial estimates of a non-linear optimization which allows us to refine the initial solution and also to recover the non-rigid 3D model. Exhaustive results on synthetic and real data prove the performance of our proposal estimating motion, non-rigid models and stereo camera parameters even when there is no rigid motion in the original sequence.     

Intrinsic Symmetry Detection on 3D Models with Skeleton‐guided Combination of Extrinsic Symmetries

The existing methods for intrinsic symmetry detection on 3D models always need complex measures such as geodesic distances for describing intrinsic geometry and statistical computation for finding non‐rigid transformations to associate symmetrical shapes. They are expensive, may miss symmetries, and cannot guarantee their obtained symmetrical parts in high quality. We observe that only extrinsic symmetries exist between convex shapes, and two intrinsically symmetric shapes can be determined if their belonged convex sub‐shapes are symmetrical to each other correspondingly and connected in a similar topological structure. Thus, we propose to decompose the model into convex parts, and use the similar structures of the skeleton of the model to guide combination of extrinsic symmetries between convex parts for intrinsic symmetry detection. In this way, we give up statistical computation for intrinsic symmetry detection, and avoid complex measures for describing intrinsic geometry. With the similar structures being from small to large gradually, we can quickly detect multi‐scale partial intrinsic symmetries in a bottom up manner. Benefited from the well segmented convex parts, our obtained symmetrical parts are in high quality. Experimental results show that our method can find many more symmetries and runs much faster than the existing methods, even by several orders of magnitude.     

Transferring of speech movements from video to 3D face space

We present a novel method for transferring speech animation recorded in low quality videos to high resolution 3D face models. The basic idea is to synthesize the animated faces by an interpolation based on a small set of 3D key face shapes which span a 3D face space. The 3D key shapes are extracted by an unsupervised learning process in 2D video space to form a set of 2D visemes which are then mapped to the 3D face space. The learning process consists of two main phases: 1) isomap-based nonlinear dimensionality reduction to embed the video speech movements into a low-dimensional manifold and 2) k-means clustering in the low-dimensional space to extract 2D key viseme frames. Our main contribution is that we use the isomap-based learning method to extract intrinsic geometry of the speech video space and thus to make it possible to define the 3D key viseme shapes. To do so, we need only to capture a limited number of 3D key face models by using a general 3D scanner. Moreover, we also develop a skull movement recovery method based on simple anatomical structures to enhance 3D realism in local mouth movements. Experimental results show that our method can achieve realistic 3D animation effects with a small number of 3D key face models