{"title": "Conditional Structure Generation through Graph Variational Generative Adversarial Nets", "book": "Advances in Neural Information Processing Systems", "page_first": 1340, "page_last": 1351, "abstract": "Graph embedding has been intensively studied recently, due to the advance of various neural network models. Theoretical analyses and empirical studies have pushed forward the translation of discrete graph structures into distributed representation vectors, but seldom considered the reverse direction, i.e., generation of graphs from given related context spaces. Particularly, since graphs often become more meaningful when associated with semantic contexts (e.g., social networks of certain communities, gene networks of certain diseases), the ability to infer graph structures according to given semantic conditions could be of great value. While existing graph generative models only consider graph structures without semantic contexts, we formulate the novel problem of conditional structure generation, and propose a novel unified model of graph variational generative adversarial nets (CondGen) to handle the intrinsic challenges of flexible context-structure conditioning and permutation-invariant generation. Extensive experiments on two deliberately created benchmark datasets of real-world context-enriched networks demonstrate the supreme effectiveness and generalizability of CondGen.", "full_text": "Conditional Structure Generation through\n\nGraph Variational Generative Adversarial Nets\n\nCarl Yang\u2217, Peiye Zhuang, Wenhan Shi, Alan Luu, Pan Li\nUniversity of Illinois at Urbana Champaign, Urbana, IL 61801\n\n{jiyang3, peiye, wenhans2, alanluu2, panli2}@illinois.edu\n\nAbstract\n\nGraph embedding has been intensively studied recently, due to the advance of\nvarious neural network models. Theoretical analyses and empirical studies have\npushed forward the translation of discrete graph structures into distributed repre-\nsentation vectors, but seldom considered the reverse direction, i.e., generation of\ngraphs from given related context spaces. Particularly, since graphs often become\nmore meaningful when associated with semantic contexts (e.g., social networks\nof certain communities, gene networks of certain diseases), the ability to infer\ngraph structures according to given semantic conditions could be of great value.\nWhile existing graph generative models only consider graph structures without\nsemantic contexts, we formulate the novel problem of conditional structure genera-\ntion, and propose a novel uni\ufb01ed model of graph variational generative adversarial\nnets (CONDGEN) to handle the intrinsic challenges of \ufb02exible context-structure\nconditioning and permutation-invariant generation. Extensive experiments on two\ndeliberately created benchmark datasets of real-world context-enriched networks\ndemonstrate the supreme effectiveness and generalizability of CONDGEN.\n\n1\n\nIntroduction\n\nGraphs (networks) provide a generic way to model real-world relational data, such as entities in\nknowledge graphs, users in social networks, genes in regulatory networks, etc. It is thus critical\nto study the generation of graph structures, which is fundamental for the understanding of their\nunderlying functional components and creation of meaningful structures with desired properties.\nNowadays, contextual data like attributes and labels are becoming ubiquitous in networks [44], the\nrich semantics of which may well correspond to particular graph structures. This brings up a natural\nbut challenging question: Can we generate graph structures w.r.t. given semantic conditions?\nIn this work, we propose and study the novel problem of conditional structure generation, whose goal\nis to learn and generate graph structures under various semantic conditions indicated by contextual\nattributes or labels in the networks. Figure 1 shows a toy example of biomedical networks, where the\ninteractions of certain genes and proteins may follow related but different patterns for individuals\nwith different diseases (e.g., cancers in different body parts and stages). Due to limited observations,\nnetwork data of some diseases may be more scarce (only one network observed for Case 1) or\ntotally missing (no network observed for Case 2), while those of other closely related diseases are\nmore available (2-3 networks observed for other cases). Since the diseases are semantically related,\ntheir corresponding gene networks may well share certain graph structures. Thus, by ef\ufb01ciently\nexploring the possible correspondence between network contexts and structures, an ideal model\nshould be able to generate more similar graphs for conditions with scarce observed graphs (Task 1),\nand generate meaningful novel graphs for conditions without any observed graphs (Task 2). The\nproblem is important because the generated networks can be valuable in various subsequent studies\n\n\u2217Corresponding author.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Toy example of conditional structure generation: Real-world networks nowadays are\noften associated with correlated semantic attributes/labels. This allows us to explore the possible\ncorrespondence between graph contexts and structures, which can be leveraged to generate structures\nfor graphs with certain semantic contexts that are hardly observed.\n\nsuch as the understanding and prediction of disease development. It is also general, if we consider\nvast other examples such as in social networks, where users in different communities may share\ncertain connection patterns, and in knowledge graphs, where different types of entities may be related\nin particular ways.\nExisting works on network generation cannot \ufb02exibly handle various semantic conditions. Speci\ufb01cally,\nearlier probabilistic graph models can only generate networks with limited pre-assumed properties\nlike random links [10], small diameters [41], power-law distribution [2], etc. Recent works based on\nneural networks can generate graphs with much richer properties learned on given graphs. However,\nthey either only work with single graphs and \ufb01xed sets of nodes [18, 4, 7, 25, 52, 12], or model single\nrepresentative sets of graphs which essentially belong to the same semantic group [47, 36, 16, 24, 46].\nOnly [24] mentions the ability of conditional generation, but the conditions in their setting are direct\ngraph properties such as number of nodes, which is fundamentally different from the semantic\nconditions as we consider in this work. Moreover, none of the existing methods really solve the\nfundamental challenge of graph permutation invariance [26, 42, 15, 48, 30] during their translation\nbetween graph structures and representations, due to the facts that their embedding spaces or generated\ngraphs are essentially not permutation-invariant (Sec. 2), so they tend to generate different graphs\ngiven the same input graphs with permutated node orders (Sec. 4).\nThanks to the surge of deep learning [20, 27], many successful neural network models like skip-gram\n[28] and CNN [17] have been studied for graph representation learning [31, 11, 38, 19, 39]. Among\nthem, graph convolutional neural networks (GCN) [19] has received extensive theoretical analyses and\nempirical studies recently [26, 42, 14, 5, 23], due to its proved ability to encode nodes, (hyper)links\nor whole graphs into a permutation-invariant space. However, how to map the distributed vectors\nback to graphs in a permutation-invariant manner still remains an open problem. Particularly, the\ngraph variational autoencoder (GVAE), as the direct application of GCN for graph generation [18],\nstill only models single networks with \ufb01xed sets of nodes (with \ufb01xed orders), thus cannot handle\n\ufb02exible semantic conditions and permutation invariance.\nIn this work, to address the essential challenges of \ufb02exible context-structure conditioning and\npermutation-invariant graph generation in conditional structure generation, we propose the novel\nmodel of CONDGEN, which is essentially a neural architecture of graph variational generative ad-\nversarial nets. It fully leverages the well developed GCN model by further collapsing the node\nencoding into permutation-invariant graph encoding through variational inference on the conjugate\nlatent distributions, which naturally allows \ufb02exible graph context-structure conditioning. To further\nguarantee permutation-invariant graph decoding/generation, GCN is leveraged again to construct\na graph discriminator before the computation of graph reconstruction loss in the standard encoder-\ndecoder framework. This allows the graph generator to explore graphs of variable sizes and arbitrary\nnode orders, which is critical for the capturing of essential graph structures. Finally, for ef\ufb01cient\nand robust model training, we let the GCNs in graph encoder and discriminator share parameters to\nenforce mapping consistency between the graph context and structure spaces and avoid the encoder\ncollapse.\n\n2\n\n\fTo fully demonstrate the value of conditional structure generation and the power of our proposed\nCONDGEN model, we create two benchmark datasets of real-world context-enriched networks\nand design a series of experiments to evaluate CONDGEN against several state-of-the-art graph\ngenerative models properly adapted to the same setting. Through close comparisons over various\ngraph properties and careful visual inspections, we comprehensively show the supreme effectiveness\nand generalizability of CONDGEN on conditional structure generation.\n\n2 Graph Variational Generative Adversarial Nets\n\n2.1 Problem Formulation\n\nWe focus on the novel problem of conditional structure generation. We are given a set of graphs\nG = {G1, G2, . . . , Gn}, where Gi = {Vi, Ei} corresponds to a particular graph structure described\nby the set of nodes Vi and the set of edges Ei. Since graphs nowadays are often contextualized with\ncertain semantic attributes or labels of interest, we construct a condition vector Ci for each graph\nGi, which describes some particular simple graph contexts of Gi (examples are shown in the data\npreparation in Sec. 4). We leave the exploration of more complex contexts as future work.\nIn this work, we aim to explore and model the possible context-structure correspondence on graphs.\nThat is, by training a model M on a set of graphs with certain conditions (i.e., T = {Gi, Ci}n\ni=1), we\nhope to (1) given a seen condition C \u2208 T , generate more graphs G mimicking the structures of those\nin the training set T , and (2) given an unseen condition C /\u2208 T , generate reasonable novel graphs G\nthat can support similar tasks in T while providing insight into the unobservable world.\nWe summarize the essential challenge of conditional structure generation as two folds in the following.\nRequirement 1. Flexible context-structure conditioning. Both the context space, structure space\nand mapping between the two spaces can be rather complex. Therefore, a model should be able to\neffectively explore the two spaces and their correspondence based on all context-structure pairs in T .\nThis means the model needs to jointly capture arbitrary contexts and generate graphs of arbitrary sizes\nand structures. Moreover, a single model has to be trained on arbitrary numbers of context-structure\npairs upon availability.\nRemark 1. Existing graph generative models only consider graph structures and ignore the rich\ngraph contexts associated for structure generation. Moreover, earlier works only model particular\nfamilies of structures [22, 33], while more recent works mostly consider single graphs with \ufb01xed sizes\n[18, 4, 7, 25, 52, 12]. GraphRNN [47] is the only one we have seen so far that can be trained with\na set of graphs and scale up to graphs with hundreds of nodes, but its GRU design with sequential\nhidden spaces makes it hard to directly apply effective semantic conditioning (as we will discuss\nmore in Sec. 3 and show in Sec. 4).\nRequirement 2. Permutation-invariant graph generation. The structure of a graph G is most\ncommonly represented by an adjacency matrix A, where Aij = 1 means vi and vj are connected and\nAij = 0 otherwise. However, the representation is not unique. In fact, since there are n! possible\npermutations for a graph with n nodes, the number of possible adjacency matrices corresponding\nto the same underlying graph is also exponential. Therefore, a model should be able to ef\ufb01ciently\ncompare the underlying graphs instead of the representations and equalize different representations\nof the same underlying graphs, essentially achieving permutation-invariance [26, 42, 15, 48, 30].\nRemark 2. Existing graph generative models are not permutation-invariant. Particularly, models\nrelying on \ufb01xed sets of nodes are not permutation-invariant, because there exists no canonical\nnode ordering and the models have to be re-trained whenever the ordering of nodes is changed\n[18, 12, 36, 7, 25, 24]. Moreover, models that convert between graphs and other structures like\nnode-edge sequences, trees and random walks are also not permutation-invariant, because there is no\nguarantee of one-to-one mapping between graphs and the selected structures [47, 16, 4, 46, 52].\n\n2.2 Proposed Model\n\nWe propose CONDGEN, which coherently joins the power of GCN, VAE and GAN for conditional\nstructure generation, and satis\ufb01es both requirements above. Figure 2 illustrates the overall architecture\nof CONDGEN. In the following, we introduce the motivations and details of our model design.\n\n3\n\n\fFigure 2: Overall framework of CONDGEN: The upper part is a graph variational autoencoder,\nwhere we collapse the node embeddings into a single graph embedding, so as to enable \ufb02exible\ngraph context-structure conditioning and allow training/generating of graphs with variable sizes. The\nlower part makes up for a graph generative adversarial nets, where we leverage GCN to guarantee\npermutation-invariant graph encoding, generation and comparison for reconstruction. Parameters in\nthe decoder and generator networks as well as those in the two GCN networks in the encoder and\ndiscriminator are shared to further boost ef\ufb01cient and robust model inference.\n\nGiven the two requirements, we get inspiration from recent works on GCN, which is promising in\ncalculating representative and permutation-invariant graph embedding [26, 42]. It is thus natural to\nthink of a permutation-invariant graph encoder-decoder framework by leveraging GCN and enable\n\ufb02exible conditioning through variational inference [37]. In fact, [18] proposed a VAE framework for\ngraph generation soon after the invention of GCN. However, they only consider learning on a single\ngraph G = {V, E} and generating/reconstructing links E on the \ufb01xed set of nodes V , thus failing to\nmeet both requirements for conditional structure generation.\nIn this work, we apply a small but necessary trick to the original GVAE framework in [18], i.e.,\nlatent space conjugation, which effectively converts node-level encoding into permutation-invariant\ngraph-level encoding, and allows learning on arbitrary numbers of graphs and generation of graphs\nwith variable sizes. Particularly, given a graph G = {V, E}, since we consider available node\ncontents as semantic conditions, we regard G as a plain network with the adjacency matrix A and\ngenerate node features X = X(A) as the standard k-dim spectral embedding2 based on A. As\nsuggested by reviewers, we later also experiment with replacing spectral embedding by Gaussian\nrandom vectors, which leads to signi\ufb01cant reduce in runtime and comparable model performance,\nthanks to the representative and permutation-invariant structure encoding of GCN (details in Sec. 4).\nFollowing [18], we introduce the stochastic latent variable Z, which can be inferred from X and A as\ni=1 q(zi|X, A). zi \u2208 Z can be regarded as the node embedding of vi \u2208 V . Different\n\nq(Z|X, A) =(cid:81)n\n\nfrom [18], we use a single distribution \u00afz to model all zi\u2019s by enforcing\n\nq(zi|X, A) \u223c N (\u00afz|\u00af\u00b5, diag(\u00af\u03c32)), where \u00af\u00b5 =\n\n1\nn\n\ng\u00b5(X, A)i, \u00af\u03c32 =\n\n1\nn2\n\nn(cid:88)\n\ni=1\n\nn(cid:88)\n\ni=1\n\ng\u03c3(X, A)2\ni ,\n\n(1)\n\nwhere g(X, A) = \u02dcAReLU( \u02dcAXW0)W1 is a two-layer GCN model. g\u00b5(X, A) and g\u03c3(X, A) com-\npute the matrices of mean and standard deviation vectors, which share the \ufb01rst-layer parameters W0.\ng(X, A)i is the ith row of g(X, A). \u02dcA = D\u2212 1\n2 is the symmetrically normalized adjacency\n\nmatrix of G, where D is its degree matrix with Dii =(cid:80)n\n\n2 AD\u2212 1\n\nj=1 Aij.\n\nThe trick of latent space conjugation leads to the modeling of \u00afz, which essentially is the mean of\nzi over G, and thus can be regarded as the graph embedding of G. While straightforward, the\nintroduction of \u00afz is critical for conditional structure generation, because (1) it allows the model to\ngenerate graphs of variable sizes and be trained on set of graphs; (2) it enables graph-level variational\ninference and \ufb02exible context-structure conditioning; (3) it guarantees permutation-invariant graph\nencoding. We discuss about these three advantages in details in the following.\n\n2https://scikit-learn.org/stable/modules/generated/sklearn.manifold.SpectralEmbedding.html\n\n4\n\n\fFirstly, by individually modeling the embedding zi of each node vi \u2208 V with separate latent\ndistributions, [18] can only generate links among the \ufb01xed set of nodes V , whereas we can gen-\nerate graphs of an arbitrary size m by sampling zi for m times from the shared distribution of\n\u00afz. Secondly, according to [29], a conditional GVAE can be directly constructed by concatenating\n((cid:12)) the condition vector C to both X and \u00afz during training and to zi\u2019s sampled from \u00afz during\ngeneration. Finally, since g(X, A) is permutation-invariant (i.e., \u2200P \u2208 {0, 1}n\u00d7n as a permuta-\ntion matrix, g(P X, P AP T ) = P g(X, A)P T [45]), \u00afz, \u00af\u00b5 and \u00af\u03c3 are also permutation-invariant (i.e.,\ni=1 g(X, A)i). It thus guarantees that \u00afz is\n\n(cid:80)n\ni=1 g(P X, P AP T )i = (cid:80)n\n\ni=1[P g(X, A)P T ]i = (cid:80)n\n\nindistinguishable if A is permutated.\nBesides this difference, after sampling a desirable number of zi\u2019s, to improve the capability of the\ngraph decoder, we append a few layers of fully connected feedforward neural networks f to zi before\ncomputing the logistic sigmoid function for link prediction, i.e.,\n\np(A|Z) =\n\np(Aij|zi, zj), with p(Aij = 1|zi, zj) = \u03c3(f (zi)T f (zj)),\n\n(2)\n\nn(cid:89)\n\nn(cid:89)\n\ni=1\n\nj=1\n\nn\n\nLvae = Lrec + Lprior = Eq(Z|X,A)[log p(A|Z)] \u2212 DKL(q(Z|X, A)||p(Z)),\n\nwhere \u03c3(z) = 1/(1 + e\u2212z). We optimize the model by minimizing the minus variational lower bound\ndivergence towards the Gaussian prior p(Z) = (cid:81)n\n(3)\nwhere Lrec is a link reconstruction loss and Lprior is a prior loss based on the Kullback-Leibler\n(cid:80)n\ni=1 p(zi) = N (\u00afz|0, I)n. The model now con-\nsists of a GCN-based graph encoder E(A) = 1\ni=1 g(X(A), A)i, and an FNN-based graph\ndecoder/generator G(Z) = f (zi)T f (zj).\nWith this modi\ufb01ed GVAE, we can compute permutation-invariant graph encoding and generate\ngraphs of variable sizes under different conditions. However, the graph generation process is still not\npermutation-invariant, because Lrec is computed between the generated adjacency matrix A(cid:48) = G(Z)\nand the original adjacency matrix A, which means A(cid:48) has to follow the same node ordering as A. In\nan ideal case, if A(cid:48) = P AP T , Lrec should be zero. This is not the case for the current model, which\nmisleads the generator/decoder to waste its capacity in capturing the n! node permutations, instead of\nthe underlying graph structures.\nTo deal with this de\ufb01ciency, we again leverage GCN, by devising a permutation-invariant graph\ndiscriminator, which learns to enforce the intrinsic structural similarity between A(cid:48) and A under\narbitrary node ordering. Particularly, we construct a discriminator D of a two-layer GCN followed\nby a two-layer FNN, and jointly train it together with the encoder E and decoder/generator G, by\noptimizing the following GAN loss of a two-player minimax game\n\nLgan = log(D(A)) + log(1 \u2212 D(A(cid:48))), with D(A) = f(cid:48)(g(cid:48)(X(A), A)),\n\n(4)\nwhere X, g(cid:48) and f(cid:48) are spectral embedding, GCN and FNN, respectively, similarly as de\ufb01ned before.\nAfter g(cid:48), the encodings g(cid:48)(A) and g(cid:48)(A(cid:48)) are permutation-invariant (i.e., \u2200A(cid:48) = P AP T , g(cid:48)(A) =\ng(cid:48)(A(cid:48))), and the reconstruction loss Lrec can be simply computed as Lrec = ||g(cid:48)(A) \u2212 g(cid:48)(A(cid:48))||2\n2,\nwhich captures the intrinsic structural difference between A and A(cid:48) regardless of the possibly different\nnode ordering.\nAt this point, we \ufb01nd our model closely related to the recently popular framework of VAEGAN\n[21, 13, 34]. Similarly to their observations, we \ufb01nd it bene\ufb01cial to include two sources of generated\nmatrix A(cid:48), i.e., one from the sampled graph encoding Zs w.r.t. the prior distribution, and another\nfrom the computed graph encoding Zc = E(A), and rede\ufb01ne the GAN loss as\n\nLgan = log(D(A)) + log(1 \u2212 D(G(Zs))) + log(1 \u2212 D(G(Zc))).\n\n(5)\n\nDifferent from VAEGAN, and motivated by the powerful framework of CycleGAN [51], we further\naim to apply additional constraints to the framework to enforce mapping consistency between the\ncontext and structure spaces. Particularly, we \ufb01nd it bene\ufb01cial to share parameters in the two GCN\nmodules g and g(cid:48), which essentially requires that the generated graph A(cid:48) can be brought back to the\nlatent space of graph encoding with contexts Z (cid:12) C by the same encoder g that maps the original\ngraph A to the space of Z (cid:12) C. Besides, in practice, it may also help prevent the encoder from\noccasional collapse due to the overwhelmingly powerful decoder/generator [1], when E keeps yielding\nthe same noise Z for different input A, but G manages to over\ufb01t the training data by generating the\ncorrect A(cid:48) solely based on the condition vector C. In this case, the model degrades into a conditional\nGAN [29], which is harder to train without E functioning as expected.\n\n5\n\n\f2.3 Training Details\nWe jointly train the encoder E, decoder/generator G and discriminator D by optimizing the following\ncombined loss function\n\nLCONDGEN = Lrec + \u03bb1Lprior + \u03bb2Lgan,\n\n(6)\n\nwhere \u03bb1 and \u03bb2 are tunable trade-off hyperparameters. As suggested in [21], it is important not to\nupdate all model parameters w.r.t. the combined loss function. Particularly, we use the following\nparameter updating rules for in each training batch\n\n+\u2190\u2212 \u2212\u2207\u03b8E (Lrec + \u03bb1Lprior), \u03b8G\n\n+\u2190\u2212 \u2212\u2207\u03b8G(Lrec \u2212 \u03bb2Lgan), \u03b8D\n\n+\u2190\u2212 \u2212\u2207\u03b8D \u03bb2Lgan.\n\n\u03b8E\n\n(7)\n\n3 Connections to Existing Works\n\n3.1 Graph Embedding\n\nGraph embedding studies the task of computing distributional representations for graph data, where\nthe major challenge lies in the lack of canonical node ordering and \ufb02exible context structures. While\ntraditional embedding methods often resort to computations in the spectral domain [6, 3, 35], recent\nadvances in neural networks and deep learning have shed new light on ef\ufb01cient approximations to\nthe heavy spectral computations [31, 38, 11, 32, 19, 8, 43]. Among the many recently developed\nneural network based graph embedding algorithms, GCN [19] has been intensively studied and\nshown promising in calculating representative and permutation-invariant graph embedding [23, 45, 5].\nParticularly, we harvest the nice properties of GCN illustrated by the following bound\n\n\u03b1d(G, G(cid:48)) \u2264 d(E(A),E(A(cid:48))) \u2264 \u03b2d(G, G(cid:48)),\n\n(8)\nwhere d(G, G(cid:48)) is the structural difference between two graphs G and G(cid:48), such as edit distance,\nd(E(A),E(A(cid:48))) is the distance between the encodings of G and G(cid:48), such as (cid:96)2 distance, \u03b1 and \u03b2 are\ntwo constants (\u03b1 \u2264 \u03b2). The \ufb01rst inequality guarantees representativeness of the encoding, while\nthe second guarantees permutation invariance. Properly trained GCN is expected to approximate a\ntight bound with \u03b1 (cid:39) \u03b2 [26, 42], which is thus ideal for our tasks of representative and permutation-\ninvariant graph encoding and discrimination.\n\n3.2 Graph Generation\n\nAlthough much work has been done regarding permutation-invariant graph embedding, how to\ngenerate graphs in a permutation-invariant manner is still an open problem. We formulate the\nrequirements of permutation-invariant graph embedding/encoding and generation/decoding as follows,\n\u2200P \u2208 {0, 1}n\u00d7n as a permutation matrix\n\nEncoding: E(P AP T ) = E(A); Decoding: L(A(cid:48), A) = L(P A(cid:48)P T , A).\n\n(9)\n\nMost existing graph generation models like [18, 12, 7, 40, 25, 52] are only generating links among\n\ufb01xed set of nodes, and thus do not satisfy Eq. 9. Similar to our leverage of GCN, GraphVAE [36]\ncomprises of a graph encoder of GCN that satis\ufb01es the \ufb01rst part of Eq. 9 and a decoder outputting a\nsymmetric adjacency matrix. Since the nodes of the output graph may have arbitrary ordering, to\nsatisfy the second part of Eq. 9, an expensive graph matching algorithm (O(n4)) must be employed\nbefore the graph reconstruction loss can be calculated. To avoid explicit matching between generated\nand original graphs, NetGAN [4] is proposed to convert graphs into biased random walks and learn\nthe generation of walks instead of graphs. However, although much more ef\ufb01cient than GraphVAE,\nNetGAN still can only model single graphs with \ufb01xed sizes. To learn with multiple graphs and\ngenerate graphs with variable sizes, GraphRNN [47] is proposed to model graph generation as a\nnode and edge insertion sequence with RNN. However, since RNN models a series of dynamic\nhidden spaces, it is hard to directly apply conditioning by concatenation. Moreover, due to the lack of\none-to-one mapping between graphs and node-edge sequences, both encoding and decoding processes\nof GraphRNN do not satisfy Eq. 9.\n\n6\n\n\f3.3 Deep Generative Models\n\nIn order to enable \ufb02exible conditioning and permutation-invariant generation, our model borrows\nideas from VAEGAN [21, 13, 34], which augments the standard VAE model with a discriminator to\nenforce style similarity and reduce image blurring. Different from VAEGAN, we incorporate a GCN\nbased discriminator to learn a loss function that is both discriminative and permutation-invariant.\nMoreover, a key aspect of learning this loss function involves cycle consistency, a concept \ufb01rst\nintroduced by CycleGAN [51] to learn a translation mapping G between images in a source space\nX to images in a target space Y . Since G is under-constrained due to the absence of aligned X, Y\nexamples, it is paired with an inverse mapping F from the target space Y to the source space X,\nwhere a cycle consistency loss is introduced to enforce similarity between X and F (G(X)). Our\nmodel leverages the idea of cycle consistency by encoding both the original graph A and generated\ngraph A(cid:48) into the same context space Z (cid:12) C where the GAN loss can then be calculated. Particularly,\nwe require E(G(E(A))) to be close to E(A) by sharing the parameters in the two GCN modules (i.e.,\ng in the encoder E and g(cid:48) in the discriminator D) and computing the reconstruction loss Lrec and\nGAN loss Lgan after g(cid:48), thus enforcing cycle consistency between the context and structure spaces.\nFinally, the idea of enabling permutation-invariance in an encoder-decoder system also applies to\nproblems like set generation, where we notice the recent development of models that achieve it by\nbackpropagating through an order-invariant set encoder [49], which can be further improved with\nmore sophisticated set pooling methods [50].\n\n4 Experimental Evaluations\n\nWe create two real-world context-rich network datasets and conduct thorough experiments to demon-\nstrate the effectiveness and generalizability of CONDGEN in conditional structure learning. All code\nand data used in our experiments have been made available on GitHub3.\n\nDatasets. Since we are the \ufb01rst to consider the novel but important problem of conditional structure\ngeneration, there is no existing dataset for evaluation. To this end, we created two benchmark datasets,\ni.e., a set of author citation networks from DBLP4 and a set of gene interaction networks from\nTCGA5.\nFrom DBLP, we create a set of 72 (8 \u00d7 3 \u00d7 3) author networks, each associated with a 10-dim\ncondition vector. The nodes are the \ufb01rst authors of research papers published in 8 conferences, i.e.,\nNIPS and ICML (representing the ML community), KDD and ICDM (DM), SIGIR and CIKM (IR),\nSIGMOD and VLDB (DB). Then each of the 8 groups of authors are further divided into 3 subgroups\nby the number of total publications (1-10, 10-30, 30+), representing the productivity of authors.\nFinally three networks are created for each of the 24 sets of authors, by adding in the citation links\ncreated in different time period (1990-1999, 2000-2009, 2010-2019). Thus, the 10-dim condition\nvector is a concatenation of a 8-dim one-hot vector denoting the conferences, and a 2-dim integral\nvector denoting the level of productivity and link creation time (each with three values 0, 1, 2). The\naverage numbers of nodes and edges in the author networks are 109 and 186, respectively.\nFrom TCGA, we create a set of 54 (6\u00d7 3\u00d7 3) gene networks, each associated with a 8-dim condition\nvector consisting of a 6-dim one-hot encoding of cancer primary sites (brain, liver, lung, ovary, skin,\nand kidney) and a 2-dim integral vector denoting age of diagnosis (30-57, 58-69, 70-90) and stage\napproximated by days-to-days (0-400, 400-800, 800-8000). For each faceted search with a particular\ncombination of primary site, age-at-diagnosis, and days-to-death \ufb01lters, a gene correlation network\nwas created using a gene expression matrix constructed from the \ufb01rst 10 RNA-Seq FPKM \ufb01les.\nFrom each RNA-Seq FPKM matrix M, a transformed matrix N = log10(M + 0.5 \u00d7 min(M )) was\ncreated and then \ufb01ltered for genes with a unique entrez ID and vector representation [9]. Finally, a\ngene correlation network was constructed using pearson correlation with p-value threshold 0.01. The\naverage numbers of nodes and edges in the gene networks are 177 and 1096, respectively.\n\nBaselines. Since no baseline is available for the novel task of conditional structure learning, we\ncarefully adapt three state-of-the-art graph generation methods, i.e., GVAE [18], NetGAN [4] and\n\n3https://github.com/KelestZ/CondGen\n4DBLP source: https://dblp.uni-trier.de/\n5TCGA source: https://www.cancer.gov/tcga\n\n7\n\n\fGraphRNN [47], by concatenating the condition vectors to both the node features of the input graph\nand the output of the last encoding layer following the standard practice in [29]. To allow a single\nGVAE or NetGAN model to be trained on a set of graphs, we \ufb01x the size of input and output graphs\nas the largest size of all networks following [36]. As suggested by reviewers, we also construct a\nvariant of CONDGEN by replacing the spectral embedding with Gaussian random vectors of the\nsame sizes to use as input node features to GCN, denoted as CONDGEN(R) (i.e., random vectors) as\nopposed to CONDGEN(S) (i.e., spectral embeddings).\n\nProtocols. To demonstrate the effectiveness and generalizability of CONDGEN, we evaluate both\ntasks of mimicking similar seen graphs and creating novel unseen graphs. We \ufb01rstly partition all\nnetworks at random by a ratio of 1:1 into training and testing sets. Note that, the testing set includes\ngraphs with both seen and unseen conditions in the training set, so a good model that performs well\non the testing set has to effectively capture the context-structure correspondence among graphs with\nthe seen conditions and generalize to graphs with unseen conditions.\n\nGraphs\n\nDBLP\nSeen\n\nDBLP\nUnseen\n\nTCGA\nSeen\n\nTCGA\nUnseen\n\nMD\nCPL\nGINI\nModels\nLCC\nTC\n3.696\n11.62\n0.3293\nReal\n96.00\n48.54\n2.32\u2217\u2217\n1.390\u2217\n0.1964\u2217\u2217\n20.91\u2217\u2217 21.76\u2217\u2217\nGVAE\n1.641\u2217\u2217 2.77\u2217\u2217\n0.0568\u2217\u2217\n21.15\u2217\u2217 22.46\u2217\u2217\nNetGAN\n1.628\u2217\u2217 7.06\u2217\u2217\n0.2446\u2217\u2217\n69.32\u2217\u2217\n6.88\u2217\nGraphRNN\n1.201\u2217\n0.1232\u2217\n7.70\u2217\nCONDGEN(R) 6.70\u2217\n1.33\n0.963\nCONDGEN(S) 6.00\n1.48\n0.0959\n11.32\n4.982\n14.29\n0.3223\nReal\n102.50\n58.21\n1.521\u2217\u2217 3.53\u2217\n0.2479\u2217\u2217\n17.40\u2217\u2217 17.02\u2217\u2217\nGVAE\n1.494\u2217\u2217 3.71\u2217\u2217\n29.57\u2217\u2217 39.85\u2217\u2217\n0.0812\nNetGAN\n1.305\u2217\n6.43\u2217\u2217\n0.1447\u2217\u2217\n73.21\u2217\u2217\nGraphRNN\n6.43\n1.445\u2217\u2217 1.92\n0.1418\u2217\u2217\nCONDGEN(R) 9.25\u2217\n10.50\n1.162\n1.92\nCONDGEN(S) 6.33\n10.17\n0.0861\nReal\n177.34\n38.27\n4.171\n0.4192\n8913.20\n54.82\u2217\u2217 2396.94\u2217\n14.10\u2217\u2217 0.2035\u2217\u2217\nGVAE\n1.538\n32.02\u2217\u2217 3614.61\u2217\u2217 1.702\u2217\u2217 17.61\u2217\u2217 0.1289\u2217\nNetGAN\n16.20\u2217\n2881.68\u2217\u2217 1.899\u2217\u2217 18.78\u2217\u2217 0.2726\u2217\u2217\nGraphRNN\nCONDGEN(R) 34.42\u2217\u2217 2594.16\u2217\u2217 1.542\n0.1509\u2217\u2217\n9.50\n8.32\n1.524\n0.1093\nCONDGEN(S) 23.72\n34.34\n177.91\n4.143\n0.4154\n13.03\u2217\u2217 0.1497\u2217\u2217\n37.18\u2217\u2217 2768.55\u2217\u2217 1.324\u2217\n18.45\u2217\u2217 0.1277\u2217\u2217\n31.36\u2217\u2217 3557.91\u2217\u2217 1.645\u2217\n15.73\u2217\u2217 2605.73\u2217\u2217 1.859\u2217\u2217 13.55\u2217\u2217 0.2647\u2217\u2217\n0.1413\u2217\u2217\n10.86\u2217\n3083.81\u2217\u2217 1.362\u2217\n2058.95\n8.68\n0.1003\n1.522\n\nReal\nGVAE\nNetGAN\nGraphRNN\nCONDGEN(R) 27.77\u2217\nCONDGEN(S) 23.97\n\n2076.05\n8053.18\n\nTable 1: Performance evaluation over compared algorithms regarding several important graph\nstatistical properties. The Real rows include the values of real graphs, while the rest are the absolute\nvalues of differences between graphs generated by each algorithm and the real graphs. Therefore,\nsmaller values indicate higher similarities to the real graphs, thus better overall performance. We\nconduct paired t-test between each baseline and CONDGEN(S), scores with \u2217 and \u2217\u2217 passed the\nsigni\ufb01cance tests with p = 0.05 and p = 0.01, respectively.\n\nGraphs GVAE NetGAN GraphRNN CONDGEN(R) CONDGEN(S)\nDBLP\nTCGA\n\n299.5\n192.4\n\n31.5\n27.6\n\n12.8\n10.9\n\n398.6\n414.0\n\n72.3\n52.1\n\nTable 2: Runtimes of training all compared algorithms on the two sets of networks (minutes).\n\nPerformances. Following existing works on generative models [4, 47, 36], we evaluate the generated\ngraphs through visual inspection and graph property comparison6. Our model can \ufb02exibly generate\n\n6Statistics we use include LCC (size of largest connected component), TC (triangle count), CPL (characteristic\n\npath length), MD (maximum node degree) and GINI (gini index), measuring different properties of graphs.\n\n8\n\n\fgraphs with arbitrary numbers of nodes and edges. For fair and clear comparison, when generating\neach graph, we set the maximum number of nodes and edges to the same as the real graph for all\ncompared algorithms. As shown in Table 1, the suite of statistics we use measure graphs from different\nperspectives, and different algorithms often excel at particular ones. Our proposed CONDGEN models\nconstantly rank top with very few exceptions on all measures over both datasets. The advantage of\nCONDGENon generating graphs with seen conditions in the training set demonstrates its utility in\ngenerating more similar graphs for conditions where observations might be sparse, while the edge on\nunseen conditions indicates its generalizability to semantically relevant conditions where observations\nare completely missing. The CONDGEN(R) model variant has quite competitive performance with\nCONDGEN(S), which can be explained by the representative and permutation-invariant structure\nencoding power of GCN. Due to space limit, we put detailed parameter settings, qualitative visual\ninspections and in-depth model analyses into the appendix in the supplemental materials.\n\nRuntimes. Similar to most neural network models, it is meaningless to compute the exact complexity\nof CONDGEN, because the actual runtimes mostly depend on the number of training iterations until\nconvergence. To this end, we record the average runtimes for the training of all compared algorithms\nuntil convergence on the two sets of networks and present in Table 2. As we can clearly observe,\nstate-of-the-art graph generation algorithms like GraphRNN and NetGAN are rather slow, due to\nthe heavy model of RNN and large number of sampled walks, respectively, while CONDGEN and\nits base model GVAE are much faster. Since CONDGEN and GVAE are basically a simple GCN\nmodel encapsulated in a VAEGAN and VAE framework, respectively, we also \ufb01nd that the memory\nconsumptions of CONDGEN and GVAE are orders of magnitudes lower than GraphRNN and NetGAN.\nAmong the two CONDGEN variants, CONDGEN(S) takes about double runtime as CONDGEN(R),\ndue to the computation of spectral embeddings. While the overhead is not signi\ufb01cant, it can get\nmore concerning as the networks become larger, due to the essential O(n3) complexity of spectral\nembedding. However, since CONDGEN(R) has quite competitive performance with CONDGEN(S),\none can use it as a substitute of CONDGEN(S) when ef\ufb01ciency is more of a concern.\n\n5 Conclusion\n\nTo the best of our knowledge, this is the \ufb01rst research effort towards the novel but important problem\nof conditional structure generation. To address the two unique challenges of \ufb02exible context-structure\nconditioning and permutation-invariant structure generation, we design CONDGEN by coherently\njoining the power of GCN, VAE and GAN networks. We created two real-world datasets including a\nset of author networks and a set of gene networks associated with semantic conditions, and thoroughly\ndemonstrated the effectiveness and generalizability of CONDGEN in comparison with several state-of-\nthe-art graph generative models adapted to the conditional structure generation setting. We hope our\nresults will inspire following-up research on conditional generative models for graph data, as well as\nfuture works on its application to various domains where the generation of semantically meaningful\nnetworks can be leveraged to support downstream data analysis and knowledge discovery.\n\nAcknowledgements\n\nResearch was sponsored in part by U.S. Army Research Lab. under Cooperative Agreement No.\nW911NF-09-2-0053 (NSCTA), DARPA under Agreements No. W911NF-17-C-0099 and FA8750-\n19-2-1004, National Science Foundation IIS 16-18481, IIS 17-04532, and IIS-17-41317, DTRA HD-\nTRA11810026, and grant 1U54GM114838 awarded by NIGMS through funds provided by the trans-\nNIH Big Data to Knowledge (BD2K) initiative (www.bd2k.nih.gov). The results shown in this work\nare or part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga.\n\nReferences\n[1] Yu Bai, Tengyu Ma, and Andrej Risteski. Approximability of discriminators implies diversity\n\nin gans. In ICLR, 2019.\n\n[2] Albert-L\u00e1szl\u00f3 Barab\u00e1si and R\u00e9ka Albert. Emergence of scaling in random networks. science,\n\n286(5439):509\u2013512, 1999.\n\n9\n\n\f[3] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data\n\nrepresentation. Neural computation, 15(6):1373\u20131396, 2003.\n\n[4] Aleksandar Bojchevski, Oleksandr Shchur, Daniel Z\u00fcgner, and Stephan G\u00fcnnemann. Netgan:\n\nGenerating graphs via random walks. In ICML, 2018.\n\n[5] Jie Chen, Tengfei Ma, and Cao Xiao. Fastgcn: fast learning with graph convolutional networks\n\nvia importance sampling. In ICLR, 2018.\n\n[6] Fan RK Chung. Spectral graph theory. Number 92. American Mathematical Soc., 1997.\n\n[7] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular\n\ngraphs. arXiv preprint arXiv:1805.11973, 2018.\n\n[8] Micha\u00ebl Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks\n\non graphs with fast localized spectral \ufb01ltering. In NIPS, pages 3844\u20133852, 2016.\n\n[9] Jingcheng Du, Peilin Jia, Yulin Dai, Cui Tao, Zhongming Zhao, and Degui Zhi. Gene2vec:\ndistributed representation of genes based on co-expression. BMC Genomics, 20(Suppl 1):82,\n2019.\n\n[10] P\u00e1l Erd\u02ddos and Alfr\u00e9d R\u00e9nyi. On the evolution of random graphs. Publ. Math. Inst. Hungar.\n\nAcad. Sci, 5:17\u201361, 1960.\n\n[11] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD,\n\npages 855\u2013864, 2016.\n\n[12] Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of\n\ngraphs. arXiv preprint arXiv:1803.10459, 2017.\n\n[13] Xiaodong Gu, Kyunghyun Cho, Jungwoo Ha, and Sunghun Kim. Dialogwae: Multimodal\n\nresponse generation with conditional wasserstein auto-encoder. In ICLR, 2019.\n\n[14] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large\n\ngraphs. In NIPS, pages 1024\u20131034, 2017.\n\n[15] Jason Hartford, Devon R Graham, Kevin Leyton-Brown, and Siamak Ravanbakhsh. Deep\n\nmodels of interactions across sets. PMLR, 2018.\n\n[16] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for\n\nmolecular graph generation. In ICML, 2018.\n\n[17] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and\nLi Fei-Fei. Large-scale video classi\ufb01cation with convolutional neural networks. In CVPR, pages\n1725\u20131732, 2014.\n\n[18] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NIPS Workshop on\n\nBayesian Deep Learning, 2016.\n\n[19] Thomas N Kipf and Max Welling. Semi-supervised classi\ufb01cation with graph convolutional\n\nnetworks. In ICLR, 2017.\n\n[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classi\ufb01cation with deep\n\nconvolutional neural networks. In NIPS, pages 1097\u20131105, 2012.\n\n[21] Anders Boesen Lindbo Larsen, S\u00f8ren Kaae S\u00f8nderby, Hugo Larochelle, and Ole Winther.\n\nAutoencoding beyond pixels using a learned similarity metric. In ICML, 2016.\n\n[22] Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahra-\nmani. Kronecker graphs: An approach to modeling networks. JMLR, 11(Feb):985\u20131042,\n2010.\n\n[23] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks\n\nfor semi-supervised learning. In AAAI, 2018.\n\n10\n\n\f[24] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep\n\ngenerative models of graphs. In ICML, 2018.\n\n[25] Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via\n\nregularizing variational autoencoders. In NIPS, pages 7113\u20137124, 2018.\n\n[26] Haggai Maron, Heli Ben-Hamu, Nadav Sharmir, and Lipman Yaron. Invariant and equivariant\n\ngraph networks. In ICLR, 2019.\n\n[27] Tom\u00e1\u0161 Mikolov, Martin Kara\ufb01\u00e1t, Luk\u00e1\u0161 Burget, Jan \u02c7Cernock`y, and Sanjeev Khudanpur. Recur-\n\nrent neural network based language model. In INTERSPEECH, 2010.\n\n[28] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep-\nresentations of words and phrases and their compositionality. In NIPS, pages 3111\u20133119,\n2013.\n\n[29] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint\n\narXiv:1411.1784, 2014.\n\n[30] Ryan L Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy\npooling: Learning deep permutation-invariant functions for variable-size inputs. ICLR, 2019.\n\n[31] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social\n\nrepresentations. In KDD, pages 701\u2013710, 2014.\n\n[32] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding\nas matrix factorization: Unifying deepwalk, line, pte, and node2vec. In WSDM, pages 459\u2013467,\n2018.\n\n[33] Garry Robins, Pip Pattison, Yuval Kalish, and Dean Lusher. An introduction to exponential\n\nrandom graph (p*) models for social networks. Social networks, 29(2):173\u2013191, 2007.\n\n[34] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Vari-\narXiv preprint\n\national approaches for auto-encoding generative adversarial networks.\narXiv:1706.04987, 2017.\n\n[35] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear\n\nembedding. science, 290(5500):2323\u20132326, 2000.\n\n[36] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs\n\nusing variational autoencoders. In ICANN, pages 412\u2013422, 2018.\n\n[37] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using\n\ndeep conditional generative models. In NIPS, pages 3483\u20133491, 2015.\n\n[38] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-\n\nscale information network embedding. In WWW, pages 1067\u20131077, 2015.\n\n[39] Petar Veli\u02c7ckovi\u00b4c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua\n\nBengio. Graph attention networks. In ICLR, 2018.\n\n[40] Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, Fuzheng Zhang, Xing Xie,\nand Minyi Guo. Graphgan: Graph representation learning with generative adversarial nets. In\nAAAI, 2018.\n\n[41] Duncan J Watts and Steven H Strogatz. Collective dynamics of \u2018small-world\u2019 networks. nature,\n\n393(6684):440, 1998.\n\n[42] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural\n\nnetworks? In ICLR, 2019.\n\n[43] Carl Yang, Yichen Feng, Pan Li, Yu Shi, and Jiawei Han. Meta-graph based hin spectral\n\nembedding: Methods, analyses, and insights. In ICDM, pages 657\u2013666, 2018.\n\n11\n\n\f[44] Carl Yang, Dai Teng, Siyang Liu, Sayantani Basu, Jieyu Zhang, Jiaming Shen, Chao Zhang,\nJ Shang, L Kaplan, T Harratty, et al. Cubenet: Multi-facet hierarchical heterogeneous network\nconstruction, analysis, and mining. In KDD, 2019.\n\n[45] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec.\nHierarchical graph representation learning with differentiable pooling. In NIPS, pages 4805\u2013\n4815, 2018.\n\n[46] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional\npolicy network for goal-directed molecular graph generation. In NIPS, pages 6410\u20136421, 2018.\n\n[47] Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. Graphrnn:\n\nGenerating realistic graphs with deep auto-regressive models. In ICML, 2018.\n\n[48] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning\n\narchitecture for graph classi\ufb01cation. In AAAI, 2018.\n\n[49] Yan Zhang, Jonathon Hare, and Adam Pr ugel-Bennett. Deep set prediction networks. In\n\nNeurIPS, 2019.\n\n[50] Yan Zhang, Jonathon Hare, and Adam Pr ugel-Bennett. Fspool: Learning set representations\n\nwith featurewise sort pooling. In ICLR, 2020.\n\n[51] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image\n\ntranslation using cycle-consistent adversarial networks. In ICCV, pages 2223\u20132232, 2017.\n\n[52] Dongmian Zou and Gilad Lerman. Encoding robust representation for graph generation. arXiv\n\npreprint arXiv:1809.10851, 2018.\n\n12\n\n\f", "award": [], "sourceid": 772, "authors": [{"given_name": "Carl", "family_name": "Yang", "institution": "University of Illinois, Urbana Champaign"}, {"given_name": "Peiye", "family_name": "Zhuang", "institution": "UIUC"}, {"given_name": "Wenhan", "family_name": "Shi", "institution": "UIUC"}, {"given_name": "Alan", "family_name": "Luu", "institution": "UIUC"}, {"given_name": "Pan", "family_name": "Li", "institution": "Stanford"}]}