{"title": "Program Synthesis and Semantic Parsing with Learned Code Idioms", "book": "Advances in Neural Information Processing Systems", "page_first": 10825, "page_last": 10835, "abstract": "Program synthesis of general-purpose source code from natural language specifications is challenging due to the need to reason about high-level patterns in the target program and low-level implementation details at the same time. In this work, we present Patois, a system that allows a neural program synthesizer to explicitly interleave high-level and low-level reasoning at every generation step. It accomplishes this by automatically mining common code idioms from a given corpus, incorporating them into the underlying language for neural synthesis, and training a tree-based neural synthesizer to use these idioms during code generation. We evaluate Patois on two complex semantic parsing datasets and show that using learned code idioms improves the synthesizer's accuracy.", "full_text": "Program Synthesis and Semantic Parsing\n\nwith Learned Code Idioms\n\nRichard Shin\u2217\nUC Berkeley\n\nMiltiadis Allamanis, Marc Brockschmidt & Oleksandr Polozov\n\nMicrosoft Research\n\nricshin@berkeley.edu\n\n{miallama,mabrocks,polozov}@microsoft.com\n\nAbstract\n\nProgram synthesis of general-purpose source code from natural language speci\ufb01-\ncations is challenging due to the need to reason about high-level patterns in the\ntarget program and low-level implementation details at the same time.\nIn this\nwork, we present PATOIS, a system that allows a neural program synthesizer to\nexplicitly interleave high-level and low-level reasoning at every generation step.\nIt accomplishes this by automatically mining common code idioms from a given\ncorpus, incorporating them into the underlying language for neural synthesis, and\ntraining a tree-based neural synthesizer to use these idioms during code genera-\ntion. We evaluate PATOIS on two complex semantic parsing datasets and show\nthat using learned code idioms improves the synthesizer\u2019s accuracy.\n\n1\n\nIntroduction\n\nProgram synthesis is a task of translating an incomplete speci\ufb01cation (e.g. natural language, input-\noutput examples, or a combination of the two) into the most likely program that satis\ufb01es this speci\ufb01-\ncation in a given language [15]. In the last decade, it has advanced dramatically thanks to the novel\nneural and neuro-symbolic techniques [5, 10, 19], \ufb01rst mass-market applications [28], and massive\ndatasets [9, 39, 41]. Table 1 shows a few examples of typical tasks of program synthesis from natural\nlanguage. Most of the successful applications apply program synthesis to manually crafted domain-\nspeci\ufb01c languages (DSLs) such as FlashFill and Karel, or to subsets of general-purpose functional\nlanguages such as SQL and Lisp. However, scaling program synthesis to real-life programs in a\ngeneral-purpose language with complex control \ufb02ow remains an open challenge.\n\nWe conjecture that one of the main current challenges of synthesizing a program is insuf\ufb01cient\nseparation between high-level and low-level reasoning. In a typical program generation process,\nbe it a neural model or a symbolic search, the program is generated in terms of its syntax tokens,\nwhich represent low-level implementation details of the latent high-level patterns in the program. In\ncontrast, humans switch between high-level reasoning (\u201ca binary search over an array\u201d) and low-\nlevel implementation (\u201cwhile l < r: m = (l+r)/2 . . . \u201d) repeatedly when writing a single function.\nReasoning over multiple abstraction levels at once complicates the generation task for a model.\n\nThis conjecture is supported by two key observations. First, recent work [12, 25] has achieved great\nresults by splitting the synthesis process into sketch generation and sketch completion. The \ufb01rst stage\ngenerates a high-level sketch of the target program, and the second stage \ufb01lls in missing details in\nthe sketch. Such separation improves the accuracy of synthesis as compared to an equivalent end-\nto-end generation. However, it allows only one stage of high-level reasoning at the root level of\nthe program, whereas (a) real-life programs involve common patterns at all syntactic levels, and (b)\nprogrammers often interleave high-level and low-level reasoning during implementation.\n\n\u2217Work done partly during an internship at Microsoft Research.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fTable 1: Representative program synthesis tasks from real-world semantic parsing datasets.\n\nDataset\n\nNatural Language Speci\ufb01cation\n\nProgram\n\nHearthstone\n[24]\n\nMana Wyrn (1, 3, 1, Minion, Mage, Common)\nWhenever you cast a spell, gain +1 Attack.\n\n# . . .\ndef create_minion(self, player):\n\nreturn Minion(1, 3, effects=[Effect(\n\nSpellCast(), ActionTag(Give(\nChangeAttack(1)), SelfSelector()))])\n\nSpider\n[41]\n\nFor each stadium, how many concerts are there?\n\nSchema:\nstadium = {stadium_id, name, ...}, ...\n\nSELECT T2.name, COUNT(*)\nFROM concert AS T1 JOIN stadium AS T2\n\nON T1.stadium_id = T2.stadium_id\n\nGROUP BY T1.stadium_id\n\nSecond, many successful applications of inductive program synthesis such as FlashFill [14] rely on\na manually designed DSL to make the underlying search process scalable. Such DSLs include high-\nlevel operators that implement common subroutines in a given domain. Thus, they (i) compress\nthe search space, ensuring that every syntactically valid DSL program expresses some useful task,\nand (ii) enable logical reasoning over the domain-speci\ufb01c operator semantics, making the search\nef\ufb01cient. However, DSL design is laborious and requires domain expertise. Recently, Ellis et al.\n[13] showed that such DSLs are learnable in the classic domains of inductive program synthesis; in\nthis work, we target general-purpose code generation, where DSL design is dif\ufb01cult even for experts.\n\nIn this work, we present a system, called PATOIS, that equips a program synthesizer with automat-\nically learned high-level code idioms (i.e. common program fragments) and trains it to use these\nidioms in program generation. While syntactic by de\ufb01nition, code idioms often represent useful\nsemantic concepts. Moreover, they compress and abstract the programs by explicitly representing\ncommon patterns with unique tokens, thus simplifying generative process for the synthesis model.\n\nPATOIS has three main components, illustrated in Figure 1. First, it employs nonparameteric\nBayesian inference to mine the code idioms that frequently occur in a given corpus. Second, it\nmarks the occurrences of these idioms in the training dataset as new named operators in an extended\ngrammar. Finally, it trains a neural generative model to optionally emit these named idioms instead\nof the original code fragments, which allows it to learn idiom usage conditioned on a task speci\ufb01-\ncation. During generation, the model has the ability to emit entire idioms in a single step instead\nof multiple steps of program tree nodes comprising the idioms\u2019 de\ufb01nitions. As a result, PATOIS\ninterleaves high-level idioms with low-level tokens at all levels of program synthesis, generalizing\nbeyond \ufb01xed top-level sketch generation.\n\nWe evaluate PATOIS on two challenging semantic parsing datasets: Hearthstone [24], a dataset of\nsmall domain-speci\ufb01c Python programs, and Spider [41], a large dataset of SQL queries over various\ndatabases. We \ufb01nd that equipping the synthesizer with learned idioms improves its accuracy in\ngenerating programs that satisfy the task description.\n\n2 Background\n\nProgram Synthesis We consider the following formulation of the program synthesis problem.\nAssume an underlying programming language L of programs. Each program P \u2208 L can be rep-\nresented either as a sequence y1 \u00b7 \u00b7 \u00b7 y|P | of its tokens, or, equivalently, as an abstract syntax tree\n(AST) T parsed according to the context-free grammar (CFG) G of the language L. The goal of a\nprogram synthesis model f : \u03c6 7\u2192 P is to generate a program P that maximizes the conditional\nprobability Pr (P | \u03c6) i.e. the most likely program given the speci\ufb01cation. We also assume a train-\ning set D = {\u27e8\u03c6j, Pj\u27e9}|D|\nj=1, sampled from an unknown true distribution D, from which we wish to\nestimate the conditional probability Pr (P | \u03c6).\n\nIn this work, we consider general-purpose programming languages L with a known context-free\ngrammar G such as Python and SQL. Each speci\ufb01cation \u03c6 is represented as a natural language\ntask description, i.e. a sequence of words X = x1 \u00b7 \u00b7 \u00b7 x|X| (although the PATOIS synthesizer can be\nconditioned on any other type of incomplete spec). In principle, we do not impose any restrictions\non the generative model f apart from it being able to emit syntactically valid programs. However,\nas we detail in Section 4, the PATOIS framework is most easily implemented on top of structural\ngenerative models such as sequence-to-tree models [38] and graph neural networks [7, 21].\n\n2\n\n\ft\nn\ni\n\n:\n1\n\u2113\n\nt\ns\ni\nl\n\n:\n0\n\u2113\n\nx\ne\nd\nn\n\nI\n\nt\np\ni\nr\nc\ns\nb\nu\nS\n\ne\nr\na\np\nm\no\nC\n\nf\nI\n\n2\n\nm \u2113\nu\nN\n\nq\nE\n\n3\n\u2113\n\ni\n\nn\ng\ns\ns\nA\n\n*\nt\n\nm\nt\ns\n\n3\n\u2113\n\nd\nd\nA\n\nm 1\nu\nN\n\np\nO\nn\nB\n\ni\n\n.\n\n.\n\n.\n\nFigure 1: Top: An overview of PATOIS. A miner 1\u20dd extracts common idioms from the programs\nin a given dataset. All the idiom occurrences in the dataset programs are 2\u20dd marked as optional\nalternative grammar operators. The dataset with marked occurrences is used to 3\u20dd train a neural\ngenerative model. At inference time, the model 4\u20dd generates programs with named idioms, which\nare inlined before program execution. Note that idioms may have named subexpressions, may repeat,\nand may occur at any program level. For clarity, we typeset idioms using function-like syntax\nIj(\u21131, . . . , \u2113k) in this paper, although they are actually represented as AST fragments with no syntax.\nBottom: AST fragment representation of the idiom I2 in Python. Here sans-serif nodes are \ufb01xed\nnon-terminals, monospaced nodes are \ufb01xed terminals, and boxed nodes are named arguments.\n\nCode Idioms Following Allamanis and Sutton [2], we de\ufb01ne code idioms as fragments I of valid\nASTs T in the CFG G, i.e. trees of nonterminals and terminals from G that may occur as subtrees\nof valid parse trees from G. The grammar G extended with a set of idiom fragments forms a tree\nsubstitution grammar (TSG). We also associate a non-unique label \u2113 with each nonterminal leaf\nin every idiom, and require that every instantiation of an idiom I must have its identically-labeled\nnonterminals instantiated to identical subtrees. This enables the role of idioms as subroutines, where\nlabels act as \u201cnamed arguments\u201d in the \u201cbody\u201d of an idiom. See Figure 1 for an example.\n\n3 Mining Code Idioms\n\nThe \ufb01rst step of PATOIS is obtaining a set of frequent and useful AST fragments as code idioms.\nThe trade-off between frequency and usefulness is crucial: it is trivial to mine commonly occurring\nshort patterns, but they are often meaningless [1]. Instead, we employ and extend the methodology\nof Allamanis et al. [3] and frame idiom mining as a nonparameteric Bayesian problem.\n\nWe represent idiom mining as inference over probabilistic tree substitution grammars (pTSG). A\npTSG is a probabilistic context-free grammar extended with production rules that expand to a whole\nAST fragment instead of a single level of symbols [8, 29]. The grammar G of our original language L\ninduces a pTSG G0 with no fragment rules and with choice probabilities estimated from the corpus D.\nTo construct a pTSG corresponding to the extension of L with common tree fragments representing\nidioms, we de\ufb01ne a distribution G over pTSGs as follows.\n\nWe \ufb01rst choose a Pitman-Yor process [36] as a prior distribution G0 over pTSGs. It is a nonpa-\nrameteric process that has proven to be effective for mining code idioms in prior work thanks to its\nmodeling of production choices as a Zip\ufb01an distribution (in other words, it implements the desired\n\u201crich get richer\u201d effect, which encourages a smaller number of larger and more common idioms).\n\n3\n\n\fFormally, it is a \u201cstick-breaking\u201d process [31] that\nde\ufb01nes G0 as a distribution for each set of id-\n\nioms eIN rooted at a nonterminal symbol N as\nPr(I \u2208 eIN ) def=\nk\u22121\u220f\n\n\u03c0k \u03b4 (I = Ik) ,\n\ndef= uk\n\n\u221e\u2211\n\nk=0\n\nIk \u223c G0\n\n\u03c0k\n\n(1 \u2212 uj), uk \u223c Beta (1 \u2212 d, \u03b1 + kd)\n\nj=1\n\nwhere \u03b4(\u00b7) is the delta function, and \u03b1, d are hyper-\nparameters. See Allamanis et al. [3] for details.\n\nFigure 2: MCMC sampling for an AST (\ufb01g-\nure from [2]). Dots show the inferred nodes\nwhere the AST is split into fragments.\n\nPATOIS uses G0 to compute a posterior distribu-\ntion G1 = Pr (G1 | T1, . . . , TN ) using Bayes\u2019 rule,\nwhere T1, . . . , TN are concrete AST fragments in the training set D. As this calculation is compu-\ntationally intractable, we approximate it using type-based MCMC [23]. At each iteration t of the\nMCMC process, PATOIS generates a pTSG Gt whose distribution approaches G1 as t \u2192 \u221e. It\nworks by sampling splitting points for each AST T in the corpus D, which by construction de\ufb01ne a\nset of fragments constituting Gt (see Figure 2). The split probabilities of this Gibbs sampling are set\nin a way that incentivizes merging adjacent tree fragments that often cooccur in D. The \ufb01nal idioms\nare then extracted from the pTSG obtained at the last MCMC iteration.\n\nWhile the Pitman-Yor process helps avoid over\ufb01tting the idioms to D, not all sampled idioms are\nuseful for synthesis. Thus we rank and \ufb01lter the idioms before using them in the training. In this\nwork, we reuse two ranking functions de\ufb01ned by Allamanis et al. [3]:\n\nScoreCov (I) def= coverage = count(T \u2208 D | I \u2208 T )\n\nScoreCXE (I) def= coverage \u00b7 cross-entropy gain =\n\ncount(T \u2208 D | I \u2208 T )\n\n|D|\n\n\u00b7\n\n1\n|I|\n\nlog\n\nPrG1 (I)\nPrG0 (I)\n\nand also \ufb01lter out any terminal idioms (i.e. those that do not contain any named arguments \u2113).\n\nWe conclude with a brief analysis of computational complexity of idiom mining. Every iteration of\nthe MCMC sampling traverses the entire dataset D once to sample the random variables that de\ufb01ne\nthe splitting points in each AST. When run for M iterations, the complexity of idiom mining is\n\nO(M \u00b7\u2211T \u2208D |T |). Idiom ranking adds an additional step with complexity O(|eI| log |eI|) where eI\n\nis the set of idioms obtained at the last iteration. In our experiments (detailed in Section 5) we set\nM = 10, and the entire idiom mining takes less than 10 minutes on a dataset of |D| \u2248 10,000 ASTs.\n\n4 Using Idioms in Program Synthesis\n\nGiven a set of common idioms eI = {I1, . . . , IN } mined by PATOIS, we now aim to learn a syn-\n\nthesis model f that emits whole idioms Ij as atomic actions instead of individual AST nodes that\ncomprise Ij . Achieving this involves two key challenges.\n\nFirst, since idioms are represented as AST fragments without concrete syntax, PATOIS works best\nwhen the synthesis model f is structural, i.e. it generates the program AST instead of its syntax.\nPrior work [7, 38, 40] also showed that tree- and graph-based code generation models outperform\nsequence-to-sequence models, and thus we adopt a similar architecture in this work.\n\nSecond, exposing the model f to idiom usage patterns is not obvious. One approach could be to\nextend the grammar with new named operators opI(\u21131, . . . , \u2113k) for each idiom I, replace every oc-\ncurrence of I with opI in the data, and train the synthesizer on the rewritten dataset. However, this\nwould not allow f to learn from the idiom de\ufb01nitions (bodies). In addition, idiom occurrences often\noverlap, and any deterministic rewriting strategy would arbitrarily discard some occurrences from\nthe corpus, thus limiting the model\u2019s exposure to idiom usage. In our experiments, we found that\ngreedy rewriting discarded as many as 75% potential idiom occurrences from the dataset. There-\nfore, a successful training strategy must preserve all occurrences and instead let the model learn a\nrewriting strategy that optimizes end-to-end synthesis accuracy.\n\n4\n\n\fTo this end, we present a novel training setup for code generation that encourages the model to\nchoose the most useful subset of idioms and the best representation of each program in terms of the\n\nidioms. It works by (a) marking occurrences of the idioms eI in the training set D, (b) at training\n\ntime, encouraging the model to emit either the whole idiom or its body for every potential idiom oc-\ncurrence in the AST, and (c) at inference time, replacing the model\u2019s state after emitting an idiom I\nwith the state the model would have if it had emitted I\u2019s body step by step.\n\n4.1 Model Architecture\n\nThe synthesis model f of PATOIS combines a spec encoder fenc and an AST decoder fdec, fol-\nlowing the formulation of Yin and Neubig [38]. The encoder fenc embeds the NL speci\ufb01cation\nX = x1 \u00b7 \u00b7 \u00b7 xn into word representations \u02c6X = \u02c6x1 \u00b7 \u00b7 \u00b7 \u02c6xn. The decoder fdec uses an LSTM to model\nthe sequential generation of the AST in the depth-\ufb01rst order, wherein each timestep t corresponds to\nan action at \u2014 either (a) expanding a production from the grammar, (b) expanding an idiom, or (c)\ngenerating a terminal token. Thus, the probability of generating an AST T given \u02c6X is\n\nPr(T | \u02c6X) = \u220ft\n\nPr(at | Tt, \u02c6X)\n\n(1)\n\nwhere at is the action taken at timestep t, and Tt is the partial AST generated before t. The proba-\nbility Pr(at | Tt, \u02c6X) is computed from the decoder\u2019s hidden state ht\u22121 depending on at.\n\nProduction Actions For actions at = APPLYRULE[R] corresponding to expanding production\nrules R \u2208 G from the original CFG G, we compute the probability Pr(at | Tt, \u02c6X) by encoding the\ncurrent partial AST structure similarly to Yin and Neubig [38]. Speci\ufb01cally, we compute the new\nhidden state as ht = fLSTM ([at\u22121 \u2225 ct \u2225 hpt \u2225 apt \u2225 nft ], ht\u22121) where at\u22121 is the embedding\nof the previous action, ct is the result of soft attention applied to the spec embeddings \u02c6X as per\nBahdanau et al. [4], pt is the timestep corresponding to expanding the parent AST node of the\ncurrent node, and nft is the embedding of the current node type. The hidden state ht is then used\nto compute probabilities of the syntactically appropriate production rules R \u2208 G:\n\nPr(at = APPLYRULE[R] | Tt, \u02c6X) = softmaxR (g(ht))\n\n(2)\n\nwhere g(\u00b7) is a 2-layer MLP with a tanh non-linearity.\n\nTerminal Actions For actions at = GETTOKEN[y], we compute the probability Pr(at | Tt, \u02c6X) by\ncombining a small vocabulary V of tokens commonly observed in the training data with a copying\nmechanism [24, 30] over the input X to handle UNK tokens. Speci\ufb01cally, we learn two functions\npgen(ht) and pcopy(ht, X) such that pgen produces a score for each vocabulary token y \u2208 V and pcopy\ncomputes a score for copying the token y from the input. The scores are then normalized across the\nentries corresponding to the same constant, as in [7, 38].\n\n4.2 Training to Emit Idioms\n\nAs discussed earlier, training the model to emit idioms presents computational and learning chal-\nlenges. Ideally, we would like to extend Eq. (1) to maximize\n\nJ = \u2211\n\n\u03c4 \u2208T\n\n|\u03c4 |\u220f\n\ni=1\n\nPr(a\u03c4i | T\u03c4i , \u02c6X)\n\n(3)\n\nwhere T is a set of different action traces that may produce the output AST T . The traces \u03c4 \u2208 T dif-\nfer only in their possible choices of idiom actions APPLYRULE[opI] that emit some tree fragments\nof T in a single step. However, computing Eq. (3) is intractable because idiom occurrences overlap\nand cause combinatorial explosion in the number of traces T . Instead, we apply Jensen\u2019s inequality\nand maximize a lower bound:\n\nlog J = log \u2211\n\n\u03c4 \u2208T\n\n|\u03c4 |\u220f\n\ni=1\n\nPr(a\u03c4i | T\u03c4i , \u02c6X) \u2265 log(|T |) +\n\n5\n\n1\n\n|T | \u2211\n\n\u03c4 \u2208T\n\n|\u03c4 |\u2211\n\ni=1\n\nlog Pr(a\u03c4i | T\u03c4i , \u02c6X)\n\n(4)\n\n\fLet A(Tt) = {a\u2217\nt } \u222a I(Tt) be the set of all valid actions to expand the AST Tt at timestep t. Here\na\u2217\nt is the action from the original action trace that generates T using the original CFG and I(Tt)\nis the set of idiom actions APPLYRULE[opI] also applicable at the node to be expanded in Tt. Let\nc(T , t) also denote the number of traces \u03c4 \u2208 T that admit an action choice for the AST Tt from\nthe original action trace. Since each action a \u2208 A(Tt) occurs in the sum in Eq. (4) with probability\n\nc(T , t)/ |A(Tt)|, we can rearrange this sum over traces as a sum over timesteps of the original trace:\n|\u03c4 |\u2211\n|T | \u2211\n= \u2211\n|A(Tt)| \u2211\n|A(Tt)|[log Pr(a\u2217\n\u2248 \u2211\n\nlog Pr(at = APPLYRULE[opI] | Tt, \u02c6X)]\n\nlog Pr(a | T\u03c4i , \u02c6X) = E\nTt\u223cT\n\nt | Tt, \u02c6X) +\u2211\n\n1\n\n|A(Tt)| \u2211\n\nlog Pr(a\u03c4i | T\u03c4i , \u02c6X) =\n\nlog Pr(a | T\u03c4i , \u02c6X)\n\nlog Pr(a | T\u03c4i , \u02c6X)\n\nc(T , t)\n|A(Tt)|\n\nc(T , t)\n\n|T |\n\n1\n\n|T | \u2211\n\n\u2211\n\nt\n\na\u2208A(Tt)\n\n1\n\n\u03c4 \u2208T\n\ni=1\n\na\u2208A(Tt)\n\na\u2208A(Tt)\n\n1\n\n1\n\nt\n\nt\n\nI\u2208M (Tt)\n\n(5)\n\nIn the last step of Equation (5), we approximate\nthe expectation over ASTs randomly drawn\nfrom all traces T using only the original trace\n(containing all possible Tt) as a Monte Carlo\nestimate.\n\nIntuitively, at each timestep during training we\nencourage the model to emit either the orig-\ninal AST action for this timestep or any ap-\nplicable idiom that matches the AST at this\nstep, with no penalty to either choice. However,\nto avoid the combinatorial explosion, we only\nteacher-force the original generation trace (not\nthe idiom bodies), thus optimizing the bound\nin Eq. (5). Figure 3 illustrates this optimization\nprocess on an example.\n\nAt inference time, whenever the model emits\nan APPLYRULE[opI] action, we teacher-force\nthe body of I by substituting the embedding\nof the previous action at\u22121 with embedding of\nthe previous action in the idiom de\ufb01nition, thus\nemulating the tree fragment expansion. Out-\nside the bounds of I (i.e. within the hole sub-\ntrees of I) we use the actual at\u22121 as usual.\n\n5 Evaluation\n\nFigure 3: Decoding the AST sorted(my_list,\nreverse=True), \ufb01gure adapted from [38]. Sup-\npose an idiom I = sorted( \u2113 , reverse=True) is\nmined and added as an operator opI(\u2113) to the\ngrammar. At training time, PATOIS adjusts the\ncross-entropy objective at timestep t2 to addition-\nally allow opI as a valid production, with no\nchange to further decoding. At inference time, if\ndecoder emits an action at2 = APPLYRULE[opI],\nPATOIS unrolls I on the \ufb02y by teacher-forcing the\nshaded portion of the AST generation.\n\nDatasets We evaluate PATOIS on two semantic parsing datasets: Hearthstone [24] and Spider [41].\n\nHearthstone is a dataset of 665 card descriptions from the trading card game of the same name, along\nwith the implementations of their effects in Python using the game APIs. The descriptions act as NL\nspecs X, and are on average 39.1 words long.\n\nSpider is a dataset of 10,181 questions describing 5,693 unique SQL queries over 200 databases\nwith multiple tables each. Each question pertains to a particular database, whose schema is given to\nthe synthesizer. Database schemas do not overlap between the train and test splits, thus challenging\nthe model to generalize across different domains. The questions are on average 13 words long and\ndatabases have on average 27.6 columns and 8.8 foreign keys.\n\nImplementation We mine the idioms using the training split of each dataset. Thus PATOIS cannot\nindirectly over\ufb01t to the test set by learning its idioms, but it also cannot generalize beyond the idioms\nthat occur in the training set. We run type-based MCMC (Section 3) for 10 iterations with \u03b1 = 5\n\n6\n\n\fTable 2: Ablation tests on the Hearthstone dev set.\n\nTable 3: Ablation tests on the Spider dev set.\n\nModel\n\nK Exact\nmatch\n\nSentence\nBLEU\n\nCorpus\nBLEU\n\nBaseline decoder \u2014 0.197\n\nPATOIS, ScoreCov\n\nPATOIS, ScoreCXE\n\n10\n20\n40\n80\n\n10\n20\n40\n80\n\n0.151\n0.091\n0.167\n0.197\n\n0.151\n0.167\n0.182\n0.151\n\n0.767\n\n0.781\n0.745\n0.765\n0.780\n\n0.780\n0.787\n0.773\n0.771\n\n0.763\n\n0.785\n0.745\n0.764\n0.774\n\n0.783\n0.782\n0.770\n0.768\n\nModel\n\nK Exact match\n\nBaseline decoder \u2014\n\nPATOIS, ScoreCov\n\nPATOIS, ScoreCXE\n\n10\n20\n40\n80\n\n10\n20\n40\n80\n\n0.395\n\n0.394\n0.379\n0.395\n0.407\n\n0.368\n0.382\n0.387\n0.416\n\nand d = 0.5. After ranking (with either ScoreCOV or ScoreCXE) and \ufb01ltering, we use K top-ranked\nidioms to train the generative model. We ran ablation experiments with K \u2208 {10, 20, 40, 80}.\n\nAs described in Section 4, for all our experiments we used a tree-based decoder with a pointer mech-\nanism as the synthesizer f , which we implemented in PyTorch [27]. For the Hearthstone dataset,\nwe use a bidirectional LSTM [16] to implement the description encoder \u02c6X = fenc(X), similarly to\nYin and Neubig [38]. The word embeddings \u02c6x and hidden LSTM states h have dimension 256. The\nmodels are trained using the Adadelta optimizer [42] with learning rate 1.0, \u03c1 = 0.95, \u03b5 = 10\u22126 for\nup to 2,600 steps with a batch size of 10.\n\nFor the Spider dataset, word embeddings \u02c6x have dimension 300, and hidden LSTM states h have\ndimension 256. The models are trained using the Adam optimizer [20] with \u03b21 = 0.9, \u03b22 = 0.999,\n\u03b5 = 10\u22129 for up to 40,000 steps with a batch size of 10. The learning rate warms up linearly up to\n2.5 \u00d7 10\u22124 during the \ufb01rst 2,000 steps, and then decays polynomially by (1 \u2212 t/T )\u22120.5 where T is\nthe total number of steps. Each model con\ufb01guration is trained on one NVIDIA GTX 1080 Ti GPU.\n\nThe Spider tasks additionally include the database schema as an input in the description. We fol-\nlow a recent approach of embedding the schema using relation-aware self-attention within the en-\ncoder [34]. Speci\ufb01cally, we initialize a representation for each column, table, and word in the\nquestion, and then update these representations using 4 layers of relation-aware self-attention [32]\nusing a graph that describes the relations between columns and tables in the schema. See Section A\nin the appendix for more details about the Spider schema encoder.\n\n5.1 Experimental Results\n\nIn each con\ufb01guration, we compare the performance of equivalent trained models on the same dataset\nwith and without idiom-based training of PATOIS. For fairness, we show the performance of the same\ndecoder implementation described in Section 4.1 as a baseline rather than the state-of-the-art results\nachieved by different approaches from the literature. Thus, our baseline is the decoder described\nin Section 4.1 trained with a regular cross-entropy objective rather than the PATOIS objective in\nEquation (5). Following prior work, we evaluate program generation as a semantic parsing task, and\nmeasure (i) exact match accuracy and BLEU scores for Hearthstone and (ii) exact match accuracy\nof program sketches for Spider.\n\nrespectively.\n\nTables 2 and 3 show our ablation analysis of different con\ufb01gurations of PATOIS on the Hearth-\nstone and Spider dev sets,\nTable 4 shows the test set results of the best\nmodel con\ufb01guration for Hearthstone (the test instances for the Spider dataset are unreleased).\nAs the results show, small numbers of idioms do not\nsigni\ufb01cantly change the exact match accuracy but\nimprove BLEU score, and K = 80 gives a signi\ufb01-\ncant improvement in both the exact match accuracy\nand BLEU scores. The improvement is even more\npronounced on the test set with 4.5% improvement\nin exact match accuracy and more than 4 BLEU\npoints, which shows that mined training set idioms\n\nTable 4: Test set results on Hearthstone (us-\ning the best con\ufb01gurations on the dev set).\n\nBaseline\nPATOIS\n\nSentence\nBLEU\n\nCorpus\nBLEU\n\nExact\nmatch\n\n0.152\n0.197\n\n0.743\n0.780\n\n0.723\n0.766\n\nModel\n\n7\n\n\fdef __init__(self):\n\n\u21130 : id =\n\nsuper().__init__( \u21130 : str , \u21131 : int ,\n\ncopy.copy( \u21131 : expr )\n\nSELECT COUNT( \u21130 : col ), \u2113\u2217\n\n1 WHERE \u2113\u2217\n\n2\n\nINTERSECT \u2113?\n\n4 : sql EXCEPT \u2113?\n\n5 : sql\n\nCHARACTER_CLASS. \u21133 : id ,\n\nCARD_RARITY. \u21134 : id , \u2113?\n\n5 )\n\nclass \u21130 : id ( \u21131 : id ):\ndef __init__(self):\n\nWHERE \u21130 : col = $terminal\n\nFigure 4: Five examples of commonly used idioms from the Hearthstone and Spider datasets.\n\nFigure 5: The distribution of used idioms in the inferred ASTs on the Hearthstone test set. Left: in\nthe ASTs exactly matched with ground truth; Right: all ASTs.\n\ngeneralize well to the whole data distribution. As mentioned above, we compare only to the same\nbaseline architecture for fairness, but PATOIS could also be easily implemented on top of the struc-\ntural CNN decoder of Sun et al. [35], the current state of the art on the Hearthstone dataset.\n\nFigure 4 shows some examples of idioms that were frequently used by the model. On Hearthstone,\nthe most popular idioms involve common syntactic elements (e.g. class and function de\ufb01nitions) and\ndomain-speci\ufb01c APIs commonly used in card implementations (e.g. CARD_RARITY enumerations or\ncopy.copy calls). On Spider, they capture the most common combinations of SQL syntax, such\nas a SELECT query with a single COUNT column and optional INTERSECT or EXCEPT clauses. Notably,\npopular idioms are also often big: for instance, the \ufb01rst idiom in Figure 4 expands to a tree fragment\nwith more than 20 nodes. Emitting it in a single step vastly simpli\ufb01es the decoding process.\n\nWe further conducted qualitative experiments to analyze actual idiom usage by PATOIS on the\nHearthstone test set. Figure 5 shows the distribution of idioms used in the inferred (not ground\ntruth) ASTs. A typical program involves 7 idioms on average, or 6 for the programs that exactly\n\nmatch the ground truth. Despite the widespread usage of idioms, not all of the mined idioms eI were\n\nuseful: only 51 out of K = 80 idioms appear in the inferred ASTs. This highlights the need for an\nend-to-end version of PATOIS where idiom mining would be directly optimized to bene\ufb01t synthesis.\n\n6 Related Work\n\nProgram synthesis & Semantic parsing Program synthesis from natural language and input-\noutput examples has a long history in Programming Languages (PL) and Machine Learning (ML)\ncommunities (see Gulwani et al. [15] for a survey). When an input speci\ufb01cation is limited to natu-\nral language, the resulting problem can be considered semantic parsing [22]. There has been a lot\nof recent interest in applying recurrent sequence-based and tree-based neural networks to semantic\nparsing [11, 18, 21, 38, 40]. These approaches commonly use insights from the PL literature, such\nas grammar-based constraints to reduce the search space, non-deterministic training oracles to en-\nable multiple executable interpretations of intent, and supervision from program execution. They\ntypically either supervise the training on one or more golden programs, or use reinforcement learn-\ning to supervise the training from a neural program execution result [26]. Our PATOIS approach is\n\n8\n\n\fapplicable to any underlying neural semantic parsing model, as long as it is supervised by a corpus\nof golden programs. It is, however, most easily applicable to tree-based and graph-based models,\nwhich directly emit the AST of the target program. In this work we have evaluated PATOIS as applied\non top of the sequence-to-tree decoder of Yin and Neubig [38], and extended it with a novel training\nregime that teaches the decoder to emit idiom operators in place of the idiomatic code fragments.\n\nSketch generation Two recent works [12, 25] learn abstractions of the target program to compress\nand abstract the reasoning process of a neural synthesizer. Both of them split the generation process\ninto sketch generation and sketch completion, wherein the \ufb01rst stage emits a partial tree/sequence\n(i.e. a sketch of the program) and the second stage \ufb01lls in the holes in this sketch. While sketch\ngeneration is typically implemented with a neural model, sketch completion can be either a different\nneural model or a combinatorial search. In contrast to PATOIS, both works de\ufb01ne the grammar of\nsketches manually by a deterministic program abstraction procedure and only allow a single top-\nlevel sketch for each program. In addition, an earlier work of Bo\u0161njak et al. [6] also formulates\nprogram synthesis as sketch completion, but in their work program sketches are manually provided\nrather than learned. In PATOIS, we learn the abstractions (code idioms) automatically from a corpus\nand allow them to appear anywhere in the program, as is common in real-life programming.\n\nLearning abstractions Recently, Ellis et al. [13] developed an Explore, Compress & Com-\npile (EC2) framework for automatically learning DSLs for program synthesis from I/O examples\n(such as the DSLs used by FlashFill [14] and DeepCoder [5]). The work\ufb02ow of EC2 is similar to\nPATOIS, with three stages: (a) learn new DSL subroutines from a corpus of tasks, (b) train a recogni-\ntion model that maps a task speci\ufb01cation to a distribution over DSL operators as in DeepCoder [5],\nand (c) use these operators in a program synthesizer. PATOIS differs from EC2 in three aspects: (i)\nwe assume a natural language speci\ufb01cation instead of examples, (ii) to handle NL speci\ufb01cations,\nour synthesizer is a neural semantic parser instead of enumerative search, and (iii) most importantly,\nwe discover idioms that compress general-purpose languages instead of extending DSLs. Unlike\nfor inductive synthesis DSLs such as FlashFill, the existence of useful DSL abstractions for general-\npurpose languages is not obvious, and our work is the \ufb01rst to demonstrate them.\n\nConcurrently with this work, Iyer et al. [17] developed a different approach of learning code idioms\nfor semantic parsing. They mine the idioms using a variation of byte-pair encoding (BPE) compres-\nsion extended to ASTs and greedily rewrite all the dataset ASTs in terms of the found idioms for\ntraining. While the BPE-based idiom mining is more computationally ef\ufb01cient than non-parametric\nBayesian inference of PATOIS, introducing ASTs greedily tends to lose information about overlap-\nping idioms, which we address in PATOIS using our novel training objective described in Section 4.2.\n\nAs described previously, our code idiom mining is an extension of the procedure developed by Alla-\nmanis et al. [2, 3]. They are the \ufb01rst to use the tree substitution grammar formalism and Bayesian\ninference to \ufb01nd non-trivial common idioms in a corpus of code. However, their problem formaliza-\ntion does not involve any application for the learned idioms beyond their explanatory power.\n\n7 Conclusion\n\nSemantic parsing, or neural program synthesis from natural language, has made tremendous progress\nover the past years, but state-of-the-art models still struggle with program generation at multiple lev-\nels of abstraction. In this work, we present a framework that allows incorporating learned coding\npatterns from a corpus into the vocabulary of a neural synthesizer, thus enabling it to emit high-level\nor low-level program constructs interchangeably at each generation step. Our current instantiation,\nPATOIS, uses Bayesian inference to mine common code idioms, and employs a novel nondeterminis-\ntic training regime to teach a tree-based generative model to optionally emit whole idiom fragments.\nSuch dataset abstraction using idioms improves the performance of neural program synthesis.\n\nPATOIS is only the \ufb01rst step toward learned abstractions in program synthesis. While code idioms\noften correlate with latent semantic concepts and our training regime allows the model to learn\nwhich idioms to use and in which context, our current method does not mine them with the intent to\ndirectly optimize their usefulness for generation. In future work, we want to alleviate this by jointly\nlearning the mining and synthesis models, thus optimizing the idioms\u2019 usefulness for synthesis by\nconstruction. We also want to incorporate program semantics into the idiom de\ufb01nition, such as data\n\ufb02ow patterns or natural language phrases from task specs.\n\n9\n\n\fReferences\n\n[1] C. C. Aggarwal and J. Han. Frequent pattern mining. Springer, 2014.\n\n[2] M. Allamanis and C. Sutton. Mining idioms from source code. In Proceedings of the 22nd ACM SIGSOFT\n\nInternational Symposium on Foundations of Software Engineering (FSE), pages 472\u2013483. ACM, 2014.\n\n[3] M. Allamanis, E. T. Barr, C. Bird, P. Devanbu, M. Marron, and C. Sutton. Mining semantic loop idioms.\n\nIEEE Transactions on Software Engineering, 2018.\n\n[4] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate.\n\nIn Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.\n\n[5] M. Balog, A. L. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow. DeepCoder: Learning to write\nprograms. In Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017.\n\n[6] M. Bo\u0161njak, T. Rockt\u00e4schel, J. Naradowsky, and S. Riedel. Programming with a differentiable Forth\ninterpreter. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70,\npages 547\u2013556, 2017.\n\n[7] M. Brockschmidt, M. Allamanis, A. L. Gaunt, and O. Polozov. Generative code modeling with graphs.\n\nIn Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.\n\n[8] T. Cohn, P. Blunsom, and S. Goldwater. Inducing tree-substitution grammars. Journal of Machine Learn-\n\ning Research, 11(Nov):3053\u20133096, 2010.\n\n[9] J. Devlin, R. Bunel, R. Singh, M. Hausknecht, and P. Kohli. Neural program meta-induction. In Advances\n\nin Neural Information Processing Systems (NIPS), pages 2080\u20132088, 2017.\n\n[10] J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A.-r. Mohamed, and P. Kohli. RobustFill: Neural program\nIn Proceedings of the 34th International Conference on Machine Learning\n\nlearning under noisy I/O.\n(ICML), 2017.\n\n[11] L. Dong and M. Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual\n\nMeeting of the Association for Computational Linguistics (ACL), 2016.\n\n[12] L. Dong and M. Lapata. Coarse-to-\ufb01ne decoding for neural semantic parsing. In Proceedings of the 56th\n\nAnnual Meeting of the Association for Computational Linguistics (ACL), 2018.\n\n[13] K. Ellis, L. Morales, M. Sabl\u00e9-Meyer, A. Solar-Lezama, and J. Tenenbaum. Learning libraries of sub-\nroutines for neurally-guided Bayesian program induction. In Advances in Neural Information Processing\nSystems, pages 7816\u20137826, 2018.\n\n[14] S. Gulwani. Automating string processing in spreadsheets using input-output examples. In Proceedings of\nthe 38th ACM Symposium on Principles of Programming Languages (POPL), volume 46, pages 317\u2013330,\n2011.\n\n[15] S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends\u00ae in Programming\n\nLanguages, 4(1-2):1\u2013119, 2017.\n\n[16] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735\u20131780, 1997.\n\n[17] S. Iyer, A. Cheung, and L. Zettlemoyer. Learning programmatic idioms for scalable semantic parsing. In\n\nEMNLP, 2019.\n\n[18] R. Jia and P. Liang. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual\n\nMeeting of the Association for Computational Linguistics (ACL), volume 1, pages 12\u201322, 2016.\n\n[19] A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search\nfor real-time program synthesis from examples. In Proceedings of the 6th International Conference on\nLearning Representations (ICLR), 2018.\n\n[20] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of 3rd International\n\nConference on Learning Representations (ICLR), 2015.\n\n[21] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. In Proceedings\n\nof the 4th International Conference on Learning Representations (ICLR), 2016.\n\n[22] P. Liang. Learning executable semantic parsers for natural language understanding. Communications of\n\nthe ACM, 59(9):68\u201376, 2016.\n\n10\n\n\f[23] P. Liang, M. I. Jordan, and D. Klein. Type-based MCMC. In Human Language Technologies: The 2010\nAnnual Conference of the North American Chapter of the Association for Computational Linguistics,\npages 573\u2013581. Association for Computational Linguistics, 2010.\n\n[24] W. Ling, P. Blunsom, E. Grefenstette, K. M. Hermann, T. Ko\u02c7cisk`y, F. Wang, and A. Senior. Latent\npredictor networks for code generation. In ACL, volume 1, pages 599\u2013609, 2016. URL https://github.\ncom/deepmind/card2code.\n\n[25] V. Murali, L. Qi, S. Chaudhuri, and C. Jermaine. Neural sketch learning for conditional program genera-\n\ntion. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.\n\n[26] A. Neelakantan, Q. V. Le, M. Abadi, A. McCallum, and D. Amodei. Learning a natural language interface\nwith neural programmer. In Proceedings of the 5th International Conference on Learning Representations\n(ICLR), 2017.\n\n[27] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and\n\nA. Lerer. Automatic differentiation in PyTorch. 2017.\n\n[28] O. Polozov and S. Gulwani. FlashMeta: A framework for inductive program synthesis.\n\nIn Proceed-\nings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems,\nLanguages, and Applications (OOPSLA), pages 107\u2013126, 2015.\n\n[29] M. Post and D. Gildea. Bayesian learning of a tree substitution grammar. In Proceedings of the ACL-\n\nIJCNLP 2009 Conference Short Papers, pages 45\u201348. Association for Computational Linguistics, 2009.\n\n[30] A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator networks. In\nProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), volume 1,\npages 1073\u20131083, 2017.\n\n[31] J. Sethuraman. A constructive de\ufb01nition of Dirichlet priors. Statistica sinica, pages 639\u2013650, 1994.\n\n[32] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations.\n\nIn Pro-\nceedings of the 2018 Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies, Volume 2 (Short Papers), 2018.\n\n[33] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-Attention with Relative Position Representations. In Pro-\nceedings of the 2018 Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464\u2013468. Association for\nComputational Linguistics, 2018. doi: 10.18653/v1/N18-2074.\n\n[34] R. Shin. Encoding database schemas with relation-aware self-attention for text-to-SQL parsers. arXiv\n\npreprint arXiv:1906.11790, 2019.\n\n[35] Z. Sun, Q. Zhu, L. Mou, Y. Xiong, G. Li, and L. Zhang. A grammar-based structural CNN decoder for\n\ncode generation. In AAAI, 2019.\n\n[36] Y. W. Teh and M. I. Jordan. Hierarchical Bayesian nonparametric models with applications. Bayesian\n\nnonparametrics, 1:158\u2013207, 2010.\n\n[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, \u0141. Kaiser, and I. Polosukhin.\nIn Advances in Neural Information Processing Systems, pages 5998\u20136008.\n\nAttention is all you need.\nCurran Associates, Inc., 2017.\n\n[38] P. Yin and G. Neubig. A syntactic neural model for general-purpose code generation. In ACL, July 2017.\n\n[39] P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig. Learning to mine aligned code and natural\nlanguage pairs from StackOver\ufb02ow. In International Conference on Mining Software Repositories (MSR),\npages 476\u2013486. ACM, 2018.\n\n[40] P. Yin, C. Zhou, J. He, and G. Neubig. StructVAE: Tree-structured latent variable models for semi-\nsupervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computa-\ntional Linguistics (ACL), 2018.\n\n[41] T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and\nD. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing\nand text-to-SQL task. In EMNLP, 2018. URL https://yale-lily.github.io/spider.\n\n[42] M. D. Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.\n\n11\n\n\f", "award": [], "sourceid": 5774, "authors": [{"given_name": "Eui Chul", "family_name": "Shin", "institution": "UC Berkeley"}, {"given_name": "Miltiadis", "family_name": "Allamanis", "institution": "Microsoft Research"}, {"given_name": "Marc", "family_name": "Brockschmidt", "institution": "Microsoft Research"}, {"given_name": "Alex", "family_name": "Polozov", "institution": "Microsoft Research"}]}