NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:6390
Title:SPoC: Search-based Pseudocode to Code

Main contributions: * New dataset of line-by-line, human-generated pseudocode for learning to map from descriptions to source code. * A new kind of execution-guided decoding method for leveraging error messages generated by the compiler while generating target code. The first stage generates a set of candidate translations from pseudocode to code for each line. The second stage enumerates over combinations of candidates, tries compiling them, and then learns to use the error messages to prioritize which combinations to explore next. There are three well-qualified reviewers who did a great job with their reviews and were active in the discussions. The discussions centered around the following points: * Is this dataset a step forward compared to NAPS? We do not think the answer is clearly yes. SPoC has more human-generated alignments, but the assumption of line-by-line alignment is stronger and less realistic than NAPS. We DO NOT want SPoC to be viewed as a replacement for NAPS, but rather an intermediate point between the synthetically generated strongly-aligned part of NAPS and the weakly-aligned human-generated NAPS annotations. * The proposed method relies strongly upon the strong line-by-line alignment, and in this sense it seems like a step backwards. However, we find the way of using the learned error localization to be a novel and interesting contribution in the context of execution-guided decoding (though methods like DeepFix also learn to localize and repair programs from error messages). * There is some concern about whether the technique would still be beneficial when using a stronger model, like even one that consumed all lines of pseudocode jointly, then produced the set of per-line candidates independently. It seems that the proposed decoder will help the most when there is ambiguity in the line-by-line translations, and some of this ambiguity would be resolved by conditioning on the full pseudocode. Overall: all three reviewers have reasonable positions, and this paper could go either way. However, AC recommends following the majority vote in this case and accepting.