Dear Gregor, We regret to inform you that your submitted artifact was not accepted by the ECOOP 2015 Artifact Evaluation Committee (AEC): Title: Concrete Types for TypeScript Authors: Gregor Richards (University of Waterloo, Canada) Francesco Zappa Nardelli (INRIA, France) Jan Vitek (Northeastern University, USA) Artifact site: https://ecoop2015aec.hotcrp.com/paper/12?cap=012a210qg9L-qnY 12 artifacts were accepted out of 15 submissions. We hope that you will make the artifact publicly available nevertheless, and that the reviews will be helpful in revising the artifact for that purpose. Below we describe the evaluation process and include the detailed reviews. Each artifact was evaluated by three AEC members. First, each reviewer would "kick the tires" of the artifact in order to check that it could be reviewed at all; this phase ruled out problems such as corrupt artifact archive files and VM images that could not be loaded. The approach used was to go through the "Getting Started Guide" that you provided as part of your artifact. The results of that phase are included in the reviews in the same form that you received on March 26, 2015, when you had the opportunity to respond to any issues raised by the reviewers after they kicked the tires. In the second phase, the reviewers evaluated the artifact and wrote the reviews. Each reviewer read your paper, and wrote a summary providing a brief characterization of the context for the artifact. In the artifact evaluation, the reviewers focused on four key questions: (1) "Is the artifact consistent with the paper?" (2) "Is the artifact complete?" (3) "Is the artifact well documented?" and (4) "Is the artifact easy to reuse?" They decided on acceptance or rejection, and provided the review text itself, containing characterizations of strong and weak points for the artifact, advice about potential improvements, and possibly more. In the third phase, AEC members discussed their reviews and converged on a decision about acceptance or rejection. Many updates were applied to the reviews during this phase, reflecting that the discussions gave rise to new insights and changes in the evaluation. All AEC members (who are not conflicted with your artifact) could see the reviews and discussions, thus allowing for a calibration of the reviews across different artifacts. Finally, a decision about acceptance or rejection was reached for every artifact, as indicated for your artifact in this notification. We would like to thank you for submitting your artifact, and we very much hope that you will continue to submit your artifacts to ECOOP and elsewhere in the future! Camil Demetrescu and Matthew Flatt ECOOP 2015 AEC co-chairs ecoop15aec@dis.uniroma1.it =========================================================================== ECOOP 2015 Artifacts Review #12A --------------------------------------------------------------------------- Paper #12: Concrete Types for TypeScript --------------------------------------------------------------------------- ===== Kicking-the-tires outcome ===== Works for me. Can boot VM, see instructions and execute the example commend. ===== Computing platform(s) used for assessing the artifact ===== Windows 7 + Oracle VirtualBox 4.3 Overall merit: C. Does not live up to the expectations raised by the associated paper (reject) Confidence: Y. I am knowledgeable in this area, but not an expert Relevance: B. Reasonably relevant ===== Summary ===== This is a virtual machine with the source code for a part of the paper - namely the modified TypeScript compiler for which the full source is included. One can directly run the modified tsc compiler on several example projects and inspect the resulting code. This is nice and as it should be. For the second part of the paper - namely the modified TruffleJS runtime - only a patch is included, which is for a version the source of which is not included. Also, the benchmark programs shown in Fig. 5 are seemingly only partially included in the artifact. There is nothing in the howto on how to run them. There is a "make" script in the benchmark folder with a wrong path; even if corrected (thanks to the other reviewer!) it remains unclear where the results are. Furthermore, the names of the benchmarks are different from Fig. 4 and Fig. 5 in the paper and most seem to be missing anyway. From the paper I expected to be able to re-run the benchmarks in the VM, including some sort of howto and so I can easily check it (i.e. a comparison with the paper). This is not the case. ===== Evaluation ===== + Virtual Machine makes reproducibility easier + source code for modified typescript compiler is included + the artifact includes the PATCH for TruffleJS. - the artifact does not include the full code for TruffleJS nor an executable - the artifact does not include some of the benchmarks programs - the scripts for benchmarking are not usable as such. =========================================================================== ECOOP 2015 Artifacts Review #12B --------------------------------------------------------------------------- Paper #12: Concrete Types for TypeScript --------------------------------------------------------------------------- ===== Kicking-the-tires outcome ===== The virtual machine image runs with no issues. Running tsc and node inside the VM as described in the HOWTO.txt file works. There's no build instructions for the source, and I can't figure out to get it to build. Running ./bootstrap or make in ~/typescript-gradual ends with the following error messages: bin/tsc.js: 1: bin/tsc.js: /bin: Permission denied bin/tsc.js: 2: bin/tsc.js: Syntax error: word unexpected (expecting ")") Makefile:135: recipe for target 'built/local/typescript.js' failed make: *** [built/local/typescript.js] Error 2 ===== Computing platform(s) used for assessing the artifact ===== Hardware: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz, 4GB RAM Software: Ubuntu 14.10. Kernel version 3.16.0-28-generic #37-Ubuntu SMP Mon Dec 8 17:15:28 UTC 2014 x86_64. VirtualBox version 4.3.18-dfsg-2ubuntu1. Used defaults in VirtualBox when loading the VM image. Overall merit: C. Does not live up to the expectations raised by the associated paper (reject) Confidence: Y. I am knowledgeable in this area, but not an expert Relevance: A. Significantly relevant ===== Summary ===== The paper extends the type system of TypeScript with "concrete types" which provide stronger guarantees like those one expects from a statically typed language where they are used, while allowing concrete types to be mixed with dynamic constructs. The artifact is a modified TypeScript compiler supporting this new construct and a modified JavaScript implementation supporting the annotations output by their compiler to reduce runtime type checking where the compiler can guarantee those checks are unnecessary. ===== Evaluation ===== The artifact is provided as a VM that opens with a text file explaining how to use it. The included TypeScript compiler performs as expected, compiling their provided example programs. The source of the compiler is included and compiles to a working version of the compiler. The Octane benchmarks appear to be missing from the VM. The benchmarking script included in the VM doesn't work due to an incorrect path for the tsc executable, which was easily fixed. After doing so, the numbers did not show the same speedups shown in the paper (e.g. I got a slowdown on bg-nbody which the paper claims a 20% speedup for); that said, the odd results may have merely been due to running the benchmarks inside a VM. It would have been nice to have a script for regenerating the benchmark tables that appear in the paper to be sure I was actually running the benchmarks correctly. =========================================================================== ECOOP 2015 Artifacts Review #12C --------------------------------------------------------------------------- Paper #12: Concrete Types for TypeScript --------------------------------------------------------------------------- ===== Kicking-the-tires outcome ===== VM loads, custom version of tsc is on the path and compiles a few randomly selected examples successfully. ===== Computing platform(s) used for assessing the artifact ===== - OSX 10.9, VirtualBox 4.3.26, iMac from 2010 - Ubuntu 14.10 (4.0-RC5 kernel), converted to VMWare Workstation 11.0 format Overall merit: C. Does not live up to the expectations raised by the associated paper (reject) Confidence: X. I am an expert in this area Relevance: C. Marginally relevant ===== Summary ===== The artifact contains: 1) a compiler, written in TypeScript, from the version of TypeScript described in the accompanying paper to JavaScript (apparently ECMAScript 5, not ECMAScript 3, which should be made clear earlier in the paper) 2) several examples of programs 3) a diff showing patches made to TruffleJS, though TruffleJS is not available directly. ===== Evaluation ===== The paper made the following claims: 1) That their implementation demonstrates performance benefits of 2-32% relative to the existing TypeScript compiler using their TruffleJS implementation on a small benchmark. EVALUATION: The artifact contains only diffs relative to a version of TruffleJS that is not publicly available, so it is impossible to evaluate this claim. Simply including the diffs to presumably mean "trust us, we did this" is a curious choice. 2) That the semantics provides various guarantees, stated as theorems in the paper. EVALUATION: There are no detailed proofs provided in either the paper or the artifact, so it is impossible to evaluate this claim rigorously. 3) That "all TypeScript programs we have tried run without changes in our implementation" including "all the large TypeScript benchmarks from [16]". EVALUATION: All of the programs provided in the artifact compile successfully. Unfortunately, there are no unit tests provided to verify that they execute correctly. The original version of the benchmarks from [16] (the Octane benchmark, a MSR JS Crypto library and the original TypeScript compiler) are not provided. I can't seem to find these online. The MAJOR PROBLEM with the artifact is that the code in the paper does not compile or execute as written. The very first code example on p8 of the paper is listed like this: var p:any = { x=3; z=4 } var f:any = func (p) { if (p.x < 10) return 10 else return p.distance() } f(p) but this gives me parse errors. It only actually compiles if I write it in more conventional JavaScript style: var p:any = {x: 3, z: 4} var f:any = function(p) { if (p.x < 10) return 10; else return p.distance(); } f(p) The next example also doesn't compile as written. If I fix it to use standard JavaScript syntax as follows: class Point { constructor(public x, public y) {} dist(p) { return 0; } }; var p:Point = {x: 3, z: 4} //Correct [comment in paper] var f:any = function (p:Point) { if (p.x < 10) return 0; else return p.distance(p) } //Wrong there are still problems. First, the line with the comment reading "Correct" isn't correct. At the least, it uses field names x and z whereas Point uses x and y. But even fixing that, I get the following error when I compile: Cannot convert {x: number, y: number} to 'Point'. If I change it to var p:Point = new Point(3, 4) then that line works. However, the next example in the paper is both wrong and confusing. It is written: var s:!Point = new Point{5, 6} f(s); // evaluates to 10 First, the syntax "new Point{5, 6}" is incorrect. Changing it to "new Point(5, 6)" fixes the parse error. However, then the compiler gives me the following error at the position just before "!Point": Expected type name This is strange given that the paper very clearly states that they've added a new type constructor "!". At this point, I go digging into the examples and find that !T does not exist anywhere. I do find an annotation of the form "like Point". Is this supposed to be the optional type, and "Point" actually has the semantics of "!Point"? No, it seems not: var p:like Point = {x: 3, y: 4} gives me the same error message as above. Both of the following work: var p:like Point = new Point(3, 4) var s:Point = new Point(3, 4) However, the comment that says f(s) should evaluate to 10 is false! The function f won't even compile: The property 'distance' does not exist on value of type 'Point' If I change the annotation of the argument of f to "like Point", I get the same error. Clearly, optional types are not working. Moving on the next example, at the top of p9, it too needs to be fixed, presumably to read as follows: class TypedPoint { constructor(public x:number, public y:number){} dist(p) { return 0 : number; } } var t:TypedPoint = new TypedPoint(1, 2) (t).x = "o"; // DYNAMIC ERR: type mismatch Unfortunately, there is no such dynamic error from the last line. The program compiles and executes just fine, and t.x has been set to "o" by the end of the program, as I verified with a console.log call. The remainder of the section is similarly impossible to repeat, so I'll just stop here and say that it is unacceptable for an artifact to differ so substantially from what was described in the paper without any mention whatsoever of the differences in the accompanying documentation! I certainly hope there is something I've missed here.