Merge branch 'master' into python-2.4

and more 3.7 grammar customization

.gitignore

@@ -21,3 +21,4 @@
 ChangeLog
 __pycache__
 build
+nohup.out

Makefile

@@ -27,19 +27,20 @@ check:
 check-short: pytest
 	$(MAKE) -C test check-short
 
+# Note for 2.6 use <=3.0.1 see requirements-dev.txt
 #: Tests for Python 2.7, 3.3 and 3.4
-check-2.7 check-3.3 check-3.4: pytest
+check-2.6 check-2.7 check-3.3 check-3.4 check-3.5: pytest
 	$(MAKE) -C test $@
 
 #: Tests for Python 3.2 and 3.5 - pytest doesn't work here
 # Or rather 3.5 doesn't work not on Travis
-check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
+check-3.0 check-3.1 check-3.2 check-3.6:
 	$(MAKE) -C test $@
 
 check-3.7: pytest
 
-#:Tests for Python 2.6 (doesn't have pytest)
-check-2.4 check-2.5 check-2.6:
+#:Tests for Python 2.4-2.5 (don't have pytest)
+check-2.4 check-2.5:
 	$(MAKE) -C test $@
 
 #:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0

NEWS

@@ -1,3 +1,38 @@
+uncompyle6 3.1.3 2018-04-16
+
+- Add some Python 3.7 rules, such as for handling LOAD_METHOD (not complete)
+- Fix some fragment bugs
+- small doc changes
+
+uncompyle6 3.1.2 2018-04-08 Eastern Orthodox Easter
+
+- Python 3.x subclass and call parsing fixes
+- Allow/note running on Python 3.1
+- improve 3.5+ BUILD_MAP_UNPACK
+- DRY instruction building code between 2.x and 3.x
+- expand testing
+
+uncompyle6 3.1.1 2018-04-01 Easter April Fool's
+
+Jesus on Friday's New York Times puzzle: "I'm stuck on 2A"
+
+- fill out 3.5+ BUILD_MAP_UNPACK (more work is needed)
+- fill out 3.4+ CALL_FUNCTION_... (more work is needed)
+- fill out 3.5 MAKE_FUNCTION  (more work is needed)
+- reduce 3.5, 3.6 control-flow bugs
+- reduce ambiguity in rules that lead to long (exponential?) parses
+- limit/isolate some 2.6/2.7,3.x grammar rules
+- more runtime testing of decompiled code
+- more removal of parenthesis around calls via setting precidence
+
+uncompyle6 3.1.0 2018-03-21 Equinox
+
+- Add code_deparse_with_offset() fragment function.
+- Correct paramenter call fragment deparse_code()
+- Lots of 3.6, 3.x, and 2.7 bug fixes
+  About 5% of 3.6 fail parsing now. But
+  semantics still needs much to be desired.
+
 uncompyle6 3.0.1 2018-02-17
 
 - All Python 2.6.9 standard library files weakly verify

README.rst

@@ -52,8 +52,17 @@ You get the idea. This code pulls all of these forks together and
 *moves forward*. There is some serious refactoring and cleanup in this
 code base over those old forks.
 
-This project has the most complete support for Python 3.3 and above
-and the best all-around Python support.
+This demonstrably does the best in decompiling Python across all
+Python versions. And even when there is another project that only
+provides decompilation for subset of Python versions, we generally do
+demonstrably better for those as well.
+
+How can we tell? By taking Python bytecode that comes distributed with
+that version of Python and decompiling these.  Among htose that
+successfully decompile, we can then make sure the resulting programs
+are syntactically correct by running the Python interpreter for that
+bytecode version.  Finally, in cases where the program has a test for
+itself, we can run the check on the decompiled code.
 
 We are serious about testing, and use automated processes to find
 bugs. In the issue trackers for other decompilers, you will find a
@@ -136,26 +145,26 @@ All of the Python decompilers that I have looked at have problems
 decompiling Python's control flow. In some cases we can detect an
 erroneous decompilation and report that.
 
-*Verification* is the process of decompiling bytecode, compiling with
-a Python for that bytecode version, and then comparing the bytecode
-produced by the decompiled/compiled program. Some allowance is made
-for inessential differences. But other semantically equivalent
-differences are not caught. For example ``1 and 0`` is decompiled to
-the equivalent ``0``; remnants of the first true evaluation (1) is
-lost when Python compiles this. When Python next compiles ``0`` the
-resulting code is simpler.
+In older versions of Python it was possible to verify bytecode by
+decompiling bytecode, and then compiling using the Python interpreter
+for that bytecode version. Having done this the bytecode produced
+could be compared with the original bytecode. However as Python's code
+generation got better, this is no longer feasible.
 
-*Weak Verification*
-on the other hand doesn't check bytecode for equivalence but does
-check to see if the resulting decompiled source is a valid Python
-program by running the Python interpreter. Because the Python language
-has changed so much, for best results you should use the same Python
-Version in checking as used in the bytecode.
+There is a kind of *weak verification* that we use that doesn't check
+bytecode for equivalence but does check to see if the resulting
+decompiled source is a valid Python program by running the Python
+interpreter. Because the Python language has changed so much, for best
+results you should use the same Python version in checking as was used
+in creating the bytecode.
 
-Finally, we have automated running the standard Python tests after
-first compiling and decompiling the test program. Results here are a
-bit weak (if not better than most other Python decompilers). But over
-time this will probably get better.
+There are however an interesting class of these programs that is
+readily available give stronger verification: those programs that
+when run check some computation, or even better themselves.
+
+And already Python has a set of programs like this: the test suite
+for the standard library that comes with Python. We have some
+code in `test/stdlib` to facilitate this kind of checking.
 
 Python support is strongest in Python 2 for 2.7 and drops off as you
 get further away from that. Support is also probably pretty good for
@@ -203,7 +212,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for Python 3 is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
 * The HISTORY_ file.

admin-tools/pycdc-runtests.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Use pycdc to run our test/bytecode* test suite
+bs=${BASH_SOURCE[0]}
+testdir=$(dirname $bs)/../test
+fulldir=$(readlink -f $testdir)
+cd $fulldir
+for dir in bytecode_* ; do
+    echo ========= $dir ================
+    cd $fulldir/$dir
+    for file in *.pyc; do
+	if ! pycdc $file > /dev/null ; then
+	    echo -----  $dir/$file ------
+	fi
+    done
+done

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.5 3.6.4 2.6.9 3.3.7 2.7.14 3.4.8'
+export PYVERSIONS='3.5.5 3.6.5 2.6.9 3.3.7 2.7.14 3.2.6 3.1.5 3.4.8'

admin-tools/uncompyle2-runtests.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Use pycdc to run our test/bytecode_2.7* test suite
+bs=${BASH_SOURCE[0]}
+topdir=$(dirname $bs)/..
+(cd $topdir && pyenv local 2.7.14)
+testdir=$topdir/test
+fulldir=$(readlink -f $testdir)
+cd $fulldir
+
+for bytecode in bytecode_2.7/*.pyc ; do
+    echo $bytecode
+    uncompyle2 $bytecode > /dev/null
+    echo ================ $bytecode rc: $? ==============
+done
+
+tmpdir=/tmp/test-2.7
+( cd bytecode_2.7_run &&
+      mkdir $tmpdir || true
+      for bytecode in *.pyc ; do
+	  shortname=$(basename $bytecode .pyc)
+	  echo $bytecode
+	  py_file=${tmpdir}/${shortname}.py
+	  typeset -i rc=0
+	  uncompyle2 $bytecode > $py_file
+	  rc=$?
+	  if (( rc == 0 )); then
+	      python $py_file
+	      rc=$?
+	  fi
+	  echo ================ $bytecode rc: $rc ==============
+      done
+)

pytest/test_fjt.py

@@ -1,8 +1,6 @@
 #!/usr/bin/env python
 from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
-from xdis.bytecode import Bytecode
-from array import array
 def bug(state, slotstate):
     if state:
         if slotstate is not None:
@@ -25,14 +23,7 @@ def test_if_in_for():
     code = bug.func_code
     scan = get_scanner(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
-        n = scan.setup_code(code)
-        bytecode = Bytecode(code, scan.opc)
-        scan.build_lines_data(code, n)
-        scan.insts = list(bytecode)
-        scan.offset2inst_index = {}
-        for i, inst in enumerate(scan.insts):
-            scan.offset2inst_index[inst.offset] = i
-        scan.build_prev_op(n)
+        scan.build_instructions(code)
         fjt = scan.find_jump_targets(False)
 
         ## FIXME: the data below is wrong.
@@ -47,14 +38,7 @@ def test_if_in_for():
         #    {'start': 62, 'end': 63, 'type': 'for-else'}]
 
         code = bug_loop.__code__
-        n = scan.setup_code(code)
-        bytecode = Bytecode(code, scan.opc)
-        scan.build_lines_data(code, n)
-        scan.insts = list(bytecode)
-        scan.build_prev_op(n)
-        scan.offset2inst_index = {}
-        for i, inst in enumerate(scan.insts):
-            scan.offset2inst_index[inst.offset] = i
+        scan.build_instructions(code)
         fjt = scan.find_jump_targets(False)
         assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
         assert scan.structs == [
@@ -68,14 +52,7 @@ def test_if_in_for():
             {'start': 48, 'end': 67, 'type': 'while-loop'}]
 
     elif 3.2 < PYTHON_VERSION <= 3.4:
-        bytecode = Bytecode(code, scan.opc)
-        scan.code = array('B', code.co_code)
-        scan.lines = scan.build_lines_data(code)
-        scan.build_prev_op()
-        scan.insts = list(bytecode)
-        scan.offset2inst_index = {}
-        for i, inst in enumerate(scan.insts):
-            scan.offset2inst_index[inst.offset] = i
+        scan.build_instructions(code)
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \
@@ -85,5 +62,6 @@ def test_if_in_for():
            {'end': 59, 'type': 'for-loop', 'start': 31},
            {'end': 63, 'type': 'for-else', 'start': 62}]
     else:
-        assert True, "FIXME: should note fixed"
+        print("FIXME: should fix for %s" % PYTHON_VERSION)
+        assert True
     return

pytest/test_grammar.py

@@ -18,46 +18,48 @@ def test_grammar():
      right_recursive, dup_rhs) = p.check_sets()
 
     # We have custom rules that create the below
-    expect_lhs = set(['expr1024', 'pos_arg', 'get_iter', 'attribute'])
+    expect_lhs = set(['pos_arg', 'get_iter', 'attribute'])
 
     unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
                       'unpack',])
     expect_right_recursive = set([('designList',
                                    ('store', 'DUP_TOP', 'designList'))])
-    if PYTHON3:
-        expect_lhs.add('load_genexpr')
+
+    if PYTHON_VERSION != 3.7:
+        unused_rhs.add('call')
+
+    if PYTHON_VERSION > 2.6:
         expect_lhs.add('kvlist')
         expect_lhs.add('kv3')
+        unused_rhs.add('dict')
+
+    if PYTHON3:
+        expect_lhs.add('load_genexpr')
 
         unused_rhs = unused_rhs.union(set("""
-        except_pop_except generator_exp classdefdeco2
-        dict
+        except_pop_except generator_exp
         """.split()))
         if PYTHON_VERSION >= 3.0:
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
-            unused_rhs.add('call')
-            if PYTHON_VERSION < 3.6:
-                # 3.6 has at least one non-custom call rule
-                # the others don't
-                unused_rhs.add('call')
-                if PYTHON_VERSION == 3.5:
+            unused_rhs.add("dict_comp")
+            unused_rhs.add("classdefdeco1")
+            if PYTHON_VERSION != 3.6:
+                if PYTHON_VERSION in (3.5, 3.7):
                     expect_right_recursive.add((('l_stmts',
-                                                 ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                                                 ('lastl_stmt', 'come_froms', 'l_stmts'))))
                     pass
                 pass
             else:
                 expect_right_recursive.add((('l_stmts',
                                              ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
-                # expect_lhs.add('kwargs1')
                 pass
             pass
         pass
     else:
         expect_lhs.add('kwarg')
-        unused_rhs.add('call')
 
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)
@@ -85,6 +87,8 @@ def test_grammar():
             """.split())
     if 2.6 <= PYTHON_VERSION <= 2.7:
         opcode_set = set(s.opc.opname).union(ignore_set)
+        if PYTHON_VERSION == 2.6:
+            opcode_set.add("THEN")
         check_tokens(tokens, opcode_set)
     elif PYTHON_VERSION == 3.4:
         ignore_set.add('LOAD_CLASSNAME')

pytest/test_single_compile.py

@@ -1,6 +1,6 @@
 from uncompyle6 import PYTHON_VERSION, deparse_code
 
-if PYTHON_VERSION >= 2.5:
+if PYTHON_VERSION >= 2.6:
     def test_single_mode():
         single_expressions = (
             'i = 1',

requirements-dev.txt

@@ -1,3 +1,3 @@
-pytest>=3.0.0
+pytest>=3.0.0,<=3.0.1
 flake8
-hypothesis<=3.8.3
+hypothesis<=3.0.0

test/.gitignore

@@ -0,0 +1 @@
+/nohup.out

test/Makefile

@@ -2,11 +2,11 @@ PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clea
       check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
       check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
       check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
-      check-3.4 check-3.5 check-5.6 5.6 5.8 \
-      grammar-coverage-2.5 grammar-coverage-2.6 grammarcoverage-2.7 \
-      grammar-coverage-3.1 grammar-coverage-3.2 grammarcoverage-3.3 \
-      grammar-coverage-3.4 grammar-coverage-3.5 grammarcoverage-3.6
-
+      check-3.4 check-3.5 check-3.6 check-3.7 check-5.6 5.6 5.8 \
+      grammar-coverage-2.5 grammar-coverage-2.6 grammar-coverage-2.7 \
+      grammar-coverage-3.1 grammar-coverage-3.2 grammar-coverage-3.3 \
+      grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6 \
+      grammar-coverage-3.7
 
 GIT2CL ?= git2cl
 PYTHON ?= python
@@ -37,18 +37,22 @@ check-3.0: check-bytecode
 #: Run working tests from Python 3.1
 check-3.1: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
 
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
 
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
@@ -59,6 +63,10 @@ check-3.5: check-bytecode
 check-3.6: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
 
+#: Run working tests from Python 3.7
+check-3.7: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify $(COMPILE)
+
 # FIXME
 #: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
 5.8 5.6:
@@ -117,26 +125,26 @@ check-bytecode-2.5:
 #: Get grammar coverage for Python 2.4
 grammar-coverage-2.4:
 	-rm $(COVER_DIR)/spark-grammar-24.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.4.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.4.cover $(PYTHON) test_pyenvlib.py --2.4.6 --max= 800
 
 #: Get grammar coverage for Python 2.5
 grammar-coverage-2.5:
-	-rm $(COVER_DIR)/spark-grammar-25.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+	-rm $(COVER_DIR)/spark-grammar-2.5.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.5.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.5.cover $(PYTHON) test_pyenvlib.py --2.5.6 --max=800
 
 #: Get grammar coverage for Python 2.6
 grammar-coverage-2.6:
-	-rm $(COVER_DIR)/spark-grammar-26.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+	-rm $(COVER_DIR)/spark-grammar-2.6.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.6.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.6.cover $(PYTHON) test_pyenvlib.py --2.6.9 --max=800
 
 #: Get grammar coverage for Python 2.7
 grammar-coverage-2.7:
-	-rm $(COVER_DIR)/spark-grammar-27.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+	-rm $(COVER_DIR)/spark-grammar-2.7.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-2.7.cover $(PYTHON) test_pyenvlib.py --2.7.14 --max=600
 
 #: Get grammar coverage for Python 3.0
 grammar-coverage-3.0:
@@ -147,33 +155,39 @@ SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pythonl
 
 #: Get grammar coverage for Python 3.1
 grammar-coverage-3.1:
-	-rm $(COVER_DIR)/spark-grammar-31.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pyenvlib.py --3.1.5
+	-rm $(COVER_DIR)/spark-grammar-3.1.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.1.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.1.cover $(PYTHON) test_pyenvlib.py --3.1.5
 
 #: Get grammar coverage for Python 3.2
 grammar-coverage-3.2:
-	-rm $(COVER_DIR)/spark-grammar-32.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pythonlib.py --bytecode-3.2
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pyenvlib.py --3.2.6
+	-rm $(COVER_DIR)/spark-grammar-3.2.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.2.cover $(PYTHON) test_pythonlib.py --bytecode-3.2
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.2.cover $(PYTHON) test_pyenvlib.py --3.2.6
 
 #: Get grammar coverage for Python 3.3
 grammar-coverage-3.3:
-	-rm $(COVER_DIR)/spark-grammar-33.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pyenvlib.py --3.3.6
+	-rm $(COVER_DIR)/spark-grammar-3.3.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.3.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.3.cover $(PYTHON) test_pyenvlib.py --3.3.7 --max=800
 
 #: Get grammar coverage for Python 3.4
 grammar-coverage-3.4:
-	-rm $(COVER_DIR)/spark-grammar-34.cover
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pyenvlib.py --3.4.2
+	-rm $(COVER_DIR)/spark-grammar-3.4.cover || true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.4.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.4.cover $(PYTHON) test_pyenvlib.py --3.4.8 --max=800
 
 #: Get grammar coverage for Python 3.5
 grammar-coverage-3.5:
-	rm $(COVER_DIR)/spark-grammar-35.cover || /bin/true
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pyenvlib.py --3.5.3
+	rm $(COVER_DIR)/spark-grammar-3.5.cover || /bin/true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.5.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.5.cover $(PYTHON) test_pyenvlib.py --3.5.5 --max=450
+
+#: Get grammar coverage for Python 3.6
+grammar-coverage-3.6:
+	rm $(COVER_DIR)/spark-grammar-3.6.cover || /bin/true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pythonlib.py --bytecode-3.6
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-3.6.cover $(PYTHON) test_pyenvlib.py --3.6.4 --max=280
 
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
@@ -192,10 +206,12 @@ check-bytecode-3.0:
 #: Check deparsing Python 3.1
 check-bytecode-3.1:
 	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
 	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.2-run --verify-run
 
 #: Check deparsing Python 3.3
 check-bytecode-3.3:
@@ -217,6 +233,10 @@ check-bytecode-3.6:
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 
+#: Check deparsing Python 3.7
+check-bytecode-3.7:
+	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)

test/bytecode_2.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.4
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.4 interpreter, they will give an error if they
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 2.4 interpreter, they are likely to give an error when they
 are miscompiled.

test/bytecode_2.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.5.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.5 interpreter, they will give an error if they
-are miscompiled.
+Furthermore the programs here are self-checking: when decompiled and
+then run again in a 2.5 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_2.6_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.6.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.6 interpreter, they will give an error if they
-are miscompiled.
+Furthermore the programs here are self-checking: when decompiled and
+then run again in a 2.6 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_2.7_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.7.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.7 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 2.7 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.0_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.0.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.0 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.0 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.1_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.1.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.1 interpreter, they will give an error if they
-are miscompiled.
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.1 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.2_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.2.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.2 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.2 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.3_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.3.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.3 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.3 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.4.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.4 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.4 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.5.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.5 interpreter, they will give an error if they
-are miscompiled.
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.5 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.6_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.6.
+
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.6 interpreter, they are likely to give an error
+when they are miscompiled.

test/grammar-cover/.gitignore

@@ -0,0 +1 @@
+/.python-version

test/grammar-cover/README.md

@@ -0,0 +1 @@
+Code in this directory gets statistics on grammar coverage

test/grammar-cover/convert.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+for VERS in 2{4,5,6,7} 3{2,3,4,5} ; do
+    GRAMMAR_TXT=grammar-${VERS}.txt
+    spark-parser-coverage --max-count 3000 --path spark-grammar-${VERS}.cover > $GRAMMAR_TXT
+done

test/grammar-cover/grammar-all.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+$SHELL ./grammar.sh 2.4 2.5 2.6 2.7 3.2 3.3 3.4 3.5 3.6

test/grammar-cover/grammar.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+# Remake Python grammar statistics
+
+typeset -A ALL_VERS=([2.4]=2.4.6 [2.5]=2.5.6 [2.6]=2.6.9 [2.7]=2.7.14 [3.2]=3.2.6 [3.3]=3.3.6 [3.4]=3.4.8 [3.5]=3.5.5 [3.6]=3.6.4)
+
+if (( $# == 0 )); then
+    echo 1>&2 "usage: $0 two-digit-version"
+    exit 1
+fi
+
+me=${BASH_SOURCE[0]}
+workdir=$(dirname $me)
+cd $workdir
+workdir=$(pwd)
+while [[ -n $1 ]]  ; do
+    SHORT_VERSION=$1; shift
+    LONG_VERSION=${ALL_VERS[$SHORT_VERSION]}
+    if [[ -z ${LONG_VERSION} ]] ; then
+       echo 1>&2 "Version $SHORT_VERSION not known"
+       exit 2
+    fi
+
+    tmpdir=$workdir/../../tmp/grammar-cover
+    COVER_FILE=${tmpdir}/spark-grammar-${SHORT_VERSION}.cover
+    [[ -d  $tmpdir ]] || mkdir $tmpdir
+    cd $workdir/../..
+    if [[ $SHORT_VERSION > 2.5 ]] ; then
+        source ./admin-tools/setup-master.sh
+    else
+        source ./admin-tools/setup-python-2.4.sh
+    fi
+    GRAMMAR_TXT=$tmpdir/grammar-${SHORT_VERSION}.txt
+    (cd ../.. && pyenv local ${LONG_VERSION})
+    cd ./test
+    if [[ -r $COVER_FILE ]]; then
+	rm $COVER_FILE
+    fi
+    if [[ -r $GRAMMAR_TXT ]]; then
+        GRAMMAR_SAVE_TXT=${tmpdir}/grammar-${SHORT_VERSION}-save.txt
+        cp $GRAMMAR_TXT $GRAMMAR_SAVE_TXT
+    fi
+    make grammar-coverage-${SHORT_VERSION};
+    spark-parser-coverage --max-count=3000 --path $COVER_FILE > $GRAMMAR_TXT
+done

test/grammar-cover/run-and-email.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+USER=${USER:-rocky}
+EMAIL=${EMAIL:-rb@dustyfeet.com}
+SUBJECT_PREFIX="grammar cover testing for"
+LOGFILE=/tmp/grammar-cover-$$.log
+/bin/bash ./grammar-all.sh  >$LOGFILE 2>&1
+rc=$?
+if ((rc == 0)); then
+    tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX ok" ${USER}@localhost
+else
+    tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX not ok" ${USER}@localhost
+    tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX not ok" $EMAIL
+fi

test/run-and-email.sh

@@ -1,13 +1,69 @@
-#!/usr/bin/bash
+#!/bin/bash
+
+function displaytime {
+    printf "ran in "
+    local T=$1
+    local D=$((T/60/60/24))
+    local H=$((T/60/60%24))
+    local M=$((T/60%60))
+    local S=$((T%60))
+    (( $D > 0 )) && printf '%d days ' $D
+    (( $H > 0 )) && printf '%d hours ' $H
+    (( $M > 0 )) && printf '%d minutes ' $M
+    (( $D > 0 || $H > 0 || $M > 0 )) && printf 'and '
+    printf '%d seconds\n' $S
+}
+
+PYVERSION=${PYVERSION:-"3.5.5 2.7.14 3.2.6 3.3.7 3.4.8 2.6.9 3.6.4"}
+# PYVERSION=${PYVERSION:-"3.5.5"}
+
+USER=${USER:-rocky}
 EMAIL=${EMAIL:-rb@dustyfeet.com}
-for VERSION in 2.7.14 2.6.9 ; do
-    LOGFILE=/tmp/pyenlib-$VERSION-$$.log
-    python ./test_pyenvlib.py --max 800 --weak-verify --$VERSION  >$LOGFILE 2>&1
-    rc=$?
-    if ((rc == 0)); then
-	tail -v $LOGFILE | mail -s \""$VERSION ok"\" rocky@localhost
+MAX_TESTS=${MAX_TESTS:-800}
+typeset -i RUN_STARTTIME=$(date +%s)
+
+for VERSION in $PYVERSION ; do
+    typeset -i rc=0
+    LOGFILE=/tmp/pyenvlib-$VERSION-$$.log
+
+    if [[ $VERSION == '3.5.5' ]] ; then
+	MAX_TESTS=224
+    elif [[ $VERSION == '3.2.6' ]] ; then
+	MAX_TESTS=700
+    elif [[ $VERSION == '3.6.4' ]] ; then
+	MAX_TESTS=400
     else
-	tail -v $LOGFILE | mail -s \""$VERSION not ok"\" rocky@localhost
-	tail -v $LOGFILE | mail -s \""$VERSION not ok"\" rb@dustyfeet.com
+	MAX_TESTS=800
     fi
+
+    if ! pyenv local $VERSION ; then
+	rc=1
+    else
+      echo Python Version $(pyenv local) > $LOGFILE
+      echo "" >> $LOGFILE
+      typeset -i ALL_FILES_STARTTIME=$(date +%s)
+      python ./test_pyenvlib.py --max ${MAX_TESTS} --weak-verify --$VERSION  >>$LOGFILE 2>&1
+      rc=$?
+
+      echo Python Version $(pyenv local) >> $LOGFILE
+      echo "" >>$LOGFILE
+
+      typeset -i ALL_FILES_ENDTIME=$(date +%s)
+      (( time_diff =  ALL_FILES_ENDTIME - ALL_FILES_STARTTIME))
+      displaytime $time_diff >> $LOGFILE
+    fi
+
+    SUBJECT_PREFIX="pyenv weak verify (max $MAX_TESTS) for"
+    if ((rc == 0)); then
+	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION ok" ${USER}@localhost
+    else
+	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${USER}@localhost
+	tail -v $LOGFILE | mail -s "$SUBJECT_PREFIX $VERSION not ok" ${EMAIL}
+    fi
+    rm .python-version
 done
+
+typeset -i RUN_ENDTIME=$(date +%s)
+(( time_diff =  RUN_ENDTIME - RUN_STARTTIME))
+elapsed_time=$(displaytime $time_diff)
+echo "Run complete $elapsed_time for versions $PYVERSION" | mail -s "pyenv weak verify in $elapsed_time" ${EMAIL}

test/simple_source/bug25/04_try_except_else.py

@@ -0,0 +1,18 @@
+# Bug found in 2.4 test_math.py
+# Bug was turning last try/except/else into try/else
+import math
+def test_exceptions():
+    try:
+        x = math.exp(-1000000000)
+    except:
+        raise RuntimeError
+
+    x = 1
+    try:
+        x = math.sqrt(-1.0)
+    except ValueError:
+        return x
+    else:
+        raise RuntimeError
+
+test_exceptions()

test/simple_source/bug27+/02_assert.py

@@ -0,0 +1,31 @@
+# From 2.7 test_argparse.py
+# Bug was turning assert into an "or raise" statement
+def __call__(arg, dest):
+    try:
+        assert arg == 'spam', 'dest: %s' % dest
+    except:
+        raise
+
+__call__('spam', __file__)
+
+# From python 2.7.14 lib2to3/refactor.py
+# Bug was mangling assert turning if into "or"
+def refactor_doctest(clipped, new):
+    assert clipped, clipped
+    if not new:
+        new += u"\n"
+    return
+
+# From 2.7.14 test_hashlib.py
+# The bug was turning assert into an "if"
+# statement which isn't wrong, but we got the
+# range of the if incorrect. When we have
+# better control flow analysis we can revisit.
+def test_threaded_hashing():
+    for threadnum in xrange(1):
+        result = 1
+        assert result > 0
+        result = 2
+    return result
+
+assert test_threaded_hashing() == 2

test/simple_source/bug27+/05_not_unconditional.py

@@ -1,5 +1,5 @@
 # From 2.7 test_itertools.py
-# Bug was in 2.7 decompiling like the commented out
+# Bug was in 2.7 decompiling the target assignment
 # code below
 from itertools import izip_longest
 for args in [

test/simple_source/bug31/05_abc_test.py

@@ -0,0 +1,18 @@
+# Extracted from Python 3.5 test_abc.py
+# Bug is class having only a single kwarg
+# subclass.
+import abc
+import unittest
+from inspect import isabstract
+
+def test_abstractmethod_integration(self):
+    for abstractthing in [abc.abstractmethod]:
+        class C(metaclass=abc.ABCMeta):
+            @abstractthing
+            def foo(self): pass  # abstract
+            def bar(self): pass  # concrete
+        assert C.__abstractmethods__, {"foo"}
+        assert isabstract(C)
+        pass
+
+test_abstractmethod_integration(None)

test/simple_source/bug33/02_named_and_kwargs.py

@@ -1,5 +1,6 @@
 # Python 3.6 subprocess.py bug
 # Bug is getting params correct: timeout before **kwargs
+import subprocess
 def call(*popenargs, timeout=None, **kwargs):
     return
 
@@ -14,6 +15,9 @@ def subprocess_shell(self, protocol_factory, cmd, *, stdin=subprocess.PIPE,
 # From 3.4 asyncio/locks.py
 # Bug was handling" "value=1, *"
 
+class Semaphore:
+    pass
+
 class BoundedSemaphore(Semaphore):
     def __init__(self, value=1, *, loop=None):
         super().__init__(value, loop=loop)