Get ready for release 3.2.0

Many bugs in 1.4 exist. For a future release

Makefile

@@ -38,6 +38,7 @@ check-3.0 check-3.1 check-3.2 check-3.6:
 	$(MAKE) -C test $@
 
 check-3.7: pytest
+	$(MAKE) -C test check
 
 #:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
 # Skip for now

NEWS

@@ -1,3 +1,16 @@
+uncompyle6 3.2.0 2018-05-19 Rocket Scientist
+
+- Add rudimentary 1.4 support (still a bit buggy)
+- add --tree+ option to show formatting rule, when it is constant
+- Python 2.7.15candidate1 support (via xdis)
+- bug fixes, especially for 3.7 (but 2.7 and 3.6 and others as well)
+
+uncompyle6 3.1.3 2018-04-16
+
+- Add some Python 3.7 rules, such as for handling LOAD_METHOD (not complete)
+- Fix some fragment bugs
+- small doc changes
+
 uncompyle6 3.1.2 2018-04-08 Eastern Orthodox Easter
 
 - Python 3.x subclass and call parsing fixes

README.rst

@@ -52,8 +52,17 @@ You get the idea. This code pulls all of these forks together and
 *moves forward*. There is some serious refactoring and cleanup in this
 code base over those old forks.
 
-This project has the most complete support for Python 3.3 and above
-and the best all-around Python support.
+This demonstrably does the best in decompiling Python across all
+Python versions. And even when there is another project that only
+provides decompilation for subset of Python versions, we generally do
+demonstrably better for those as well.
+
+How can we tell? By taking Python bytecode that comes distributed with
+that version of Python and decompiling these.  Among htose that
+successfully decompile, we can then make sure the resulting programs
+are syntactically correct by running the Python interpreter for that
+bytecode version.  Finally, in cases where the program has a test for
+itself, we can run the check on the decompiled code.
 
 We are serious about testing, and use automated processes to find
 bugs. In the issue trackers for other decompilers, you will find a
@@ -136,26 +145,26 @@ All of the Python decompilers that I have looked at have problems
 decompiling Python's control flow. In some cases we can detect an
 erroneous decompilation and report that.
 
-*Verification* is the process of decompiling bytecode, compiling with
-a Python for that bytecode version, and then comparing the bytecode
-produced by the decompiled/compiled program. Some allowance is made
-for inessential differences. But other semantically equivalent
-differences are not caught. For example ``1 and 0`` is decompiled to
-the equivalent ``0``; remnants of the first true evaluation (1) is
-lost when Python compiles this. When Python next compiles ``0`` the
-resulting code is simpler.
+In older versions of Python it was possible to verify bytecode by
+decompiling bytecode, and then compiling using the Python interpreter
+for that bytecode version. Having done this the bytecode produced
+could be compared with the original bytecode. However as Python's code
+generation got better, this is no longer feasible.
 
-*Weak Verification*
-on the other hand doesn't check bytecode for equivalence but does
-check to see if the resulting decompiled source is a valid Python
-program by running the Python interpreter. Because the Python language
-has changed so much, for best results you should use the same Python
-Version in checking as used in the bytecode.
+There is a kind of *weak verification* that we use that doesn't check
+bytecode for equivalence but does check to see if the resulting
+decompiled source is a valid Python program by running the Python
+interpreter. Because the Python language has changed so much, for best
+results you should use the same Python version in checking as was used
+in creating the bytecode.
 
-Finally, we have automated running the standard Python tests after
-first compiling and decompiling the test program. Results here are a
-bit weak (if not better than most other Python decompilers). But over
-time this will probably get better.
+There are however an interesting class of these programs that is
+readily available give stronger verification: those programs that
+when run check some computation, or even better themselves.
+
+And already Python has a set of programs like this: the test suite
+for the standard library that comes with Python. We have some
+code in `test/stdlib` to facilitate this kind of checking.
 
 Python support is strongest in Python 2 for 2.7 and drops off as you
 get further away from that. Support is also probably pretty good for
@@ -203,7 +212,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for Python 3 is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
 * The HISTORY_ file.

__pkginfo__.py

@@ -56,7 +56,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
-                      'xdis >= 3.8.0, < 3.9.0', 'six']
+                      'xdis >= 3.8.2, < 3.9.0', 'six']
 
 license            = 'GPL3'
 mailing_list       = 'python-debugger@googlegroups.com'

admin-tools/pycdc-runtests.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+# Use pycdc to run our test/bytecode* test suite
+bs=${BASH_SOURCE[0]}
+testdir=$(dirname $bs)/../test
+fulldir=$(readlink -f $testdir)
+cd $fulldir
+for dir in bytecode_* ; do
+    echo ========= $dir ================
+    cd $fulldir/$dir
+    for file in *.pyc; do
+	if ! pycdc $file > /dev/null ; then
+	    echo -----  $dir/$file ------
+	fi
+    done
+done

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.5 3.6.4 2.6.9 3.3.7 2.7.14 3.2.6 3.1.5 3.4.8'
+export PYVERSIONS='3.5.5 3.6.5 2.6.9 3.3.7 2.7.14 3.2.6 3.1.5 3.4.8'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.6.4
+PYTHON_VERSION=3.6.5
 
 # FIXME put some of the below in a common routine
 function finish {

admin-tools/uncompyle2-runtests.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Use pycdc to run our test/bytecode_2.7* test suite
+bs=${BASH_SOURCE[0]}
+topdir=$(dirname $bs)/..
+(cd $topdir && pyenv local 2.7.14)
+testdir=$topdir/test
+fulldir=$(readlink -f $testdir)
+cd $fulldir
+
+for bytecode in bytecode_2.7/*.pyc ; do
+    echo $bytecode
+    uncompyle2 $bytecode > /dev/null
+    echo ================ $bytecode rc: $? ==============
+done
+
+tmpdir=/tmp/test-2.7
+( cd bytecode_2.7_run &&
+      mkdir $tmpdir || true
+      for bytecode in *.pyc ; do
+	  shortname=$(basename $bytecode .pyc)
+	  echo $bytecode
+	  py_file=${tmpdir}/${shortname}.py
+	  typeset -i rc=0
+	  uncompyle2 $bytecode > $py_file
+	  rc=$?
+	  if (( rc == 0 )); then
+	      python $py_file
+	      rc=$?
+	  fi
+	  echo ================ $bytecode rc: $rc ==============
+      done
+)

circle.yml

@@ -6,6 +6,7 @@ machine:
 
 dependencies:
     override:
+        - pip install --upgrade setuptools
         - pip install -e .
         - pip install pytest==3.2.5 hypothesis
 test:

pytest/test_grammar.py

@@ -26,6 +26,9 @@ def test_grammar():
     expect_right_recursive = set([('designList',
                                    ('store', 'DUP_TOP', 'designList'))])
 
+    if PYTHON_VERSION != 3.7:
+        unused_rhs.add('call')
+
     if PYTHON_VERSION > 2.6:
         expect_lhs.add('kvlist')
         expect_lhs.add('kv3')
@@ -41,14 +44,10 @@ def test_grammar():
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
-            unused_rhs.add('call')
             unused_rhs.add("dict_comp")
             unused_rhs.add("classdefdeco1")
-            if PYTHON_VERSION < 3.6:
-                # 3.6 has at least one non-custom call rule
-                # the others don't
-                unused_rhs.add('call')
-                if PYTHON_VERSION == 3.5:
+            if PYTHON_VERSION != 3.6:
+                if PYTHON_VERSION in (3.5, 3.7):
                     expect_right_recursive.add((('l_stmts',
                                                  ('lastl_stmt', 'come_froms', 'l_stmts'))))
                     pass
@@ -61,7 +60,6 @@ def test_grammar():
         pass
     else:
         expect_lhs.add('kwarg')
-        unused_rhs.add('call')
 
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)

test/Makefile

@@ -1,12 +1,13 @@
 PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
-      check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
+      check-bytecode-1  check-bytecode-1.4 check-bytecode-1.5 \
+      check-bytecode-2 check-bytecode-3 \
       check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
       check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
-      check-3.4 check-3.5 check-5.6 5.6 5.8 \
+      check-3.4 check-3.5 check-3.6 check-3.7 check-5.6 5.6 5.8 \
       grammar-coverage-2.5 grammar-coverage-2.6 grammar-coverage-2.7 \
       grammar-coverage-3.1 grammar-coverage-3.2 grammar-coverage-3.3 \
-      grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6
-
+      grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6 \
+      grammar-coverage-3.7
 
 GIT2CL ?= git2cl
 PYTHON ?= python
@@ -63,6 +64,10 @@ check-3.5: check-bytecode
 check-3.6: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
 
+#: Run working tests from Python 3.7
+check-3.7: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify $(COMPILE)
+
 # FIXME
 #: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
 5.8 5.6:
@@ -72,7 +77,7 @@ check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1: check-bytecode-1.5
+check-bytecode-1: check-bytecode-1.4 check-bytecode-1.5
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -89,11 +94,17 @@ check-bytecode-3:
 #: Check deparsing bytecode that works running Python 2 and Python 3
 check-bytecode: check-bytecode-3
 	$(PYTHON) test_pythonlib.py \
+	--bytecode-1.4 --bytecode-1.5 \
+        --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
 	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
         --bytecode-pypy2.7
 
 
+#: Check deparsing bytecode 1.4 only
+check-bytecode-1.4:
+	$(PYTHON) test_pythonlib.py --bytecode-1.4
+
 #: Check deparsing bytecode 1.5 only
 check-bytecode-1.5:
 	$(PYTHON) test_pythonlib.py --bytecode-1.5
@@ -229,6 +240,10 @@ check-bytecode-3.6:
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 
+#: Check deparsing Python 3.7
+check-bytecode-3.7:
+	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)

test/bytecode_2.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.4
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.4 interpreter, they will give an error if they
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 2.4 interpreter, they are likely to give an error when they
 are miscompiled.

test/bytecode_2.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.5.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.5 interpreter, they will give an error if they
-are miscompiled.
+Furthermore the programs here are self-checking: when decompiled and
+then run again in a 2.5 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_2.6_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.6.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.6 interpreter, they will give an error if they
-are miscompiled.
+Furthermore the programs here are self-checking: when decompiled and
+then run again in a 2.6 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_2.7_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.7.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 2.7 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 2.7 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.0_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.0.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.0 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.0 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.1_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.1.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.1 interpreter, they will give an error if they
-are miscompiled.
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.1 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.2_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.2.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.2 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.2 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.3_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.3.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.3 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.3 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.4.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.4 interpreter, they will give an error if they
-are miscompiled.
+Furthermore, the programs here are self-checking: when decompiled and
+then run again in a 3.4 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.5.
 
-Furthrmore the programs here are self-checking: when decompiled and
-then run again in a 3.5 interpreter, they will give an error if they
-are miscompiled.
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.5 interpreter, they are likely to give an error
+when they are miscompiled.

test/bytecode_3.6_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.6.
+
+Furthrmore, the programs here are self-checking: when decompiled and
+then run again in a 3.6 interpreter, they are likely to give an error
+when they are miscompiled.

test/simple_source/branching/02_ifelse_lambda.py

@@ -2,7 +2,8 @@
 # lambda's have to be more or less on a line
 
 f = lambda x: 1 if x<2 else 3
-f(5)
+assert f(3) == 3
+assert f(1) == 1
 
 # If that wasn't enough ...
 # Python will create dead code
@@ -10,10 +11,18 @@ f(5)
 # not to include the else expression
 
 g = lambda: 1 if True else 3
-g()
+assert g() == 1
 
 h = lambda: 1 if False else 3
-h()
+assert h() == 3
 
 # From 2.7 test_builtin
-lambda c: 'a' <= c <= 'z', 'Hello World'
+i = lambda c: 'a' <= c <= 'z', 'Hello World'
+assert i[0]('a') == True
+assert i[0]('A') == False
+
+# Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
+# in addition the the "conditional_lambda" rule
+j = lambda a: False if not a else True
+assert j(True) == True
+assert j(False) == False

test/simple_source/bug26/01_ifelse_listcomp.py

@@ -0,0 +1,4 @@
+# Bug from issue #171:  parsing "if x if a else y" inside a list comprehension on 2.7
+# This is RUNNABLE!
+assert [False, True, True, True, True] == [False if not a else True for a in range(5)]
+assert [True, False, False, False, False] == [False if a else True for a in range(5)]

test/simple_source/bug26/03_complex_and.py

@@ -0,0 +1,17 @@
+# From 2.6 test_datetime.py
+# Bug is in parsing (x is 0 or x is 1) and (y is 5 or y is 2)
+# correctly.
+
+# This code is RUNNABLE!
+result = []
+for y in (1, 2, 10):
+    x = cmp(1, y)
+    if (x is 0 or x is 1) and (y is 5 or y is 2):
+        expected = 10
+    elif y is 2:
+        expected = 2
+    else:
+        expected = 3
+    result.append(expected)
+
+assert result == [10, 2, 3]

test/simple_source/bug27+/03_for_try_raise.py

@@ -0,0 +1,12 @@
+# Code in 2.7 needing rule:
+#   try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK except_handler
+# Generally we need a COME_FROM. But not in the situation below.
+
+for package in [1,2]:
+    try:
+        pass
+    except IndexError:
+        with __file__ as f:
+            pass
+    except:
+        raise

test/simple_source/bug27+/05_try_else.py

@@ -0,0 +1,11 @@
+# From Python 2.7 test_ziplib.py
+# Bug is distinguishing try from try/else.
+def testAFakeZlib(self):
+    try:
+        self.doTest()
+    except ImportError:
+        if self.compression != 3:
+            self.fail("expected test to not raise ImportError")
+    else:
+        if self.compression != 4:
+            self.fail("expected test to raise ImportError")

test/simple_source/bug275/01_while1.py

@@ -0,0 +1,9 @@
+# Issue #173. Bug is that 2.7.5 omits POP_BLOCK in
+# in later 2.7 grammar.
+#  while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK COME_FROM
+#  while1stmt        ::= SETUP_LOOP l_stmts_opt CONTINUE COME_FROM
+# which is included in later code generation
+ms=0
+if ms==1:
+    while 1:
+        pass

test/simple_source/comprehension/05_dict_comp.py

@@ -1,4 +1,7 @@
+# Bug was in dictionary comprehension involving "if not"
 # Issue #162
+#
+# This code is RUNNABLE!
 def x(s):
     return {k: v
             for (k, v) in s

test/simple_source/def/10_classdec.py

@@ -25,3 +25,7 @@ class MyClass(object):
         pass
 
 x = MyClass()
+
+# Try class without parens
+class Feature:
+    pass

test/simple_source/looping/04_while1_while1.py

@@ -19,10 +19,15 @@ while 1:
 while 1:
     if __name__:
         while 1:
-            if y:
+            if __name__:
                 break
             raise RuntimeError
     elif __file__:
         x = 2
     else:
         raise RuntimeError
+
+# Degenerate case. Note: we can't run becase this causes an infinite loop.
+# Suggested in issue #172
+while 1:
+    pass

test/stdlib/runtests.sh

@@ -1,6 +1,13 @@
 #!/bin/bash
 me=${BASH_SOURCE[0]}
 
+typeset -i batch=1
+isatty=$(/usr/bin/tty 2>/dev/null)
+if [[ -n $isatty ]] && [[ "$isatty" != 'not a tty' ]] ; then
+    batch=0
+fi
+
+
 function displaytime {
   local T=$1
   local D=$((T/60/60/24))
@@ -49,6 +56,8 @@ case $PYVERSION in
     2.6)
 	SKIP_TESTS=(
 	    [test_compile.py]=1  # Intermittent - sometimes works and sometimes doesn't
+	    [test_grammar.py]=1  # Need real flow control. "and" in side "or"
+	                         # "and" inside ifelse need to simulatenously work
 	    [test_grp.py]=1      # Long test - might work Control flow?
 	    [test_opcodes.py]=1
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
@@ -66,7 +75,25 @@ case $PYVERSION in
 	    # .pyenv/versions/2.6.9/lib/python2.6/sre_parse.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tabnanny.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tarfile.pyc
-	    )
+
+	    # Not getting set by bach below?
+	    [test_pprint.py]=1
+
+	)
+	if (( batch )) ; then
+	    # Fails in crontab environment?
+	    # Figure out what's up here
+	    SKIP_TESTS[test_aifc.py]=1
+	    SKIP_TESTS[test_array.py]=1
+
+	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+	    SKIP_TESTS[test_base64.py]=1
+
+	    # output indicates expected == output, but this fails anyway.
+	    # Maybe the underlying encoding is subtlely different so it
+	    # looks the same?
+	    SKIP_TESTS[test_pprint.py]=1
+	fi
 	;;
     2.7)
 	SKIP_TESTS=(
@@ -96,13 +123,28 @@ case $PYVERSION in
 	    [test_unicode.py]=1  # Too long to run 11 seconds
 	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
+	    [test_zipimport.py]=1  # We can't distinguish try from try/else yet
         )
+	if (( batch )) ; then
+	    # Fails in crontab environment?
+	    # Figure out what's up here
+	    SKIP_TESTS[test_array.py]=1
+	    SKIP_TESTS[test_ast.py]=1
+	    SKIP_TESTS[test_audioop.py]=1
+
+	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+	    SKIP_TESTS[test_base64.py]=1
+	fi
 	;;
     3.5)
 	SKIP_TESTS=(
 	    [test_decorators.py]=1  # Control flow wrt "if elif"
-	    [test_quopri.py]=1      # Fails in crontab environment?
 	)
+	if (( batch )) ; then
+	    # Fails in crontab environment?
+	    # Figure out what's up here
+	    SKIP_TESTS[test_quopri.py]=1
+	fi
 	;;
 
     3.6)

test/test_pythonlib.py

@@ -78,10 +78,10 @@ for  vers in (2.7, 3.4, 3.5, 3.6):
     test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
     pass
 
-for  vers in (1.5,
+for  vers in (1.4, 1.5,
               2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
               3.0, 3.1, 3.2, 3.3,
-              3.4, 3.5, 3.6, 'pypy3.2', 'pypy2.7'):
+              3.4, 3.5, 3.6, 3.7, 'pypy3.2', 'pypy2.7'):
     bytecode = "bytecode_%s" % vers
     key = "bytecode-%s" % vers
     test_options[key] =  (bytecode, PYC, bytecode, vers)

uncompyle6/bin/uncompile.py

@@ -46,6 +46,7 @@ Debugging Options:
   --asm     -a  include byte-code         (disables --verify)
   --grammar -g  show matching grammar
   --tree    -t  include syntax tree       (disables --verify)
+  --tree++      add template rules to --tree when possible
 
 Extensions of generated files:
   '.pyc_dis' '.pyo_dis'   successfully decompiled (and verified if --verify)
@@ -61,7 +62,7 @@ from uncompyle6.version import VERSION
 
 def usage():
     print("""usage:
-   %s [--verify | --weak-verify ] [--asm] [--tree] [--grammar] [-o <path>] FILE|DIR...
+    %s [--verify | --weak-verify ] [--asm] [--tree[+]] [--grammar] [-o <path>] FILE|DIR...
    %s [--help | -h | --version | -V]
 """  % (program, program))
     sys.exit(1)
@@ -70,8 +71,10 @@ def usage():
 def main_bin():
     if not (sys.version_info[0:2] in ((2, 6), (2, 7),
                                       (3, 1), (3, 2), (3, 3),
-                                      (3, 4), (3, 5), (3, 6))):
-        print('Error: %s requires Python 2.6-2.7, or 3.1-3.6' % program,
+                                      (3, 4), (3, 5), (3, 6),
+                                      (3, 7)
+        )):
+        print('Error: %s requires Python 2.6-2.7, or 3.1-3.7' % program,
               file=sys.stderr)
         sys.exit(-1)
 
@@ -85,8 +88,10 @@ def main_bin():
 
     try:
         opts, files = getopt.getopt(sys.argv[1:], 'hagtdrVo:c:p:',
-                                    'help asm grammar linemaps recurse timestamp tree '
-                                    'fragments verify verify-run version weak-verify '
+                                    'help asm grammar linemaps recurse '
+                                    'timestamp tree tree+ '
+                                    'fragments verify verify-run version '
+                                    'weak-verify '
                                     'showgrammar'.split(' '))
     except getopt.GetoptError as e:
         print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
@@ -116,6 +121,9 @@ def main_bin():
         elif opt in ('--tree', '-t'):
             options['showast'] = True
             options['do_verify'] = None
+        elif opt in ('--tree+',):
+            options['showast'] = 'Full'
+            options['do_verify'] = None
         elif opt in ('--grammar', '-g'):
             options['showgrammar'] = True
         elif opt == '-o':

uncompyle6/parser.py

@@ -721,6 +721,12 @@ def get_python_parser(
                 p = parse36.Python36Parser(debug_parser)
             else:
                 p = parse36.Python36ParserSingle(debug_parser)
+        elif version == 3.7:
+            import uncompyle6.parsers.parse37 as parse37
+            if compile_mode == 'exec':
+                p = parse37.Python37Parser(debug_parser)
+            else:
+                p = parse37.Python37ParserSingle(debug_parser)
         else:
             if compile_mode == 'exec':
                 p = parse3.Python3Parser(debug_parser)

uncompyle6/parsers/parse14.py

@@ -0,0 +1,31 @@
+#  Copyright (c) 2018 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse15 import Python15Parser
+
+class Python14Parser(Python15Parser):
+
+    def p_misc14(self, args):
+        """
+        # Nothing here yet, but will need to add UNARY_CALL, BINARY_CALL,
+        # RAISE_EXCEPTION, BUILD_FUNCTION, UNPACK_ARG, UNPACK_VARARG, LOAD_LOCAL,
+        # SET_FUNC_ARGS, and RESERVE_FAST
+        """
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python14Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+
+class Python14ParserSingle(Python14Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python14Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse15.py

@@ -23,7 +23,7 @@ class Python15Parser(Python21Parser):
         importlist  ::= IMPORT_FROM
         """
 
-class Python15ParserSingle(Python21Parser, PythonParserSingle):
+class Python15ParserSingle(Python15Parser, PythonParserSingle):
     pass
 
 if __name__ == '__main__':

uncompyle6/parsers/parse2.py

@@ -274,12 +274,12 @@ class Python2Parser(PythonParser):
              'LOAD',      'LOOKUP',     'MAKE',      'SETUP',
              'RAISE',     'UNPACK'))
 
-        # Opcode names in the custom_ops_seen set have rules that get added
+        # Opcode names in the custom_seen_ops set have rules that get added
         # unconditionally and the rules are constant. So they need to be done
         # only once and if we see the opcode a second we don't have to consider
         # adding more rules.
         #
-        custom_ops_seen = set()
+        custom_seen_ops = set()
 
         for i, token in enumerate(tokens):
             opname = token.kind
@@ -287,7 +287,7 @@ class Python2Parser(PythonParser):
             # Do a quick breakout before testing potentially
             # each of the dozen or so instruction in if elif.
             if (opname[:opname.find('_')] not in customize_instruction_basenames
-                    or opname in custom_ops_seen):
+                    or opname in custom_seen_ops):
                 continue
 
             opname_base = opname[:opname.rfind('_')]
@@ -355,32 +355,32 @@ class Python2Parser(PythonParser):
                        + 'expr ' * nak + opname
             elif opname == 'CONTINUE_LOOP':
                 self.addRule('continue ::= CONTINUE_LOOP', nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_ATTR':
                 self.addRule('del_stmt ::= expr DELETE_ATTR', nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_DEREF':
                 self.addRule("""
                    stmt           ::= del_deref_stmt
                    del_deref_stmt ::= DELETE_DEREF
                    """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_SUBSCR':
                 self.addRule("""
                     del_stmt ::= delete_subscr
                     delete_subscr ::= expr expr DELETE_SUBSCR
                    """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'GET_ITER':
                 self.addRule("""
                     expr      ::= get_iter
                     attribute ::= expr GET_ITER
                     """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base in ('DUP_TOPX', 'RAISE_VARARGS'):
                 # FIXME: remove these conditions if they are not needed.
@@ -413,18 +413,18 @@ class Python2Parser(PythonParser):
                   expr      ::= attribute
                   attribute ::= expr LOAD_ATTR
                   """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOAD_LISTCOMP':
                 self.addRule("expr ::= listcomp", nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOAD_SETCOMP':
                 self.add_unique_rules([
                     "expr ::= set_comp",
                     "set_comp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1"
                     ], customize)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse3
@@ -433,7 +433,7 @@ class Python2Parser(PythonParser):
                              attribute ::= expr LOOKUP_METHOD
                              """,
                              nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base == 'MAKE_FUNCTION':
                 if i > 0 and tokens[i-1] == 'LOAD_LAMBDA':
@@ -482,7 +482,7 @@ class Python2Parser(PythonParser):
                         "try_except_pypy ::= SETUP_EXCEPT suite_stmts_opt except_handler_pypy",
                         "except_handler_pypy ::= COME_FROM except_stmts END_FINALLY COME_FROM"
                         ], customize)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'SETUP_FINALLY':
                 if 'PyPy' in customize:
@@ -491,13 +491,13 @@ class Python2Parser(PythonParser):
                         tryfinallystmt_pypy ::= SETUP_FINALLY suite_stmts_opt COME_FROM_FINALLY
                                                 suite_stmts_opt END_FINALLY""", nop_func)
 
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base in ('UNPACK_TUPLE', 'UNPACK_SEQUENCE'):
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 rule = 'unpack ::= ' + opname + ' store' * token.attr
             elif opname_base == 'UNPACK_LIST':
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 rule = 'unpack_list ::= ' + opname + ' store' * token.attr
             else:
                 continue