Get ready for release 3.6.0

With grammar reduction tests. Same as tryelsestmt. Lots of stdlib tests work
now. (More remain though.)

.circleci/config.yml

@@ -1,11 +1,3 @@
-# This configuration was automatically generated from a CircleCI 1.0 config.
-# It should include any build commands you had along with commands that CircleCI
-# inferred from your project structure. We strongly recommend you read all the
-# comments in this file to understand the structure of CircleCI 2.0, as the idiom
-# for configuration has changed substantially in 2.0 to allow arbitrary jobs rather
-# than the prescribed lifecycle of 1.0. In general, we recommend using this generated
-# configuration as a reference rather than using it in production, though in most
-# cases it should duplicate the execution of your original 1.0 config.
 version: 2
 jobs:
   build:

.gitignore

@@ -6,6 +6,7 @@
 /.eggs
 /.hypothesis
 /.idea
+/.mypy_cache
 /.pytest_cache
 /.python-version
 /.tox

.travis.yml

@@ -5,6 +5,7 @@ python:
   - '2.7'
   - '3.4'
   - '3.6'
+  - '3.8'
 
 matrix:
   include:

Makefile

@@ -47,8 +47,12 @@ check-3.8:
 # Skip for now
 2.6 5.0 5.3 5.6 5.8:
 
-#:PyPy pypy3-2.4.0 Python 3:
-pypy-3.2 2.4:
+#:PyPy pypy3-2.4.0 Python 3.6.1:
+7.1 pypy-3.2 2.4:
+	$(MAKE) -C test $@
+
+#:PyPy pypy3-2.4.0 Python 3.6.9:
+7.2:
 	$(MAKE) -C test $@
 
 #: Run py.test tests

NEWS.md

@@ -1,3 +1,54 @@
+3.6.0: 2019-12-10 gecko gecko
+=============================
+
+The main focus in this release was more accurate decompilation especially
+for 3.7 and 3.8. However there are some improvments to Python 2.x as well,
+including one of the long-standing problems of detecting the difference between
+`try ... ` and `try else ...`.
+
+With this release we now rebase Python 3.7 on off of a 3.7 base; This
+is also as it is (now) in decompyle3.  This facilitates removing some of the
+cruft in control-flow detection in the 2.7 uncompyle2 base.
+
+Alas, decompilation speed for 3.7 on is greatly increased. Hopefull
+this is temporary (cough, cough) until we can do a static control flow
+pass.
+
+Finally, runing in 3.9-dev is tolerated. We can disassemble, but no parse tables yet.
+
+
+3.5.1 2019-11-17 JNC
+====================
+
+- Pypy 3.3, 3.5, 3.6, and 3.6.9 support
+- bump xdis version to handle newer Python releases, e.g. 2.7.17, 3.5.8, and 3.5.9
+- Improve 3.0 decompilation
+    - no parse errors on stlib bytecode. However accurate translation in
+	  control-flow and and/or detection needs work
+- Remove extraneous iter() in "for" of list comprehension  Fixes #272
+- "for" block without a `POP_BLOCK `and confusing `JUMP_BACK` for `CONTINUE`. Fixes #293
+- Fix unmarshal incompletness detected in Pypy 3.6
+- Miscellaneous bugs fixed
+
+3.5.0 2019-10-12 Stony Brook Ride
+=================================
+
+- Fix fragment bugs
+   * missing `RETURN_LAST` introduced when adding transformation layer
+   * more parent entries on tokens
+- Preliminary support for decompiling Python 1.0, 1.1, 1.2, and 1.6
+   * Newer _xdis_ version needed
+
+3.4.1 2019-10-02
+================
+
+- Correct assert{,2} transforms Fixes #289
+- Fragment parsing fixes:
+     * Wasn't handling 3-arg `%p`
+   	 * fielding error in `code_deparse()`
+- Use newer _xdis_ to better track Python 3.8.0
+
+
 3.4.0 2019-08-24 Totoro
 =======================
 

README.rst

@@ -1,4 +1,4 @@
-|buildstatus| |Latest Version| |Supported Python Versions|
+|buildstatus|  |Pypi Installs| |Latest Version| |Supported Python Versions|
 
 |packagestatus|
 
@@ -13,9 +13,9 @@ Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 1.3 to version
+source code. It accepts bytecodes from Python version 1.0 to version
 3.8, spanning over 24 years of Python releases. We include Dropbox's
-Python 2.5 bytecode and some PyPy bytecode.
+Python 2.5 bytecode and some PyPy bytecodes.
 
 Why this?
 ---------
@@ -46,14 +46,15 @@ not exist and there is just bytecode. Again, my debuggers make use of
 this.
 
 There were (and still are) a number of decompyle, uncompyle,
-uncompyle2, uncompyle3 forks around. Almost all of them come basically
-from the same code base, and (almost?) all of them are no longer
-actively maintained. One was really good at decompiling Python 1.5-2.3
-or so, another really good at Python 2.7, but that only. Another
-handles Python 3.2 only; another patched that and handled only 3.3.
-You get the idea. This code pulls all of these forks together and
-*moves forward*. There is some serious refactoring and cleanup in this
-code base over those old forks.
+uncompyle2, uncompyle3 forks around. Many of them come basically from
+the same code base, and (almost?) all of them are no longer actively
+maintained. One was really good at decompiling Python 1.5-2.3, another
+really good at Python 2.7, but that only. Another handles Python 3.2
+only; another patched that and handled only 3.3.  You get the
+idea. This code pulls all of these forks together and *moves
+forward*. There is some serious refactoring and cleanup in this code
+base over those old forks. Even more experimental refactoring is going
+on in decompile3_.
 
 This demonstrably does the best in decompiling Python across all
 Python versions. And even when there is another project that only
@@ -75,11 +76,11 @@ fixed in the other decompilers.
 Requirements
 ------------
 
-The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
-or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
-python-2.4 branch.  The bytecode files it can read have been tested on
-Python bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and the
-above-mentioned PyPy versions.
+The code here can be run on Python versions 2.6 or later, PyPy 3-2.4
+and later.  Python versions 2.4-2.7 are supported in the python-2.4
+branch.  The bytecode files it can read have been tested on Python
+bytecodes from versions 1.4, 2.1-2.7, and 3.0-3.8 and later PyPy
+versions.
 
 Installation
 ------------
@@ -88,9 +89,9 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -e .  # set up to run from source tree
-                      # Or if you want to install instead
-    python setup.py install # may need sudo
+    $ pip install -e .  # set up to run from source tree
+                        # Or if you want to install instead
+    $ python setup.py install # may need sudo
 
 A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
@@ -186,15 +187,21 @@ they had been rare.  Perhaps to compensate for the additional
 added. So in sum handling control flow by ad hoc means as is currently
 done is worse.
 
-Between Python 3.5, 3.6 and 3.7 there have been major changes to the
-:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.
+Between Python 3.5, 3.6, 3.7 there have been major changes to the
+:code:`MAKE_FUNCTION` and :code:`CALL_FUNCTION` instructions.  Python
+
+Python 3.8 removes :code:`SETUP_LOOP`, :code:`SETUP_EXCEPT`,
+:code:`BREAK_LOOP`, and :code:`CONTINUE_LOOP`, instructions which may
+make control-flow detection harder, lacking the more sophisticated
+control-flow analysis that is planned. We'll see.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
 changes several times within a version.
 
-**We support only released versions, not candidate versions.** Note however
-that the magic of a released version is usually the same as the *last* candidate version prior to release.
+**We support only released versions, not candidate versions.** Note
+however that the magic of a released version is usually the same as
+the *last* candidate version prior to release.
 
 There are also customized Python interpreters, notably Dropbox,
 which use their own magic and encrypt bytcode. With the exception of
@@ -216,7 +223,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://github.com/rocky/python-decompile3 : Much smaller and more modern code, focusing on 3.7+. Changes in that will get migrated back ehre.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
 * https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. There are situations where :code:`uncompyle6` results are incorrect while :code:`uncompyle2` results are not, but more often uncompyle6 is correct when uncompyle2 is not. Because :code:`uncompyle6` adheres to accuracy over idiomatic Python, :code:`uncompyle2` can produce more natural-looking code when it is correct. Currently :code:`uncompyle2` is lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
@@ -225,6 +232,7 @@ See Also
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
+* https://github.com/zrax/pycdc : The README for this C++ code syas it aims to support all versions of Python. It is best for Python versions around 2.7 and 3.3 when the code was initially developed. Accuracy for current versions of Python3 and early versions of Python is lacking. Without major effort, it is unlikely it can be made to support current Python 3. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2g
@@ -233,6 +241,7 @@ See Also
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc
+.. _decompile3: https://github.com/rocky/python-decompile3
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
@@ -244,3 +253,4 @@ See Also
 .. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
 .. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg
 		 :target: https://badge.fury.io/py/uncompyle6
+.. |Pypi Installs| image:: https://pepy.tech/badge/uncompyle6/month

__pkginfo__.py

@@ -58,7 +58,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ["spark-parser >= 1.8.9, < 1.9.0",
-                      "xdis >= 4.0.3, < 4.1.0"]
+                      "xdis >= 4.2.0, < 4.3.0"]
 
 license            = "GPL3"
 mailing_list       = "python-debugger@googlegroups.com"

admin-tools/how-to-make-a-release.md

@@ -56,19 +56,21 @@
 
     $ . ./admin-tools/make-dist-older.sh
     $ git tag release-python-2.4-$VERSION
-
+	$ twine check dist/uncompyle6-$VERSION*
     $ . ./admin-tools/make-dist-newer.sh
-
-Goto https://github.com/rocky/python-uncompyle6/releases
+	$ twine check dist/uncompyle6-$VERSION*
 
 # Upload single package and look at Rst Formating
 
+	$ twine check dist/uncompyle6-${VERSION}*
     $ twine upload dist/uncompyle6-${VERSION}-py3.3.egg
 
 # Upload rest of versions
 
     $ twine upload dist/uncompyle6-${VERSION}*
 
+Goto https://github.com/rocky/python-uncompyle6/releases
+
 # Push tags:
 
     $ git push --tags

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.6.8 3.7.3 2.6.9 3.3.7 2.7.16 3.2.6 3.1.5 3.4.8'
+export PYVERSIONS='3.5.9 3.6.9 2.6.9 3.3.7 2.7.17 3.2.6 3.1.5 3.4.10 3.7.5'

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.6.8
+PYTHON_VERSION=3.7.5
 
 # FIXME put some of the below in a common routine
 function finish {

pytest/test_grammar.py

@@ -1,69 +1,81 @@
 import re
-from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY # , PYTHON_VERSION
+from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY  # , PYTHON_VERSION
 from uncompyle6.parser import get_python_parser, python_parser
 from uncompyle6.scanner import get_scanner
 
-def test_grammar():
 
+def test_grammar():
     def check_tokens(tokens, opcode_set):
         remain_tokens = set(tokens) - opcode_set
-        remain_tokens = set([re.sub(r'_\d+$','', t) for t in remain_tokens])
-        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
-        remain_tokens = set([re.sub('LOAD_CODE$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub(r"_\d+$", "", t) for t in remain_tokens])
+        remain_tokens = set([re.sub("_CONT$", "", t) for t in remain_tokens])
+        remain_tokens = set([re.sub("LOAD_CODE$", "", t) for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
-        assert remain_tokens == set([]), \
-            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar())
+        assert remain_tokens == set([]), "Remaining tokens %s\n====\n%s" % (
+            remain_tokens,
+            p.dump_grammar(),
+        )
 
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
-    (lhs, rhs, tokens,
-     right_recursive, dup_rhs) = p.check_sets()
+    (lhs, rhs, tokens, right_recursive, dup_rhs) = p.check_sets()
 
     # We have custom rules that create the below
-    expect_lhs = set(['pos_arg', 'attribute'])
+    expect_lhs = set(["pos_arg"])
+
     if PYTHON_VERSION < 3.8:
-        expect_lhs.add('get_iter')
+        if PYTHON_VERSION < 3.7:
+            expect_lhs.add("attribute")
 
+        expect_lhs.add("get_iter")
+    else:
+        expect_lhs.add("async_with_as_stmt")
+        expect_lhs.add("async_with_stmt")
 
-    unused_rhs = set(['list', 'mkfunc',
-                      'mklambda',
-                      'unpack',])
+    unused_rhs = set(["list", "mkfunc", "mklambda", "unpack"])
 
-    expect_right_recursive = set([('designList',
-                                   ('store', 'DUP_TOP', 'designList'))])
+    expect_right_recursive = set([("designList", ("store", "DUP_TOP", "designList"))])
 
-    if PYTHON_VERSION < 3.7:
-        unused_rhs.add('call')
+    if PYTHON_VERSION <= 3.7:
+        unused_rhs.add("call")
 
     if PYTHON_VERSION > 2.6:
-        expect_lhs.add('kvlist')
-        expect_lhs.add('kv3')
-        unused_rhs.add('dict')
+        expect_lhs.add("kvlist")
+        expect_lhs.add("kv3")
+        unused_rhs.add("dict")
 
     if PYTHON3:
-        expect_lhs.add('load_genexpr')
+        expect_lhs.add("load_genexpr")
 
-        unused_rhs = unused_rhs.union(set("""
+        unused_rhs = unused_rhs.union(
+            set(
+                """
         except_pop_except generator_exp
-        """.split()))
+        """.split()
+            )
+        )
         if PYTHON_VERSION >= 3.0:
-            expect_lhs.add("annotate_arg")
-            expect_lhs.add("annotate_tuple")
-            unused_rhs.add("mkfunc_annotate")
+            if PYTHON_VERSION < 3.7:
+                expect_lhs.add("annotate_arg")
+                expect_lhs.add("annotate_tuple")
+                unused_rhs.add("mkfunc_annotate")
+
             unused_rhs.add("dict_comp")
             unused_rhs.add("classdefdeco1")
             unused_rhs.add("tryelsestmtl")
             if PYTHON_VERSION >= 3.5:
-                expect_right_recursive.add((('l_stmts',
-                                             ('lastl_stmt', 'come_froms', 'l_stmts'))))
+                expect_right_recursive.add(
+                    (("l_stmts", ("lastl_stmt", "come_froms", "l_stmts")))
+                )
                 pass
             elif 3.0 < PYTHON_VERSION < 3.3:
-                expect_right_recursive.add((('l_stmts',
-                                             ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                expect_right_recursive.add(
+                    (("l_stmts", ("lastl_stmt", "COME_FROM", "l_stmts")))
+                )
                 pass
             pass
         pass
     else:
-        expect_lhs.add('kwarg')
+        expect_lhs.add("kwarg")
 
     assert expect_lhs == set(lhs)
 
@@ -73,9 +85,16 @@ def test_grammar():
 
     assert expect_right_recursive == right_recursive
 
-    expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
-                                ('LOAD_CONST',),
-                                ('JUMP_BACK',), ('JUMP_FORWARD',)])
+    expect_dup_rhs = frozenset(
+        [
+            ("COME_FROM",),
+            ("CONTINUE",),
+            ("JUMP_ABSOLUTE",),
+            ("LOAD_CONST",),
+            ("JUMP_BACK",),
+            ("JUMP_FORWARD",),
+        ]
+    )
     reduced_dup_rhs = dict((k, dup_rhs[k]) for k in dup_rhs if k not in expect_dup_rhs)
     for k in reduced_dup_rhs:
         print(k, reduced_dup_rhs[k])
@@ -83,7 +102,7 @@ def test_grammar():
 
     s = get_scanner(PYTHON_VERSION, IS_PYPY)
     ignore_set = set(
-            """
+        """
             JUMP_BACK CONTINUE
             COME_FROM COME_FROM_EXCEPT
             COME_FROM_EXCEPT_CLAUSE
@@ -92,22 +111,33 @@ def test_grammar():
             LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_STR LOAD_CODE
             LAMBDA_MARKER
             RETURN_END_IF RETURN_END_IF_LAMBDA RETURN_VALUE_LAMBDA RETURN_LAST
-            """.split())
+            """.split()
+    )
     if 2.6 <= PYTHON_VERSION <= 2.7:
         opcode_set = set(s.opc.opname).union(ignore_set)
         if PYTHON_VERSION == 2.6:
             opcode_set.add("THEN")
         check_tokens(tokens, opcode_set)
     elif PYTHON_VERSION == 3.4:
-        ignore_set.add('LOAD_CLASSNAME')
-        ignore_set.add('STORE_LOCALS')
+        ignore_set.add("LOAD_CLASSNAME")
+        ignore_set.add("STORE_LOCALS")
         opcode_set = set(s.opc.opname).union(ignore_set)
         check_tokens(tokens, opcode_set)
 
+
 def test_dup_rule():
     import inspect
-    python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
-                  is_pypy=IS_PYPY,
-                  parser_debug={
-                      'dups': True, 'transition': False, 'reduce': False,
-                      'rules': False, 'errorstack': None, 'context': True})
+
+    python_parser(
+        PYTHON_VERSION,
+        inspect.currentframe().f_code,
+        is_pypy=IS_PYPY,
+        parser_debug={
+            "dups": True,
+            "transition": False,
+            "reduce": False,
+            "rules": False,
+            "errorstack": None,
+            "context": True,
+        },
+    )

setup.py

@@ -4,8 +4,8 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 8)):
-    mess = "Python Release 2.6 .. 3.8 are supported in this code branch."
+if not ((2, 6) <= SYS_VERSION  <= (3, 9)):
+    mess = "Python Release 2.6 .. 3.9 are supported in this code branch."
     if ((2, 4) <= SYS_VERSION <= (2, 7)):
         mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
                  sys.version[0:3])
@@ -32,6 +32,7 @@ setup(
        install_requires   = install_requires,
        license            = license,
        long_description   = long_description,
+       long_description_content_type = "text/x-rst",
        name               = modname,
        packages           = find_packages(),
        py_modules         = py_modules,

test/Makefile

@@ -1,5 +1,6 @@
 PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
-      check-bytecode-1  check-bytecode-1.3 check-bytecode-1.4 check-bytecode-1.5 \
+      check-bytecode-1.0 check-bytecode-1.1 check-bytecode-1.2 check-bytecode-1.3 \
+      check-bytecode-1 check-bytecode-1.4 check-bytecode-1.5 check-bytecode-1.6 \
       check-bytecode-2 check-bytecode-3 check-bytecode-3-short \
       check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
       check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
@@ -85,7 +86,7 @@ check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1: check-bytecode-1.4 check-bytecode-1.5
+check-bytecode-1: check-bytecode-1.0 check-bytecode-1.1 check-bytecode-1.2 check-bytecode-1.3 check-bytecode-1.4 check-bytecode-1.5 check-bytecode-1.6
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -98,8 +99,8 @@ check-bytecode-3:
 	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
 	--bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
 	--bytecode-3.4 --bytecode-3.5 --bytecode-3.6 \
-        --bytecode-3.7 --bytecode-3.8 \
-	--bytecode-pypy3.2
+        --bytecode-3.7  \
+	--bytecode-pypy3.2 --bytecode-pypy3.6 --bytecode-3.8
 
 #: Check deparsing on selected bytecode 3.x
 check-bytecode-3-short:
@@ -109,6 +110,7 @@ check-bytecode-3-short:
 #: Check deparsing bytecode on all Python 2 and Python 3 versions
 check-bytecode: check-bytecode-3
 	$(PYTHON) test_pythonlib.py \
+	--bytecode-1.0 --bytecode-1.1 --bytecode-1.2 \
 	--bytecode-1.3 --bytecode-1.4 --bytecode-1.5 \
         --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
 	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
@@ -122,6 +124,18 @@ check-bytecode-short: check-bytecode-3-short
         --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
 
+#: Check deparsing bytecode 1.0 only
+check-bytecode-1.0:
+	$(PYTHON) test_pythonlib.py --bytecode-1.0
+
+#: Check deparsing bytecode 1.1 only
+check-bytecode-1.1:
+	$(PYTHON) test_pythonlib.py --bytecode-1.1
+
+#: Check deparsing bytecode 1.2 only
+check-bytecode-1.2:
+	$(PYTHON) test_pythonlib.py --bytecode-1.2
+
 #: Check deparsing bytecode 1.3 only
 check-bytecode-1.3:
 	$(PYTHON) test_pythonlib.py --bytecode-1.3
@@ -134,6 +148,10 @@ check-bytecode-1.4:
 check-bytecode-1.5:
 	$(PYTHON) test_pythonlib.py --bytecode-1.5
 
+#: Check deparsing bytecode 1.6 only
+check-bytecode-1.6:
+	$(PYTHON) test_pythonlib.py --bytecode-1.6
+
 #: Check deparsing Python 2.1
 check-bytecode-2.1:
 	$(PYTHON) test_pythonlib.py --bytecode-2.1
@@ -314,8 +332,16 @@ pypy-2.7 5.0 5.3 6.0:
 pypy-3.2 2.4:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.2 --verify
 
-#: PyPy 5.0.x with Python 3.6 ...
+#: PyPy 5.0.x with Python 3.6.1 ...
+check-bytecode-pypy3.6: 7.1
 7.1:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
+
+#: PyPy 5.0.x with Python 3.6.9
+check-bytecode-pypy3.6: 7.2
+7.2:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6-run --verify-run
 	$(PYTHON) test_pythonlib.py --bytecode-pypy3.6 --verify
 
 

test/add-test.py

@@ -2,22 +2,23 @@
 """ Trivial helper program to bytecompile and run an uncompile
 """
 import os, sys, py_compile
+
 assert len(sys.argv) >= 2
 version = sys.version[0:3]
-if sys.argv[1] == '--run':
-    suffix = '_run'
+if sys.argv[1] in ("--run", "-r"):
+    suffix = "_run"
     py_source = sys.argv[2:]
 else:
-    suffix = ''
+    suffix = ""
     py_source = sys.argv[1:]
 
 for path in py_source:
     short = os.path.basename(path)
-    if hasattr(sys, 'pypy_version_info'):
-        cfile =  "bytecode_pypy%s%s/%s" % (version, suffix, short) + 'c'
+    if hasattr(sys, "pypy_version_info"):
+        cfile = "bytecode_pypy%s%s/%s" % (version, suffix, short) + "c"
     else:
-        cfile =  "bytecode_%s%s/%s" % (version, suffix, short) + 'c'
+        cfile = "bytecode_%s%s/%s" % (version, suffix, short) + "c"
     print("byte-compiling %s to %s" % (path, cfile))
     py_compile.compile(path, cfile)
     if isinstance(version, str) or version >= (2, 6, 0):
-        os.system("../bin/uncompyle6 -a -t %s" % cfile)
+        os.system("../bin/uncompyle6 -a -T %s" % cfile)

test/simple_source/bug26/03_complex_and.py

@@ -14,4 +14,4 @@ for y in (1, 2, 10):
         expected = 3
     result.append(expected)
 
-assert result == [10, 2, 3]
+assert result == [3, 2, 3]

test/simple_source/bug30/02_try_except_except.py

@@ -7,3 +7,13 @@ def start_new_thread(function, args, kwargs={}):
         pass
     except:
         args()
+
+# Adapted from 3.0.1 code.py
+# Bug is again JUMP_FORWARD elimination compared
+# to earlier and later Pythons.
+def interact():
+    while 1:
+        try:
+            more = 1
+        except KeyboardInterrupt:
+            more = 0

test/simple_source/bug30/02_while1_if_while1.py

@@ -7,3 +7,8 @@ while 1:
             raise RuntimeError
     else:
         raise RuntimeError
+
+# Adapted from 3.0.1 cgi.py
+def _parseparam(s, end):
+    while end > 0 and s.count(''):
+        end = s.find(';')

test/simple_source/bug30/03_ifelse.py

@@ -0,0 +1,50 @@
+# Adapted from 3.0 base64
+# Problem was handling if/else which
+# needs to be like Python 2.6 (and not like 2.7 or 3.1)
+def main(args, f, func, sys):
+    """Small main program"""
+    if args and args[0] != '-':
+        func(f, sys.stdout.buffer)
+    else:
+        func(sys.stdin.buffer, sys.stdout.buffer)
+
+# From Python 3.0 _markupbase.py.
+#
+# The Problem was in the way "if"s are generated in 3.0 which is sort
+# of like a more optimized Python 2.6, with reduced extraneous jumps,
+# but still 2.6-ish and not 2.7- or 3.1-ish.
+def parse_marked_section(fn, i, rawdata, report=1):
+    if report:
+        j = 1
+        fn(rawdata[i: j])
+    return 10
+
+# From 3.0.1 _abcoll.py
+# Bug was in genexpr_func which doesn't have a JUMP_BACK but
+# in its gen_comp_body, we can use COME_FROM in its place.
+# As above omission of JUMPs is a feature of 3.0 that doesn't
+# seem to be in later versions (or earlier like 2.6).
+def __and__(self, other, Iterable):
+    if not isinstance(other, Iterable):
+        return NotImplemented
+    return self._from_iterable(value for value in other if value in self)
+
+# Adapted from 3.0.1 abc.py
+# Bug was in handling multiple COME_FROMs in return_if_stmt
+def __instancecheck__(subtype, subclass, cls):
+    if subtype:
+        if (cls and subclass):
+            return False
+
+
+# Adapted from 3.0.1 abc.py
+# Bug was rule in "jump_absolute_else" and disallowing
+# "else" to the wrong place.
+
+def _strptime(locale_time, found_zone, time):
+    for tz_values in locale_time:
+        if found_zone:
+            if (time and found_zone):
+                break
+            else:
+                break

test/simple_source/bug30/04_and_del.py

@@ -18,3 +18,13 @@ assert normpath(['.']) == []
 assert normpath(['a', 'b', '..']) == ['a']
 assert normpath(['a', 'b', '', 'c']) == ['a', 'b', 'c']
 assert normpath(['a', 'b', '.', '', 'c', '..']) == ['a', 'b']
+
+# Adapted from 3.0.1 cgi.py
+# Bug was in detecting "or" and "and" due to lack of PUSH/POP_IF instructions.
+def handle(format, html, text):
+    formatter = (format and html) or text
+    return formatter
+
+assert handle(False, False, True)
+assert not handle(True, False, False)
+assert handle(True, True, False)

test/simple_source/bug37/01_assert2.py

@@ -0,0 +1,8 @@
+# Self-checking test.
+# Bug was in if transform not inverting expression
+# This file is RUNNABLE!
+def test_assert2(c):
+    if c < 2:
+        raise SyntaxError('Oops')
+
+test_assert2(5)

test/simple_source/bug37/03_ifelse_chained_for.py

@@ -0,0 +1,24 @@
+# From decompyle3/semantics/customize3.py
+# The bug is handling "for" loop inside the
+# chained compare ifelse
+def n_classdef3(a, b, c, l):
+    r = 1
+    if 3.0 <= a <= 3.2:
+        for n in l:
+            if b:
+                break
+            elif c:
+                r = 2
+            pass
+        pass
+    else:
+        r = 3
+        pass
+    return r
+
+assert n_classdef3(10, True, True, []) == 3
+assert n_classdef3(0, False, True, []) == 3
+assert n_classdef3(3.1, True, True, []) == 1
+assert n_classdef3(3.1, True, False, [1]) == 1
+assert n_classdef3(3.1, True, True, [2]) == 1
+assert n_classdef3(3.1, False, True, [3]) == 2

test/simple_source/bug38/01_extra_iter.py

@@ -0,0 +1,7 @@
+# Adapted from From 3.3 urllib/parse.py
+qs = "https://travis-ci.org/rocky/python-uncompyle6/builds/605260823?utm_medium=notification&utm_source=email"
+expect = ['https://travis-ci.org/rocky/python-uncompyle6/builds/605260823?utm_medium=notification', 'utm_source=email']
+
+# Should visually see that we don't add an extra iter() which is not technically wrong, just
+# unnecessary.
+assert expect == [s2 for s1 in qs.split('&') for s2 in s1.split(';')]

test/simple_source/bug38/01_for_continue.py

@@ -0,0 +1,5 @@
+# Bug is turning a JUMP_BACK for a CONTINUE so for has no JUMP_BACK.
+# Also there is no POP_BLOCK since there isn't anything in the loop.
+# In the future when we have better control flow, we might redo all of this.
+for i in range(2):
+    pass

test/simple_source/bug38/01_named_expr.py

@@ -0,0 +1,5 @@
+(x, y) = "foo", 0
+if x := __name__:
+    y = 1
+assert x == "__main__", "Walrus operator changes value"
+assert y == 1, "Walrus operator branch taken"

test/simple_source/bug38/04_for_no_jump_back.py

@@ -0,0 +1,17 @@
+# from mult_by_const/instruction.py
+# Bug in 3.8 was handling no JUMP_BACK in "for" loop. It is
+# in the "if" instead
+def instruction_sequence_value(instrs, a, b):
+    for instr in instrs:
+        if a:
+            a = 6
+        elif b:
+            return 0
+        pass
+
+    return a
+
+assert instruction_sequence_value([], True, True) == 1
+assert instruction_sequence_value([1], True, True) == 6
+assert instruction_sequence_value([1], False, True) == 0
+assert instruction_sequence_value([1], False, False) == False

test/simple_source/expression/02_true_divide.py

@@ -1,4 +1,4 @@
 # Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE
-from __future__ import division  # needed on 2.6 and 2.7
+from __future__ import division  # needed on 2.2 .. 2.7
 x = len(__file__) / 1
 x /= 1

test/simple_source/expression/14_mixed_expressions.py

@@ -0,0 +1,53 @@
+# Covers a large number of operators
+#
+# This code is RUNNABLE!
+
+import sys
+PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
+
+assert PYTHON_VERSION >= 3.7
+
+# some floats (from 01_float.py)
+
+x = 1e300
+assert 0.0 == x * 0
+assert x * 1e300 == float("inf")
+assert str(float("inf") * 0.0) == "nan"
+assert str(float("-inf") * 0.0) == "nan"
+assert -1e300 * 1e300 == float("-inf")
+
+# Complex (adapted from 02_complex.py)
+y = 5j
+assert y ** 2 == -25
+y **= 3
+assert y == (-0-125j)
+
+
+# Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE (from 02_try_divide.py)
+x = 2
+assert 4 / x == 2
+
+x = 5
+assert x / 2 == 2.5
+x = 3
+x /= 2
+assert x == 1.5
+
+x = 2
+assert 4 // x == 2
+x = 7
+x //= 2
+assert x == 3
+
+x = 3
+assert x % 2 == 1
+x %= 2
+assert x == 1
+
+assert x << 2 == 4
+x <<= 3
+assert x == 8
+
+assert x >> 1 == 4
+x >>= 1
+assert x == 4

test/simple_source/expression/15_mixed_expressions.py

@@ -0,0 +1,55 @@
+# Covers a large number of operators
+#
+# This code is RUNNABLE!
+from __future__ import division  # needed on 2.2 .. 2.7
+
+import sys
+PYTHON_VERSION = sys.version_info[0] + (sys.version_info[1] / 10.0)
+
+# some floats (from 01_float.py)
+
+x = 1e300
+assert 0.0 == x * 0
+assert x * 1e300 == float("inf")
+if PYTHON_VERSION > 2.41:
+    assert str(float("inf") * 0.0) == "nan"
+else:
+    assert str(float("inf") * 0.0) == "-nan"
+assert -1e300 * 1e300 == float("-inf")
+
+# Complex (adapted from 02_complex.py)
+y = 5j
+assert y ** 2 == -25
+y **= 3
+assert y == (-0-125j)
+
+
+# Tests BINARY_TRUE_DIVIDE and INPLACE_TRUE_DIVIDE (from 02_try_divide.py)
+x = 2
+assert 4 / x == 2
+
+if PYTHON_VERSION >= 2.19:
+    x = 5
+    assert x / 2 == 2.5
+    x = 3
+    x /= 2
+    assert x == 1.5
+
+x = 2
+assert 4 // x == 2
+x = 7
+x //= 2
+assert x == 3
+
+x = 3
+assert x % 2 == 1
+x %= 2
+assert x == 1
+
+assert x << 2 == 4
+x <<= 3
+assert x == 8
+
+assert x >> 1 == 4
+x >>= 1
+assert x == 4

test/stdlib/runtests.sh

@@ -34,10 +34,17 @@ case $PYVERSION in
 	SKIP_TESTS=(
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_pwd.py]=1 # Long test - might work? Control flow?
 	    [test_pep247.py]=1 # Long test - might work? Control flow?
+	    [test_pwd.py]=1 # Long test - might work? Control flow?
+	    [test_pyclbr.py]=1 # Investigate
+	    [test_pyexpat.py]=1 # Investigate
 	    [test_queue.py]=1 # Control flow?
-	    # [test_threading.py]=1 # Long test - works
+	    [test_re.py]=1 # try confused with try-else again
+	    [test_socketserver.py]=1 # -- test takes too long to run: 40 seconds
+	    [test_threading.py]=1 # Line numbers are expected to be different
+	    [test_thread.py]=1 # test takes too long to run: 36 seconds
+	    [test_trace.py]=1 # Long test - works
+	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
 	)
 	;;
     2.5)
@@ -47,53 +54,35 @@ case $PYVERSION in
 	    [test_grammar.py]=1    # Too many stmts. Handle large stmts
 	    [test_grp.py]=1        # Long test - might work Control flow?
 	    [test_pdb.py]=1        # Line-number specific
+	    [test_pep247.py]=1     # "assert xxx or .." not detected properly in check_hash_module()
+	    [test_pep352.py]=1     # Investigate
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
+	    [test_pyclbr.py]=1 # Investigate
 	    [test_queue.py]=1 # Control flow?
-	    [test_re.py]=1 # Probably Control flow?
+	    [test_re.py]=1 # Possibly try confused with try-else again
+	    [test_struct.py]=1 # "if and" confused for if .. assert and
+	    [test_sys.py]=1 # try confused with try-else again; in test_current_frames()
+	    [test_tarfile.py]=1  # try confused with try-else again; top-level import
+	    [test_threading.py]=1 # Line numbers are expected to be different
+	    [test_thread.py]=1 # test takes too long to run: 36 seconds
 	    [test_trace.py]=1  # Line numbers are expected to be different
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
 	)
 	;;
     2.6)
 	SKIP_TESTS=(
-	    [test_aepack.py]=1
-	    [test_aifc.py]=1
-	    [test_array.py]=1
-	    [test_audioop.py]=1
-	    [test_base64.py]=1
-	    [test_bigmem.py]=1
-	    [test_binascii.py]=1
-	    [test_builtin.py]=1
-	    [test_bytes.py]=1
-	    [test_class.py]=1
-	    [test_codeccallbacks.py]=1
-	    [test_codecencodings_cn.py]=1
-	    [test_codecencodings_hk.py]=1
-	    [test_codecencodings_jp.py]=1
-	    [test_codecencodings_kr.py]=1
-	    [test_codecencodings_tw.py]=1
-	    [test_codecencodings_cn.py]=1
-	    [test_codecmaps_hk.py]=1
-	    [test_codecmaps_jp.py]=1
-	    [test_codecmaps_kr.py]=1
-	    [test_codecmaps_tw.py]=1
-	    [test_codecs.py]=1
+	    [test_aepack.py]=1 # Fails on its own
+	    [test_codeccallbacks.py]=1  # Fails on its own
 	    [test_compile.py]=1  # Intermittent - sometimes works and sometimes doesn't
-	    [test_cookielib.py]=1
-	    [test_copy.py]=1
-	    [test_decimal.py]=1
-	    [test_descr.py]=1    # Problem in pickle.py?
 	    [test_exceptions.py]=1
-	    [test_extcall.py]=1
-	    [test_float.py]=1
-	    [test_future4.py]=1
-	    [test_generators.py]=1
+	    [test_generators.py]=1 # Investigate
 	    [test_grp.py]=1      # Long test - might work Control flow?
-	    [test_opcodes.py]=1
+	    [test_pep352.py]=1     # Investigate
+	    [test_pprint.py]=1
+	    [test_pyclbr.py]=1 # Investigate
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
-	    [test_re.py]=1 # Probably Control flow?
-	    [test_queue.py]=1 # Control flow?
 	    [test_trace.py]=1  # Line numbers are expected to be different
+	    [test_urllib2net.py]=1 # Fails on its own. May need interactive input
 	    [test_zipfile64.py]=1  # Skip Long test
 	    [test_zlib.py]=1  # Takes too long to run (more than 3 minutes 39 seconds)
 	    # .pyenv/versions/2.6.9/lib/python2.6/lib2to3/refactor.pyc
@@ -105,8 +94,6 @@ case $PYVERSION in
 	    # .pyenv/versions/2.6.9/lib/python2.6/tabnanny.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tarfile.pyc
 
-	    # Not getting set by bach below?
-	    [test_pprint.py]=1
 
 	)
 	if (( batch )) ; then
@@ -133,41 +120,36 @@ case $PYVERSION in
 
 	    [test_capi.py]=1
 	    [test_curses.py]=1  # Possibly fails on its own but not detected
+	    [test test_cmd_line.py]=1 # Takes too long, maybe hangs, or looking for interactive input?
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_doctest.py]=1 # Fails on its own
 	    [test_exceptions.py]=1
 	    [test_format.py]=1  # control flow. uncompyle2 does not have problems here
 	    [test_generators.py]=1  # control flow. uncompyle2 has problem here too
 	    [test_grammar.py]=1     # Too many stmts. Handle large stmts
+	    [test_grp.py]=1     # test takes to long, works interactively though
+	    [test_hashlib.py]=1 # Investigate
 	    [test_io.py]=1 # Test takes too long to run
 	    [test_ioctl.py]=1 # Test takes too long to run
-	    [test_itertools.py]=1 # Fix erroneous reduction to "conditional_true".
-	                          # See test/simple_source/bug27+/05_not_unconditional.py
 	    [test_long.py]=1
 	    [test_long_future.py]=1
 	    [test_math.py]=1
 	    [test_memoryio.py]=1 # FIX
-	    [test_multiprocessing.py]=1 # On uncompyle2, taks 24 secs
-	    [test_pep352.py]=1  # ?
-	    [test_posix.py]=1   # Bug in try-else detection inside test_initgroups()
-	                        # Deal with when we have better flow-control detection
+	    [test_modulefinder.py]=1 # FIX
+	    [test_multiprocessing.py]=1 # On uncompyle2, takes 24 secs
 	    [test_pwd.py]=1     # Takes too long
 	    [test_pty.py]=1
-	    [test_queue.py]=1   # Control flow?
-	    [test_re.py]=1      # Probably Control flow?
 	    [test_runpy.py]=1   # Long and fails on its own
 	    [test_select.py]=1  # Runs okay but takes 11 seconds
 	    [test_socket.py]=1  # Runs ok but takes 22 seconds
 	    [test_subprocess.py]=1 # Runs ok but takes 22 seconds
 	    [test_sys_setprofile.py]=1
 	    [test_sys_settrace.py]=1 # Line numbers are expected to be different
-	    [test_strtod.py]=1 # FIX
 	    [test_traceback.py]=1 # Line numbers change - duh.
-	    [test_types.py]=1     # try/else confusions
 	    [test_unicode.py]=1  # Too long to run 11 seconds
 	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
-	    [test_zipimport.py]=1  # We can't distinguish try from try/else yet
+	    [test_zipimport.py]=1  # FIXME: improper try from try/else ?
         )
 	if (( batch )) ; then
 	    # Fails in crontab environment?

test/test_pyenvlib.py

@@ -43,6 +43,9 @@ TEST_VERSIONS = (
     "pypy3.5-5.7.1-beta",
     "pypy3.5-5.9.0",
     "pypy3.5-6.0.0",
+    "pypy3.6-7.1.0",
+    "pypy3.6-7.1.1",
+    "pypy3.6-7.2.0",
     "native",
 ) + tuple(python_versions)
 
@@ -77,6 +80,9 @@ for vers in TEST_VERSIONS:
     else:
         if vers == "native":
             short_vers = os.path.basename(sys.path[-1])
+            from xdis import PYTHON_VERSION
+            if PYTHON_VERSION > 3.0:
+                PYC = "*.cpython-%d.pyc" % int(PYTHON_VERSION * 10)
             test_options[vers] = (sys.path[-1], PYC, short_vers)
         else:
             short_vers = vers[:3]

test/test_pythonlib.py

@@ -77,9 +77,13 @@ for vers in (2.7, 3.4, 3.5, 3.6):
     pass
 
 for vers in (
+    1.0,
+    1.1,
+    1.2,
     1.3,
     1.4,
     1.5,
+    1.6,
     2.1,
     2.2,
     2.3,

uncompyle6/disas.py

@@ -34,12 +34,11 @@ from __future__ import print_function
 import sys
 from collections import deque
 
-import uncompyle6
-
 from xdis.code import iscode
 from xdis.load import check_object_path, load_module
 from uncompyle6.scanner import get_scanner
 
+
 def disco(version, co, out=None, is_pypy=False):
     """
     diassembles and deparses a given code block 'co'
@@ -49,10 +48,9 @@ def disco(version, co, out=None, is_pypy=False):
 
     # store final output stream for case of error
     real_out = out or sys.stdout
-    print('# Python %s' % version, file=real_out)
+    print("# Python %s" % version, file=real_out)
     if co.co_filename:
-        print('# Embedded file name: %s' % co.co_filename,
-              file=real_out)
+        print("# Embedded file name: %s" % co.co_filename, file=real_out)
 
     scanner = get_scanner(version, is_pypy=is_pypy)
 
@@ -63,10 +61,12 @@ def disco(version, co, out=None, is_pypy=False):
 def disco_loop(disasm, queue, real_out):
     while len(queue) > 0:
         co = queue.popleft()
-        if co.co_name != '<module>':
-            print('\n# %s line %d of %s' %
-                      (co.co_name, co.co_firstlineno, co.co_filename),
-                      file=real_out)
+        if co.co_name != "<module>":
+            print(
+                "\n# %s line %d of %s"
+                % (co.co_name, co.co_firstlineno, co.co_filename),
+                file=real_out,
+            )
         tokens, customize = disasm(co)
         for t in tokens:
             if iscode(t.pattr):
@@ -77,6 +77,7 @@ def disco_loop(disasm, queue, real_out):
             pass
         pass
 
+
 # def disassemble_fp(fp, outstream=None):
 #     """
 #     disassemble Python byte-code from an open file
@@ -90,6 +91,7 @@ def disco_loop(disasm, queue, real_out):
 #         disco(version, co, outstream, is_pypy=is_pypy)
 #     co = None
 
+
 def disassemble_file(filename, outstream=None):
     """
     disassemble Python byte-code file (.pyc)
@@ -98,8 +100,7 @@ def disassemble_file(filename, outstream=None):
     try to find the corresponding compiled object.
     """
     filename = check_object_path(filename)
-    (version, timestamp, magic_int, co, is_pypy,
-     source_size) = load_module(filename)
+    (version, timestamp, magic_int, co, is_pypy, source_size) = load_module(filename)
     if type(co) == list:
         for con in co:
             disco(version, con, outstream)
@@ -107,6 +108,7 @@ def disassemble_file(filename, outstream=None):
         disco(version, co, outstream, is_pypy=is_pypy)
     co = None
 
+
 def _test():
     """Simple test program to disassemble a file."""
     argc = len(sys.argv)

uncompyle6/parser.py

@@ -628,12 +628,30 @@ def get_python_parser(
 
     if version < 3.0:
         if version < 2.2:
+            if version == 1.0:
+                import uncompyle6.parsers.parse10 as parse10
+                if compile_mode == 'exec':
+                    p = parse10.Python10Parser(debug_parser)
+                else:
+                    p = parse10.Python01ParserSingle(debug_parser)
+            elif version == 1.1:
+                import uncompyle6.parsers.parse11 as parse11
+                if compile_mode == 'exec':
+                    p = parse11.Python11Parser(debug_parser)
+                else:
+                    p = parse11.Python11ParserSingle(debug_parser)
+            if version == 1.2:
+                import uncompyle6.parsers.parse12 as parse12
+                if compile_mode == 'exec':
+                    p = parse12.Python12Parser(debug_parser)
+                else:
+                    p = parse12.Python12ParserSingle(debug_parser)
             if version == 1.3:
                 import uncompyle6.parsers.parse13 as parse13
                 if compile_mode == 'exec':
-                    p = parse13.Python14Parser(debug_parser)
+                    p = parse13.Python13Parser(debug_parser)
                 else:
-                    p = parse13.Python14ParserSingle(debug_parser)
+                    p = parse13.Python13ParserSingle(debug_parser)
             elif version == 1.4:
                 import uncompyle6.parsers.parse14 as parse14
                 if compile_mode == 'exec':
@@ -646,6 +664,12 @@ def get_python_parser(
                     p = parse15.Python15Parser(debug_parser)
                 else:
                     p = parse15.Python15ParserSingle(debug_parser)
+            elif version == 1.6:
+                import uncompyle6.parsers.parse16 as parse16
+                if compile_mode == 'exec':
+                    p = parse16.Python16Parser(debug_parser)
+                else:
+                    p = parse16.Python16ParserSingle(debug_parser)
             elif version == 2.1:
                 import uncompyle6.parsers.parse21 as parse21
                 if compile_mode == 'exec':

uncompyle6/parsers/parse10.py

@@ -0,0 +1,25 @@
+#  Copyright (c) 2019 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse11 import Python11Parser
+
+
+class Python10Parser(Python11Parser):
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python11Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+
+class Python10ParserSingle(Python10Parser, PythonParserSingle):
+    pass
+
+
+if __name__ == "__main__":
+    # Check grammar
+    p = Python10Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse11.py

@@ -0,0 +1,25 @@
+#  Copyright (c) 2019 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse12 import Python12Parser
+
+
+class Python11Parser(Python12Parser):
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python12Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+
+class Python11ParserSingle(Python11Parser, PythonParserSingle):
+    pass
+
+
+if __name__ == "__main__":
+    # Check grammar
+    p = Python12Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse12.py

@@ -0,0 +1,25 @@
+#  Copyright (c) 2019 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse13 import Python13Parser
+
+
+class Python12Parser(Python13Parser):
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python12Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+
+class Python12ParserSingle(Python12Parser, PythonParserSingle):
+    pass
+
+
+if __name__ == "__main__":
+    # Check grammar
+    p = Python12Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse16.py

@@ -0,0 +1,46 @@
+#  Copyright (c) 2019 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle, nop_func
+from uncompyle6.parsers.parse21 import Python21Parser
+
+class Python16Parser(Python21Parser):
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python16Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+    def p_import16(self, args):
+        """
+        import      ::= filler IMPORT_NAME STORE_FAST
+        import      ::= filler IMPORT_NAME STORE_NAME
+
+        import_from ::= filler IMPORT_NAME importlist
+        import_from ::= filler filler IMPORT_NAME importlist POP_TOP
+
+        importlist  ::= importlist IMPORT_FROM
+        importlist  ::= IMPORT_FROM
+        """
+
+    def customize_grammar_rules(self, tokens, customize):
+        super(Python16Parser, self).customize_grammar_rules(tokens, customize)
+        for i, token in enumerate(tokens):
+            opname = token.kind
+            opname_base = opname[:opname.rfind('_')]
+
+            if opname_base == 'UNPACK_LIST':
+                self.addRule("store ::= unpack_list", nop_func)
+
+
+
+class Python16ParserSingle(Python16Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python15Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse2.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2018 Rocky Bernstein
+#  Copyright (c) 2015-2019 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #
 #  Copyright (c) 1999 John Aycock
@@ -31,10 +31,10 @@ from uncompyle6.parser import PythonParser, PythonParserSingle, nop_func
 from uncompyle6.parsers.treenode import SyntaxTree
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
 
-class Python2Parser(PythonParser):
 
+class Python2Parser(PythonParser):
     def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
-        super(Python2Parser, self).__init__(SyntaxTree, 'stmts', debug=debug_parser)
+        super(Python2Parser, self).__init__(SyntaxTree, "stmts", debug=debug_parser)
         self.new_rules = set()
 
     def p_print2(self, args):
@@ -52,7 +52,7 @@ class Python2Parser(PythonParser):
         """
 
     def p_print_to(self, args):
-        '''
+        """
         stmt ::= print_to
         stmt ::= print_to_nl
         stmt ::= print_nl_to
@@ -62,10 +62,10 @@ class Python2Parser(PythonParser):
         print_to_items ::= print_to_items print_to_item
         print_to_items ::= print_to_item
         print_to_item ::= DUP_TOP expr ROT_TWO PRINT_ITEM_TO
-        '''
+        """
 
     def p_grammar(self, args):
-        '''
+        """
         sstmt ::= stmt
         sstmt ::= return RETURN_LAST
 
@@ -176,12 +176,12 @@ class Python2Parser(PythonParser):
         jmp_abs ::= JUMP_ABSOLUTE
         jmp_abs ::= JUMP_BACK
         jmp_abs ::= CONTINUE
-        '''
+        """
 
     def p_generator_exp2(self, args):
-        '''
+        """
         generator_exp ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1
-        '''
+        """
 
     def p_expr2(self, args):
         """
@@ -252,25 +252,41 @@ class Python2Parser(PythonParser):
         this.
         """
 
-        if 'PyPy' in customize:
+        if "PyPy" in customize:
             # PyPy-specific customizations
-            self.addRule("""
+            self.addRule(
+                """
                         stmt ::= assign3_pypy
                         stmt ::= assign2_pypy
                         assign3_pypy ::= expr expr expr store store store
                         assign2_pypy ::= expr expr store store
                         list_comp    ::= expr  BUILD_LIST_FROM_ARG for_iter store list_iter
                                          JUMP_BACK
-                        """, nop_func)
+                        """,
+                nop_func,
+            )
 
         # For a rough break out on the first word. This may
         # include instructions that don't need customization,
         # but we'll do a finer check after the rough breakout.
         customize_instruction_basenames = frozenset(
-            ('BUILD',     'CALL',       'CONTINUE',  'DELETE',
-             'DUP',       'EXEC',       'GET',       'JUMP',
-             'LOAD',      'LOOKUP',     'MAKE',      'SETUP',
-             'RAISE',     'UNPACK'))
+            (
+                "BUILD",
+                "CALL",
+                "CONTINUE",
+                "DELETE",
+                "DUP",
+                "EXEC",
+                "GET",
+                "JUMP",
+                "LOAD",
+                "LOOKUP",
+                "MAKE",
+                "SETUP",
+                "RAISE",
+                "UNPACK",
+            )
+        )
 
         # Opcode names in the custom_seen_ops set have rules that get added
         # unconditionally and the rules are constant. So they need to be done
@@ -284,139 +300,191 @@ class Python2Parser(PythonParser):
 
             # Do a quick breakout before testing potentially
             # each of the dozen or so instruction in if elif.
-            if (opname[:opname.find('_')] not in customize_instruction_basenames
-                    or opname in custom_seen_ops):
+            if (
+                opname[: opname.find("_")] not in customize_instruction_basenames
+                or opname in custom_seen_ops
+            ):
                 continue
 
-            opname_base = opname[:opname.rfind('_')]
+            opname_base = opname[: opname.rfind("_")]
 
             # The order of opname listed is roughly sorted below
-            if opname_base in ('BUILD_LIST', 'BUILD_SET', 'BUILD_TUPLE'):
+            if opname_base in ("BUILD_LIST", "BUILD_SET", "BUILD_TUPLE"):
                 # We do this complicated test to speed up parsing of
                 # pathelogically long literals, especially those over 1024.
                 build_count = token.attr
-                thousands = (build_count//1024)
-                thirty32s = ((build_count//32) % 32)
+                thousands = build_count // 1024
+                thirty32s = (build_count // 32) % 32
                 if thirty32s > 0:
-                    rule = "expr32 ::=%s" % (' expr' * 32)
+                    rule = "expr32 ::=%s" % (" expr" * 32)
                     self.add_unique_rule(rule, opname_base, build_count, customize)
                 if thousands > 0:
-                    self.add_unique_rule("expr1024 ::=%s" % (' expr32' * 32),
-                                         opname_base, build_count, customize)
-                collection = opname_base[opname_base.find('_')+1:].lower()
-                rule = (('%s ::= ' % collection) + 'expr1024 '*thousands +
-                        'expr32 '*thirty32s + 'expr '*(build_count % 32) + opname)
-                self.add_unique_rules([
-                    "expr ::= %s" % collection,
-                    rule], customize)
+                    self.add_unique_rule(
+                        "expr1024 ::=%s" % (" expr32" * 32),
+                        opname_base,
+                        build_count,
+                        customize,
+                    )
+                collection = opname_base[opname_base.find("_") + 1 :].lower()
+                rule = (
+                    ("%s ::= " % collection)
+                    + "expr1024 " * thousands
+                    + "expr32 " * thirty32s
+                    + "expr " * (build_count % 32)
+                    + opname
+                )
+                self.add_unique_rules(["expr ::= %s" % collection, rule], customize)
                 continue
-            elif opname_base == 'BUILD_MAP':
-                if opname == 'BUILD_MAP_n':
+            elif opname_base == "BUILD_MAP":
+                if opname == "BUILD_MAP_n":
                     # PyPy sometimes has no count. Sigh.
-                    self.add_unique_rules([
-                        'kvlist_n ::=  kvlist_n kv3',
-                        'kvlist_n ::=',
-                        'dict ::= BUILD_MAP_n kvlist_n',
-                    ], customize)
+                    self.add_unique_rules(
+                        [
+                            "kvlist_n ::=  kvlist_n kv3",
+                            "kvlist_n ::=",
+                            "dict ::= BUILD_MAP_n kvlist_n",
+                        ],
+                        customize,
+                    )
                     if self.version >= 2.7:
                         self.add_unique_rule(
-                            'dict_comp_func ::= BUILD_MAP_n LOAD_FAST FOR_ITER store '
-                            'comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST',
-                            'dict_comp_func', 0, customize)
+                            "dict_comp_func ::= BUILD_MAP_n LOAD_FAST FOR_ITER store "
+                            "comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST",
+                            "dict_comp_func",
+                            0,
+                            customize,
+                        )
 
                 else:
-                    kvlist_n =  ' kv3' * token.attr
+                    kvlist_n = " kv3" * token.attr
                     rule = "dict ::= %s%s" % (opname, kvlist_n)
                     self.addRule(rule, nop_func)
                 continue
-            elif opname_base == 'BUILD_SLICE':
-                slice_num  = token.attr
+            elif opname_base == "BUILD_SLICE":
+                slice_num = token.attr
                 if slice_num == 2:
-                     self.add_unique_rules([
-                        'expr ::= build_slice2',
-                        'build_slice2 ::= expr expr BUILD_SLICE_2'
-                        ], customize)
+                    self.add_unique_rules(
+                        [
+                            "expr ::= build_slice2",
+                            "build_slice2 ::= expr expr BUILD_SLICE_2",
+                        ],
+                        customize,
+                    )
                 else:
-                    assert slice_num == 3, ("BUILD_SLICE value must be 2 or 3; is %s" %
-                                            slice_num)
-                    self.add_unique_rules([
-                        'expr ::= build_slice3',
-                        'build_slice3 ::= expr expr expr BUILD_SLICE_3',
-                        ], customize)
+                    assert slice_num == 3, (
+                        "BUILD_SLICE value must be 2 or 3; is %s" % slice_num
+                    )
+                    self.add_unique_rules(
+                        [
+                            "expr ::= build_slice3",
+                            "build_slice3 ::= expr expr expr BUILD_SLICE_3",
+                        ],
+                        customize,
+                    )
                 continue
-            elif opname_base in ('CALL_FUNCTION', 'CALL_FUNCTION_VAR',
-                                 'CALL_FUNCTION_VAR_KW', 'CALL_FUNCTION_KW'):
+            elif opname_base in (
+                "CALL_FUNCTION",
+                "CALL_FUNCTION_VAR",
+                "CALL_FUNCTION_VAR_KW",
+                "CALL_FUNCTION_KW",
+            ):
 
                 args_pos, args_kw = self.get_pos_kw(token)
 
                 # number of apply equiv arguments:
-                nak = ( len(opname_base)-len('CALL_FUNCTION') ) // 3
-                rule = 'call ::= expr ' + 'expr '*args_pos + 'kwarg '*args_kw \
-                       + 'expr ' * nak + opname
-            elif opname_base == 'CALL_METHOD':
+                nak = (len(opname_base) - len("CALL_FUNCTION")) // 3
+                rule = (
+                    "call ::= expr "
+                    + "expr " * args_pos
+                    + "kwarg " * args_kw
+                    + "expr " * nak
+                    + opname
+                )
+            elif opname_base == "CALL_METHOD":
                 # PyPy only - DRY with parse3
 
                 args_pos, args_kw = self.get_pos_kw(token)
 
                 # number of apply equiv arguments:
-                nak = ( len(opname_base)-len('CALL_METHOD') ) // 3
-                rule = 'call ::= expr ' + 'expr '*args_pos + 'kwarg '*args_kw \
-                       + 'expr ' * nak + opname
-            elif opname == 'CONTINUE_LOOP':
-                self.addRule('continue ::= CONTINUE_LOOP', nop_func)
+                nak = (len(opname_base) - len("CALL_METHOD")) // 3
+                rule = (
+                    "call ::= expr "
+                    + "expr " * args_pos
+                    + "kwarg " * args_kw
+                    + "expr " * nak
+                    + opname
+                )
+            elif opname == "CONTINUE_LOOP":
+                self.addRule("continue ::= CONTINUE_LOOP", nop_func)
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'DELETE_ATTR':
-                self.addRule('del_stmt ::= expr DELETE_ATTR', nop_func)
+            elif opname == "DELETE_ATTR":
+                self.addRule("del_stmt ::= expr DELETE_ATTR", nop_func)
                 custom_seen_ops.add(opname)
                 continue
-            elif opname.startswith('DELETE_SLICE'):
-                self.addRule("""
+            elif opname.startswith("DELETE_SLICE"):
+                self.addRule(
+                    """
                 del_expr ::= expr
                 del_stmt ::= del_expr DELETE_SLICE+0
                 del_stmt ::= del_expr del_expr DELETE_SLICE+1
                 del_stmt ::= del_expr del_expr DELETE_SLICE+2
                 del_stmt ::= del_expr del_expr del_expr DELETE_SLICE+3
-                """, nop_func)
+                """,
+                    nop_func,
+                )
                 custom_seen_ops.add(opname)
-                self.check_reduce['del_expr'] = 'AST'
+                self.check_reduce["del_expr"] = "AST"
                 continue
-            elif opname == 'DELETE_DEREF':
-                self.addRule("""
+            elif opname == "DELETE_DEREF":
+                self.addRule(
+                    """
                    stmt           ::= del_deref_stmt
                    del_deref_stmt ::= DELETE_DEREF
-                   """, nop_func)
+                   """,
+                    nop_func,
+                )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'DELETE_SUBSCR':
-                self.addRule("""
+            elif opname == "DELETE_SUBSCR":
+                self.addRule(
+                    """
                     del_stmt ::= delete_subscript
                     delete_subscript ::= expr expr DELETE_SUBSCR
-                   """, nop_func)
-                self.check_reduce['delete_subscript'] = 'AST'
+                   """,
+                    nop_func,
+                )
+                self.check_reduce["delete_subscript"] = "AST"
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'GET_ITER':
-                self.addRule("""
+            elif opname == "GET_ITER":
+                self.addRule(
+                    """
                     expr      ::= get_iter
                     attribute ::= expr GET_ITER
-                    """, nop_func)
+                    """,
+                    nop_func,
+                )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname_base in ('DUP_TOPX', 'RAISE_VARARGS'):
+            elif opname_base in ("DUP_TOPX", "RAISE_VARARGS"):
                 # FIXME: remove these conditions if they are not needed.
                 # no longer need to add a rule
                 continue
-            elif opname == 'EXEC_STMT':
-                self.addRule("""
+            elif opname == "EXEC_STMT":
+                self.addRule(
+                    """
                     stmt      ::= exec_stmt
                     exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
                     exec_stmt ::= expr exprlist EXEC_STMT
                     exprlist  ::= expr+
-                    """, nop_func)
+                    """,
+                    nop_func,
+                )
                 continue
-            elif opname == 'JUMP_IF_NOT_DEBUG':
-                self.addRule("""
+            elif opname == "JUMP_IF_NOT_DEBUG":
+                self.addRule(
+                    """
                     jmp_true_false ::= POP_JUMP_IF_TRUE
                     jmp_true_false ::= POP_JUMP_IF_FALSE
                     stmt ::= assert_pypy
@@ -426,107 +494,155 @@ class Python2Parser(PythonParser):
                     assert2_pypy ::= JUMP_IF_NOT_DEBUG assert_expr jmp_true_false
                                      LOAD_ASSERT expr CALL_FUNCTION_1
                                      RAISE_VARARGS_1 COME_FROM
-                     """, nop_func)
+                     """,
+                    nop_func,
+                )
                 continue
-            elif opname == 'LOAD_ATTR':
-                self.addRule("""
+            elif opname == "LOAD_ATTR":
+                self.addRule(
+                    """
                   expr      ::= attribute
                   attribute ::= expr LOAD_ATTR
-                  """, nop_func)
+                  """,
+                    nop_func,
+                )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'LOAD_LISTCOMP':
+            elif opname == "LOAD_LISTCOMP":
                 self.addRule("expr ::= listcomp", nop_func)
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'LOAD_SETCOMP':
-                self.add_unique_rules([
-                    "expr ::= set_comp",
-                    "set_comp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1"
-                    ], customize)
+            elif opname == "LOAD_SETCOMP":
+                self.add_unique_rules(
+                    [
+                        "expr ::= set_comp",
+                        "set_comp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1",
+                    ],
+                    customize,
+                )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'LOOKUP_METHOD':
+            elif opname == "LOOKUP_METHOD":
                 # A PyPy speciality - DRY with parse3
-                self.addRule("""
+                self.addRule(
+                    """
                              expr      ::= attribute
                              attribute ::= expr LOOKUP_METHOD
                              """,
-                             nop_func)
+                    nop_func,
+                )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname_base == 'MAKE_FUNCTION':
-                if i > 0 and tokens[i-1] == 'LOAD_LAMBDA':
-                    self.addRule('mklambda ::= %s LOAD_LAMBDA %s' %
-                                 ('pos_arg ' * token.attr, opname), nop_func)
-                rule = 'mkfunc ::= %s LOAD_CODE %s' % ('expr ' * token.attr, opname)
-            elif opname_base == 'MAKE_CLOSURE':
+            elif opname_base == "MAKE_FUNCTION":
+                if i > 0 and tokens[i - 1] == "LOAD_LAMBDA":
+                    self.addRule(
+                        "mklambda ::= %s LOAD_LAMBDA %s"
+                        % ("pos_arg " * token.attr, opname),
+                        nop_func,
+                    )
+                rule = "mkfunc ::= %s LOAD_CODE %s" % ("expr " * token.attr, opname)
+            elif opname_base == "MAKE_CLOSURE":
                 # FIXME: use add_unique_rules to tidy this up.
-                if i > 0 and tokens[i-1] == 'LOAD_LAMBDA':
-                    self.addRule('mklambda ::= %s load_closure LOAD_LAMBDA %s' %
-                                 ('expr ' * token.attr, opname),  nop_func)
+                if i > 0 and tokens[i - 1] == "LOAD_LAMBDA":
+                    self.addRule(
+                        "mklambda ::= %s load_closure LOAD_LAMBDA %s"
+                        % ("expr " * token.attr, opname),
+                        nop_func,
+                    )
                 if i > 0:
-                    prev_tok = tokens[i-1]
-                    if prev_tok == 'LOAD_GENEXPR':
-                        self.add_unique_rules([
-                            ('generator_exp ::= %s load_closure LOAD_GENEXPR %s expr'
-                                 ' GET_ITER CALL_FUNCTION_1' %
-                            ('expr ' * token.attr, opname))], customize)
+                    prev_tok = tokens[i - 1]
+                    if prev_tok == "LOAD_GENEXPR":
+                        self.add_unique_rules(
+                            [
+                                (
+                                    "generator_exp ::= %s load_closure LOAD_GENEXPR %s expr"
+                                    " GET_ITER CALL_FUNCTION_1"
+                                    % ("expr " * token.attr, opname)
+                                )
+                            ],
+                            customize,
+                        )
                         pass
-                self.add_unique_rules([
-                    ('mkfunc ::= %s load_closure LOAD_CODE %s' %
-                     ('expr ' * token.attr, opname))], customize)
+                self.add_unique_rules(
+                    [
+                        (
+                            "mkfunc ::= %s load_closure LOAD_CODE %s"
+                            % ("expr " * token.attr, opname)
+                        )
+                    ],
+                    customize,
+                )
 
                 if self.version >= 2.7:
                     if i > 0:
-                        prev_tok = tokens[i-1]
-                        if prev_tok == 'LOAD_DICTCOMP':
-                            self.add_unique_rules([
-                                ('dict_comp ::= %s load_closure LOAD_DICTCOMP %s expr'
-                                 ' GET_ITER CALL_FUNCTION_1' %
-                                ('expr ' * token.attr, opname))], customize)
-                        elif prev_tok == 'LOAD_SETCOMP':
-                            self.add_unique_rules([
-                                "expr ::= set_comp",
-                                ('set_comp ::= %s load_closure LOAD_SETCOMP %s expr'
-                                ' GET_ITER CALL_FUNCTION_1' %
-                                ('expr ' * token.attr, opname))
-                                ], customize)
+                        prev_tok = tokens[i - 1]
+                        if prev_tok == "LOAD_DICTCOMP":
+                            self.add_unique_rules(
+                                [
+                                    (
+                                        "dict_comp ::= %s load_closure LOAD_DICTCOMP %s expr"
+                                        " GET_ITER CALL_FUNCTION_1"
+                                        % ("expr " * token.attr, opname)
+                                    )
+                                ],
+                                customize,
+                            )
+                        elif prev_tok == "LOAD_SETCOMP":
+                            self.add_unique_rules(
+                                [
+                                    "expr ::= set_comp",
+                                    (
+                                        "set_comp ::= %s load_closure LOAD_SETCOMP %s expr"
+                                        " GET_ITER CALL_FUNCTION_1"
+                                        % ("expr " * token.attr, opname)
+                                    ),
+                                ],
+                                customize,
+                            )
                         pass
                     pass
                 continue
-            elif opname == 'SETUP_EXCEPT':
-                if 'PyPy' in customize:
-                    self.add_unique_rules([
-                        "stmt ::= try_except_pypy",
-                        "try_except_pypy ::= SETUP_EXCEPT suite_stmts_opt except_handler_pypy",
-                        "except_handler_pypy ::= COME_FROM except_stmts END_FINALLY COME_FROM"
-                        ], customize)
+            elif opname == "SETUP_EXCEPT":
+                if "PyPy" in customize:
+                    self.add_unique_rules(
+                        [
+                            "stmt ::= try_except_pypy",
+                            "try_except_pypy ::= SETUP_EXCEPT suite_stmts_opt except_handler_pypy",
+                            "except_handler_pypy ::= COME_FROM except_stmts END_FINALLY COME_FROM",
+                        ],
+                        customize,
+                    )
                 custom_seen_ops.add(opname)
                 continue
-            elif opname == 'SETUP_FINALLY':
-                if 'PyPy' in customize:
-                    self.addRule("""
+            elif opname == "SETUP_FINALLY":
+                if "PyPy" in customize:
+                    self.addRule(
+                        """
                         stmt ::= tryfinallystmt_pypy
                         tryfinallystmt_pypy ::= SETUP_FINALLY suite_stmts_opt COME_FROM_FINALLY
-                                                suite_stmts_opt END_FINALLY""", nop_func)
+                                                suite_stmts_opt END_FINALLY""",
+                        nop_func,
+                    )
 
                 custom_seen_ops.add(opname)
                 continue
-            elif opname_base in ('UNPACK_TUPLE', 'UNPACK_SEQUENCE'):
+            elif opname_base in ("UNPACK_TUPLE", "UNPACK_SEQUENCE"):
                 custom_seen_ops.add(opname)
-                rule = 'unpack ::= ' + opname + ' store' * token.attr
-            elif opname_base == 'UNPACK_LIST':
+                rule = "unpack ::= " + opname + " store" * token.attr
+            elif opname_base == "UNPACK_LIST":
                 custom_seen_ops.add(opname)
-                rule = 'unpack_list ::= ' + opname + ' store' * token.attr
+                rule = "unpack_list ::= " + opname + " store" * token.attr
             else:
                 continue
             self.addRule(rule, nop_func)
             pass
 
-        self.check_reduce['raise_stmt1'] = 'tokens'
-        self.check_reduce['aug_assign2'] = 'AST'
-        self.check_reduce['or'] = 'AST'
+        self.check_reduce["raise_stmt1"] = "tokens"
+        self.check_reduce["assert_expr_and"] = "AST"
+        self.check_reduce["tryelsestmt"] = "AST"
+        self.check_reduce["tryelsestmtl"] = "AST"
+        self.check_reduce["aug_assign2"] = "AST"
+        self.check_reduce["or"] = "AST"
         # self.check_reduce['_stmts'] = 'AST'
 
         # Dead code testing...
@@ -541,24 +657,61 @@ class Python2Parser(PythonParser):
         # Dead code testing...
         # if lhs == 'while1elsestmt':
         #     from trepan.api import debug; debug()
-        if lhs in ('aug_assign1', 'aug_assign2') and ast[0] and ast[0][0] in ('and', 'or'):
+        if (
+            lhs in ("aug_assign1", "aug_assign2")
+            and ast[0]
+            and ast[0][0] in ("and", "or")
+        ):
             return True
-        elif lhs in ('raise_stmt1',):
+        elif lhs == "assert_expr_and":
+            jmp_false = ast[1]
+            jump_target = jmp_false[0].attr
+            return jump_target > tokens[last].off2int()
+        elif lhs in ("raise_stmt1",):
             # We will assume 'LOAD_ASSERT' will be handled by an assert grammar rule
-            return (tokens[first] == 'LOAD_ASSERT' and (last >= len(tokens)))
-        elif rule == ('or', ('expr', 'jmp_true', 'expr', '\\e_come_from_opt')):
+            return tokens[first] == "LOAD_ASSERT" and (last >= len(tokens))
+        elif rule == ("or", ("expr", "jmp_true", "expr", "\\e_come_from_opt")):
             expr2 = ast[2]
-            return expr2 == 'expr' and expr2[0] == 'LOAD_ASSERT'
-        elif lhs in ('delete_subscript', 'del_expr'):
+            return expr2 == "expr" and expr2[0] == "LOAD_ASSERT"
+        elif lhs in ("delete_subscript", "del_expr"):
             op = ast[0][0]
-            return op.kind in ('and', 'or')
+            return op.kind in ("and", "or")
+        elif lhs in ("tryelsestmt", "tryelsestmtl"):
+            # Check the end of the except handler that there isn't a jump from
+            # inside the except handler to the end. If that happens
+            # then this is a "try" with no "else".
+            except_handler = ast[3]
+            if except_handler == "except_handler":
+
+                come_from = except_handler[-1]
+                # We only care about the *first* come_from because that is the
+                # the innermost one. So if the "tryelse" is invalid (should be a "try")
+                # ti will be invalid here.
+                if come_from == "COME_FROM":
+                    first_come_from = except_handler[-1]
+                else:
+                    assert come_from == "come_froms"
+                    first_come_from = come_from[0]
+                leading_jump = except_handler[0]
+
+                # We really don't care that this is a jump per-se. But
+                # we could also check that this jumps to the end of the except if
+                # desired.
+                if isinstance(leading_jump, SyntaxTree):
+                    except_handler_first_offset = leading_jump.first_child().off2int()
+                else:
+                    except_handler_first_offset = leading_jump.off2int()
+                return first_come_from.attr > except_handler_first_offset
+
 
         return False
 
+
 class Python2ParserSingle(Python2Parser, PythonParserSingle):
     pass
 
-if __name__ == '__main__':
+
+if __name__ == "__main__":
     # Check grammar
     p = Python2Parser()
     p.check_grammar()

uncompyle6/parsers/parse25.py

@@ -25,11 +25,17 @@ class Python25Parser(Python26Parser):
         setupwithas ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0
                         setup_finally
         # opcode SETUP_WITH
-        setupwith ::= DUP_TOP LOAD_ATTR STORE_NAME LOAD_ATTR CALL_FUNCTION_0 POP_TOP
-        withstmt ::= expr setupwith SETUP_FINALLY suite_stmts_opt
-                     POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+        setupwith ::= DUP_TOP LOAD_ATTR store LOAD_ATTR CALL_FUNCTION_0 POP_TOP
+        withstmt  ::= expr setupwith SETUP_FINALLY suite_stmts_opt
+                      POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+
+        # Semantic actions want store to be at index 2
+        withasstmt ::= expr setupwithas store suite_stmts_opt
+                       POP_BLOCK LOAD_CONST COME_FROM with_cleanup
+
 
         store ::= STORE_NAME
+        store ::= STORE_FAST
 
         # tryelsetmtl doesn't need COME_FROM since the jump might not
         # be the the join point at the end of the "try" but instead back to the

uncompyle6/parsers/parse26.py

@@ -254,7 +254,7 @@ class Python26Parser(Python2Parser):
                          POP_TOP jb_pb_come_from
 
         generator_exp ::= LOAD_GENEXPR MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1 COME_FROM
-        list_if ::= list_if ::= expr jmp_false_then list_iter
+        list_if ::= expr jmp_false_then list_iter
         '''
 
     def p_ret26(self, args):
@@ -467,7 +467,7 @@ if __name__ == '__main__':
     p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 2.6:
-        lhs, rhs, tokens, right_recursive = p.check_sets()
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse27.py

@@ -41,8 +41,6 @@ class Python27Parser(Python2Parser):
         comp_body ::= set_comp_body
         comp_for ::= expr for_iter store comp_iter JUMP_BACK
 
-        comp_iter     ::= comp_body
-
         dict_comp_body ::= expr expr MAP_ADD
         set_comp_body ::= expr SET_ADD
 
@@ -216,13 +214,17 @@ class Python27Parser(Python2Parser):
 
 
         super(Python27Parser, self).customize_grammar_rules(tokens, customize)
-        self.check_reduce['and'] = 'AST'
-        # self.check_reduce['or'] = 'AST'
-        self.check_reduce['raise_stmt1'] = 'AST'
-        self.check_reduce['list_if_not'] = 'AST'
-        self.check_reduce['list_if'] = 'AST'
-        self.check_reduce['if_expr_true'] = 'tokens'
-        self.check_reduce['whilestmt'] = 'tokens'
+        self.check_reduce["and"] = "AST"
+        self.check_reduce["conditional"] = "AST"
+        # self.check_reduce["or"] = "AST"
+        self.check_reduce["raise_stmt1"] = "AST"
+        self.check_reduce["iflaststmtl"] = "AST"
+        self.check_reduce["list_if_not"] = "AST"
+        self.check_reduce["list_if"] = "AST"
+        self.check_reduce["comp_if"] = "AST"
+        self.check_reduce["if_expr_true"] = "tokens"
+        self.check_reduce["whilestmt"] = "tokens"
+
         return
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
@@ -233,54 +235,99 @@ class Python27Parser(Python2Parser):
         if invalid:
             return invalid
 
-        if rule == ('and', ('expr', 'jmp_false', 'expr', '\\e_come_from_opt')):
-            # If the instruction after the instructions formin "and"  is an "YIELD_VALUE"
+        if rule == ("and", ("expr", "jmp_false", "expr", "\\e_come_from_opt")):
+            # If the instruction after the instructions forming the "and"  is an "YIELD_VALUE"
             # then this is probably an "if" inside a comprehension.
-            if tokens[last] == 'YIELD_VALUE':
+            if tokens[last] == "YIELD_VALUE":
                 # Note: We might also consider testing last+1 being "POP_TOP"
                 return True
 
+            # Test that jump_false jump somewhere beyond the end of the "and"
+            # it might not be exactly the end of the "and" because this and can
+            # be a part of a larger condition. Oddly in 2.7 there doesn't seem to be
+            # an optimization where the "and" jump_false is back to a loop.
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                while (first < last and isinstance(tokens[last].offset, str)):
+                    last -= 1
+                if jmp_false[0].attr < tokens[last].offset:
+                    return True
+
             # Test that jmp_false jumps to the end of "and"
             # or that it jumps to the same place as the end of "and"
             jmp_false = ast[1][0]
             jmp_target = jmp_false.offset + jmp_false.attr + 3
             return not (jmp_target == tokens[last].offset or
                         tokens[last].pattr == jmp_false.pattr)
-        elif rule[0] == ('raise_stmt1'):
-            return ast[0] == 'expr' and ast[0][0] == 'or'
-        elif rule[0] in ('assert', 'assert2'):
+        elif rule == ("comp_if", ("expr", "jmp_false", "comp_iter")):
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                return tokens[first].offset < jmp_false[0].attr < tokens[last].offset
+            pass
+        elif (rule[0], rule[1][0:5]) == (
+                "conditional",
+                ("expr", "jmp_false", "expr", "JUMP_ABSOLUTE", "expr")):
+            jmp_false = ast[1]
+            if jmp_false[0] == "POP_JUMP_IF_FALSE":
+                else_instr = ast[4].first_child()
+                if jmp_false[0].attr != else_instr.offset:
+                    return True
+                end_offset = ast[3].attr
+                return end_offset < tokens[last].offset
+            pass
+        elif rule[0] == ("raise_stmt1"):
+            return ast[0] == "expr" and ast[0][0] == "or"
+        elif rule[0] in ("assert", "assert2"):
             jump_inst = ast[1][0]
             jump_target = jump_inst.attr
             return not (last >= len(tokens)
                         or jump_target == tokens[last].offset
                         or jump_target == next_offset(ast[-1].op, ast[-1].opc, ast[-1].offset))
-        elif rule == ('list_if_not', ('expr', 'jmp_true', 'list_iter')):
+        elif rule == ("iflaststmtl", ("testexpr", "c_stmts")):
+            testexpr = ast[0]
+            if testexpr[0] == "testfalse":
+                testfalse = testexpr[0]
+                if testfalse[1] == "jmp_false":
+                    jmp_false = testfalse[1]
+                    if last == len(tokens):
+                        last -= 1
+                    while (isinstance(tokens[first].offset, str) and first < last):
+                        first += 1
+                    if first == last:
+                        return True
+                    while (first < last and isinstance(tokens[last].offset, str)):
+                        last -= 1
+                    return tokens[first].offset < jmp_false[0].attr < tokens[last].offset
+                    pass
+                pass
+            pass
+        elif rule == ("list_if_not", ("expr", "jmp_true", "list_iter")):
             jump_inst = ast[1][0]
             jump_offset = jump_inst.attr
             return jump_offset > jump_inst.offset and jump_offset < tokens[last].offset
-        elif rule == ('list_if', ('expr', 'jmp_false', 'list_iter')):
+        elif rule == ("list_if", ("expr", "jmp_false", "list_iter")):
             jump_inst = ast[1][0]
             jump_offset = jump_inst.attr
             return jump_offset > jump_inst.offset and jump_offset < tokens[last].offset
-        elif rule == ('or', ('expr', 'jmp_true', 'expr', '\\e_come_from_opt')):
-            # Test that jmp_true doesn't jump inside the middle the "or"
+        elif rule == ("or", ("expr", "jmp_true", "expr", "\\e_come_from_opt")):
+            # Test that jmp_true doesn"t jump inside the middle the "or"
             # or that it jumps to the same place as the end of "and"
             jmp_true = ast[1][0]
             jmp_target = jmp_true.offset + jmp_true.attr + 3
             return not (jmp_target == tokens[last].offset or
                         tokens[last].pattr == jmp_true.pattr)
 
-        elif (rule[0] == 'whilestmt' and
+        elif (rule[0] == "whilestmt" and
               rule[1][0:-2] ==
-                      ('SETUP_LOOP', 'testexpr', 'l_stmts_opt',
-                       'JUMP_BACK', 'JUMP_BACK')):
+                      ("SETUP_LOOP", "testexpr", "l_stmts_opt",
+                       "JUMP_BACK", "JUMP_BACK")):
             # Make sure that the jump backs all go to the same place
             i = last-1
-            while (tokens[i] != 'JUMP_BACK'):
+            while (tokens[i] != "JUMP_BACK"):
                 i -= 1
             return tokens[i].attr != tokens[i-1].attr
-        elif rule[0] == 'if_expr_true':
-            return (first) > 0 and tokens[first-1] == 'POP_JUMP_IF_FALSE'
+        elif rule[0] == "if_expr_true":
+            return (first) > 0 and tokens[first-1] == "POP_JUMP_IF_FALSE"
 
         return False
 
@@ -288,13 +335,13 @@ class Python27Parser(Python2Parser):
 class Python27ParserSingle(Python27Parser, PythonParserSingle):
     pass
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     # Check grammar
     p = Python27Parser()
     p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 2.7:
-        lhs, rhs, tokens, right_recursive = p.check_sets()
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(
@@ -304,9 +351,9 @@ if __name__ == '__main__':
             """.split()))
         remain_tokens = set(tokens) - opcode_set
         import re
-        remain_tokens = set([re.sub(r'_\d+$', '', t)
+        remain_tokens = set([re.sub(r"_\d+$", "", t)
                              for t in remain_tokens])
-        remain_tokens = set([re.sub('_CONT$', '', t)
+        remain_tokens = set([re.sub("_CONT$", "", t)
                              for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         print(remain_tokens)

uncompyle6/parsers/parse3.py

@@ -86,10 +86,8 @@ class Python3Parser(PythonParser):
         dict_comp_func ::= BUILD_MAP_0 LOAD_FAST FOR_ITER store
                            comp_iter JUMP_BACK RETURN_VALUE RETURN_LAST
 
-        comp_iter     ::= comp_if
         comp_iter     ::= comp_if_not
         comp_if_not   ::= expr jmp_true comp_iter
-        comp_iter     ::= comp_body
         """
 
     def p_grammar(self, args):
@@ -432,7 +430,7 @@ class Python3Parser(PythonParser):
         else:
             return "%s_0" % (token.kind)
 
-    def custom_build_class_rule(self, opname, i, token, tokens, customize):
+    def custom_build_class_rule(self, opname, i, token, tokens, customize, is_pypy):
         """
         # Should the first rule be somehow folded into the 2nd one?
         build_class ::= LOAD_BUILD_CLASS mkfunc
@@ -483,10 +481,18 @@ class Python3Parser(PythonParser):
             call_function = call_fn_tok.kind
             if call_function.startswith("CALL_FUNCTION_KW"):
                 self.addRule("classdef ::= build_class_kw store", nop_func)
-                rule = "build_class_kw ::= LOAD_BUILD_CLASS mkfunc %sLOAD_CONST %s" % (
-                    "expr " * (call_fn_tok.attr - 1),
-                    call_function,
-                )
+                if is_pypy:
+                    args_pos, args_kw = self.get_pos_kw(call_fn_tok)
+                    rule = "build_class_kw ::= LOAD_BUILD_CLASS mkfunc %s%s%s" % (
+                        "expr " * (args_pos - 1),
+                        "kwarg " * (args_kw),
+                        call_function,
+                    )
+                else:
+                    rule = (
+                        "build_class_kw ::= LOAD_BUILD_CLASS mkfunc %sLOAD_CONST %s"
+                        % ("expr " * (call_fn_tok.attr - 1), call_function)
+                    )
             else:
                 call_function = self.call_fn_name(call_fn_tok)
                 rule = "build_class ::= LOAD_BUILD_CLASS mkfunc %s%s" % (
@@ -496,7 +502,7 @@ class Python3Parser(PythonParser):
         self.addRule(rule, nop_func)
         return
 
-    def custom_classfunc_rule(self, opname, token, customize, next_token):
+    def custom_classfunc_rule(self, opname, token, customize, next_token, is_pypy):
         """
         call ::= expr {expr}^n CALL_FUNCTION_n
         call ::= expr {expr}^n CALL_FUNCTION_VAR_n
@@ -514,18 +520,28 @@ class Python3Parser(PythonParser):
         # Yes, this computation based on instruction name is a little bit hoaky.
         nak = (len(opname) - len("CALL_FUNCTION")) // 3
 
-        token.kind = self.call_fn_name(token)
         uniq_param = args_kw + args_pos
 
         # Note: 3.5+ have subclassed this method; so we don't handle
         # 'CALL_FUNCTION_VAR' or 'CALL_FUNCTION_EX' here.
-        rule = (
-            "call ::= expr "
-            + ("pos_arg " * args_pos)
-            + ("kwarg " * args_kw)
-            + "expr " * nak
-            + token.kind
-        )
+        if is_pypy and self.version >= 3.6:
+            if token == "CALL_FUNCTION":
+                token.kind = self.call_fn_name(token)
+            rule = (
+                "call ::= expr "
+                + ("pos_arg " * args_pos)
+                + ("kwarg " * args_kw)
+                + token.kind
+            )
+        else:
+            token.kind = self.call_fn_name(token)
+            rule = (
+                "call ::= expr "
+                + ("pos_arg " * args_pos)
+                + ("kwarg " * args_kw)
+                + "expr " * nak
+                + token.kind
+            )
 
         self.add_unique_rule(rule, token.kind, uniq_param, customize)
 
@@ -543,7 +559,12 @@ class Python3Parser(PythonParser):
         this has an effect on many rules.
         """
         if self.version >= 3.3:
-            new_rule = rule % (("LOAD_STR ") * 1)
+            if PYTHON3 or not self.is_pypy:
+                load_op = "LOAD_STR "
+            else:
+                load_op = "LOAD_CONST "
+
+            new_rule = rule % ((load_op) * 1)
         else:
             new_rule = rule % (("LOAD_STR ") * 0)
         self.add_unique_rule(new_rule, opname, attr, customize)
@@ -571,7 +592,7 @@ class Python3Parser(PythonParser):
 
         """
 
-        is_pypy = False
+        self.is_pypy = False
 
         # For a rough break out on the first word. This may
         # include instructions that don't need customization,
@@ -616,7 +637,7 @@ class Python3Parser(PythonParser):
         # a specific instruction seen.
 
         if "PyPy" in customize:
-            is_pypy = True
+            self.is_pypy = True
             self.addRule(
                 """
               stmt ::= assign3_pypy
@@ -821,7 +842,9 @@ class Python3Parser(PythonParser):
                         """
                     self.addRule(rule, nop_func)
 
-                self.custom_classfunc_rule(opname, token, customize, tokens[i + 1])
+                self.custom_classfunc_rule(
+                    opname, token, customize, tokens[i + 1], self.is_pypy
+                )
                 # Note: don't add to custom_ops_processed.
 
             elif opname_base == "CALL_METHOD":
@@ -880,21 +903,30 @@ class Python3Parser(PythonParser):
                 self.addRule(
                     """
                     stmt        ::= assert_pypy
-                    stmt        ::= assert2_pypy", nop_func)
+                    stmt        ::= assert_not_pypy
+                    stmt        ::= assert2_pypy
+                    stmt        ::= assert2_not_pypy
                     assert_pypy ::=  JUMP_IF_NOT_DEBUG assert_expr jmp_true
                                      LOAD_ASSERT RAISE_VARARGS_1 COME_FROM
+                    assert_not_pypy ::=  JUMP_IF_NOT_DEBUG assert_expr jmp_false
+                                     LOAD_ASSERT RAISE_VARARGS_1 COME_FROM
                     assert2_pypy ::= JUMP_IF_NOT_DEBUG assert_expr jmp_true
                                      LOAD_ASSERT expr CALL_FUNCTION_1
                                      RAISE_VARARGS_1 COME_FROM
                     assert2_pypy ::= JUMP_IF_NOT_DEBUG assert_expr jmp_true
                                      LOAD_ASSERT expr CALL_FUNCTION_1
-                                     RAISE_VARARGS_1 COME_FROM,
+                                     RAISE_VARARGS_1 COME_FROM
+                    assert2_not_pypy ::= JUMP_IF_NOT_DEBUG assert_expr jmp_false
+                                     LOAD_ASSERT expr CALL_FUNCTION_1
+                                     RAISE_VARARGS_1 COME_FROM
                     """,
                     nop_func,
                 )
                 custom_ops_processed.add(opname)
             elif opname == "LOAD_BUILD_CLASS":
-                self.custom_build_class_rule(opname, i, token, tokens, customize)
+                self.custom_build_class_rule(
+                    opname, i, token, tokens, customize, self.is_pypy
+                )
                 # Note: don't add to custom_ops_processed.
             elif opname == "LOAD_CLASSDEREF":
                 # Python 3.4+
@@ -967,7 +999,7 @@ class Python3Parser(PythonParser):
                     j = 1
                 else:
                     j = 2
-                if is_pypy or (i >= j and tokens[i - j] == "LOAD_LAMBDA"):
+                if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_LAMBDA"):
                     rule_pat = "mklambda ::= %sload_closure LOAD_LAMBDA %%s%s" % (
                         "pos_arg " * args_pos,
                         opname,
@@ -982,7 +1014,7 @@ class Python3Parser(PythonParser):
                     self.add_make_function_rule(rule_pat, opname, token.attr, customize)
 
                     if has_get_iter_call_function1:
-                        if is_pypy or (i >= j and tokens[i - j] == "LOAD_LISTCOMP"):
+                        if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_LISTCOMP"):
                             # In the tokens we saw:
                             #   LOAD_LISTCOMP LOAD_CONST MAKE_FUNCTION (>= 3.3) or
                             #   LOAD_LISTCOMP MAKE_FUNCTION (< 3.3) or
@@ -996,7 +1028,7 @@ class Python3Parser(PythonParser):
                             self.add_make_function_rule(
                                 rule_pat, opname, token.attr, customize
                             )
-                        if is_pypy or (i >= j and tokens[i - j] == "LOAD_SETCOMP"):
+                        if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_SETCOMP"):
                             rule_pat = (
                                 "set_comp ::= %sload_closure LOAD_SETCOMP %%s%s expr "
                                 "GET_ITER CALL_FUNCTION_1"
@@ -1005,7 +1037,7 @@ class Python3Parser(PythonParser):
                             self.add_make_function_rule(
                                 rule_pat, opname, token.attr, customize
                             )
-                        if is_pypy or (i >= j and tokens[i - j] == "LOAD_DICTCOMP"):
+                        if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_DICTCOMP"):
                             self.add_unique_rule(
                                 "dict_comp ::= %sload_closure LOAD_DICTCOMP %s "
                                 "expr GET_ITER CALL_FUNCTION_1"
@@ -1051,17 +1083,24 @@ class Python3Parser(PythonParser):
                         )
 
                 elif self.version >= 3.4:
+                    if PYTHON3 or not self.is_pypy:
+                        load_op = "LOAD_STR"
+                    else:
+                        load_op = "LOAD_CONST"
+
                     if annotate_args > 0:
-                        rule = "mkfunc_annotate ::= %s%s%sannotate_tuple load_closure LOAD_CODE LOAD_STR %s" % (
+                        rule = "mkfunc_annotate ::= %s%s%sannotate_tuple load_closure %s %s %s" % (
                             "pos_arg " * args_pos,
                             kwargs_str,
                             "annotate_arg " * (annotate_args - 1),
+                            load_op,
                             opname,
                         )
                     else:
-                        rule = "mkfunc ::= %s%s load_closure LOAD_CODE LOAD_STR %s" % (
+                        rule = "mkfunc ::= %s%s load_closure LOAD_CODE %s %s" % (
                             "pos_arg " * args_pos,
                             kwargs_str,
+                            load_op,
                             opname,
                         )
 
@@ -1119,6 +1158,14 @@ class Python3Parser(PythonParser):
                         opname,
                     )
                     self.add_unique_rule(rule, opname, token.attr, customize)
+                    if not PYTHON3 and self.is_pypy:
+                        rule = "mkfunc ::= %s%s%s%s" % (
+                            "expr " * stack_count,
+                            "load_closure " * closure,
+                            "LOAD_CODE LOAD_CONST ",
+                            opname,
+                        )
+                        self.add_unique_rule(rule, opname, token.attr, customize)
 
                     if has_get_iter_call_function1:
                         rule_pat = (
@@ -1135,7 +1182,7 @@ class Python3Parser(PythonParser):
                         self.add_make_function_rule(
                             rule_pat, opname, token.attr, customize
                         )
-                        if is_pypy or (i >= 2 and tokens[i - 2] == "LOAD_LISTCOMP"):
+                        if self.is_pypy or (i >= 2 and tokens[i - 2] == "LOAD_LISTCOMP"):
                             if self.version >= 3.6:
                                 # 3.6+ sometimes bundles all of the
                                 # 'exprs' in the rule above into a
@@ -1156,7 +1203,7 @@ class Python3Parser(PythonParser):
                                 rule_pat, opname, token.attr, customize
                             )
 
-                    if is_pypy or (i >= 2 and tokens[i - 2] == "LOAD_LAMBDA"):
+                    if self.is_pypy or (i >= 2 and tokens[i - 2] == "LOAD_LAMBDA"):
                         rule_pat = "mklambda ::= %s%sLOAD_LAMBDA %%s%s" % (
                             ("pos_arg " * args_pos),
                             ("kwarg " * args_kw),
@@ -1184,7 +1231,7 @@ class Python3Parser(PythonParser):
                     )
                     self.add_make_function_rule(rule_pat, opname, token.attr, customize)
 
-                    if is_pypy or (i >= j and tokens[i - j] == "LOAD_LISTCOMP"):
+                    if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_LISTCOMP"):
                         # In the tokens we saw:
                         #   LOAD_LISTCOMP LOAD_CONST MAKE_FUNCTION (>= 3.3) or
                         #   LOAD_LISTCOMP MAKE_FUNCTION (< 3.3) or
@@ -1199,7 +1246,7 @@ class Python3Parser(PythonParser):
                         )
 
                 # FIXME: Fold test  into add_make_function_rule
-                if is_pypy or (i >= j and tokens[i - j] == "LOAD_LAMBDA"):
+                if self.is_pypy or (i >= j and tokens[i - j] == "LOAD_LAMBDA"):
                     rule_pat = "mklambda ::= %s%sLOAD_LAMBDA %%s%s" % (
                         ("pos_arg " * args_pos),
                         ("kwarg " * args_kw),
@@ -1390,8 +1437,9 @@ class Python3Parser(PythonParser):
                                          except_handler COME_FROM else_suitel
                                          opt_come_from_except
                     """,
-                    nop_func,
+                    nop_func
                 )
+
                 custom_ops_processed.add(opname)
             elif opname_base in ("UNPACK_EX",):
                 before_count, after_count = token.attr
@@ -1494,12 +1542,24 @@ class Python3Parser(PythonParser):
             for i in range(cfl - 1, first, -1):
                 if tokens[i] != "POP_BLOCK":
                     break
-            if tokens[i].kind not in ("JUMP_BACK", "RETURN_VALUE"):
+            if tokens[i].kind not in ("JUMP_BACK", "RETURN_VALUE", "BREAK_LOOP"):
                 if not tokens[i].kind.startswith("COME_FROM"):
                     return True
 
             # Check that the SETUP_LOOP jumps to the offset after the
             # COME_FROM_LOOP
+
+            # Python 3.0 has additional:
+            #     JUMP_FORWARD here
+            #     COME_FROM
+            #     POP_TOP
+            #     COME_FROM
+            #  here:
+            #     (target of SETUP_LOOP)
+            # We won't check this.
+            if self.version == 3.0:
+                return False
+
             if 0 <= last < len(tokens) and tokens[last] in (
                 "COME_FROM_LOOP",
                 "JUMP_BACK",

uncompyle6/parsers/parse30.py

@@ -12,13 +12,19 @@ class Python30Parser(Python31Parser):
     def p_30(self, args):
         """
 
+        pt_bp             ::= POP_TOP POP_BLOCK
+
         assert            ::= assert_expr jmp_true LOAD_ASSERT RAISE_VARARGS_1 COME_FROM POP_TOP
+        assert2           ::= assert_expr jmp_true LOAD_ASSERT expr CALL_FUNCTION_1 RAISE_VARARGS_1
+                              come_froms
+        call_stmt         ::= expr _come_froms POP_TOP
+
         return_if_lambda  ::= RETURN_END_IF_LAMBDA COME_FROM POP_TOP
         compare_chained2  ::= expr COMPARE_OP RETURN_END_IF_LAMBDA
 
         # FIXME: combine with parse3.2
-        whileTruestmt     ::= SETUP_LOOP l_stmts_opt JUMP_BACK
-                              COME_FROM_LOOP
+        whileTruestmt     ::= SETUP_LOOP l_stmts_opt
+                              jb_or_c COME_FROM_LOOP
         whileTruestmt     ::= SETUP_LOOP returns
                               COME_FROM_LOOP
 
@@ -43,10 +49,22 @@ class Python30Parser(Python31Parser):
 
         else_suitel ::= l_stmts COME_FROM_LOOP JUMP_BACK
 
-        ifelsestmtl ::= testexpr c_stmts_opt jb_pop_top else_suitel
+        jump_absolute_else ::= COME_FROM JUMP_ABSOLUTE COME_FROM POP_TOP
+
+        jump_cf_pop   ::= _come_froms _jump  _come_froms POP_TOP
+
+        ifelsestmt  ::= testexpr c_stmts_opt jump_cf_pop else_suite COME_FROM
+        ifelsestmtl ::= testexpr c_stmts_opt jump_cf_pop else_suitel
+        ifelsestmtc ::= testexpr c_stmts_opt jump_absolute_else else_suitec
+        ifelsestmtc ::= testexpr c_stmts_opt jump_cf_pop else_suitec
+
+        iflaststmt  ::= testexpr c_stmts_opt JUMP_ABSOLUTE COME_FROM
         iflaststmtl ::= testexpr c_stmts_opt jb_pop_top
+        iflaststmtl ::= testexpr c_stmts_opt come_froms JUMP_BACK COME_FROM POP_TOP
+
         iflaststmt  ::= testexpr c_stmts_opt JUMP_ABSOLUTE COME_FROM POP_TOP
 
+
         withasstmt    ::= expr setupwithas store suite_stmts_opt
                           POP_BLOCK LOAD_CONST COME_FROM_FINALLY
                           LOAD_FAST DELETE_FAST WITH_CLEANUP END_FINALLY
@@ -54,8 +72,8 @@ class Python30Parser(Python31Parser):
         setup_finally ::= STORE_FAST SETUP_FINALLY LOAD_FAST DELETE_FAST
 
         # Need to keep LOAD_FAST as index 1
-        set_comp_func_header ::=  BUILD_SET_0 DUP_TOP STORE_FAST
-        set_comp_func ::= set_comp_func_header
+        set_comp_header  ::= BUILD_SET_0 DUP_TOP STORE_FAST
+        set_comp_func ::= set_comp_header
                           LOAD_FAST FOR_ITER store comp_iter
                           JUMP_BACK POP_TOP JUMP_BACK RETURN_VALUE RETURN_LAST
 
@@ -63,8 +81,10 @@ class Python30Parser(Python31Parser):
         list_comp        ::= list_comp_header
                              LOAD_FAST FOR_ITER store comp_iter
                              JUMP_BACK
+        list_comp        ::= list_comp_header
+                             LOAD_FAST FOR_ITER store comp_iter
+                             JUMP_BACK _come_froms POP_TOP JUMP_BACK
 
-        set_comp_header  ::= BUILD_SET_0 DUP_TOP STORE_FAST
         set_comp         ::= set_comp_header
                              LOAD_FAST FOR_ITER store comp_iter
                              JUMP_BACK
@@ -73,6 +93,24 @@ class Python30Parser(Python31Parser):
         dict_comp        ::= dict_comp_header
                              LOAD_FAST FOR_ITER store dict_comp_iter
                              JUMP_BACK
+        dict_comp        ::= dict_comp_header
+                             LOAD_FAST FOR_ITER store dict_comp_iter
+                             JUMP_BACK _come_froms POP_TOP JUMP_BACK
+
+        stmt         ::= try_except30
+        try_except30 ::= SETUP_EXCEPT suite_stmts_opt
+                        _come_froms pt_bp
+                         except_handler opt_come_from_except
+
+        # From Python 2.6
+
+
+        list_iter  ::= list_if JUMP_BACK
+        list_iter  ::= list_if JUMP_BACK _come_froms POP_TOP
+        lc_body    ::= LOAD_NAME expr LIST_APPEND
+	lc_body    ::= LOAD_FAST expr LIST_APPEND
+        list_if    ::= expr jmp_false_then list_iter
+        #############
 
         dict_comp_iter   ::= expr expr ROT_TWO expr STORE_SUBSCR
 
@@ -88,19 +126,52 @@ class Python30Parser(Python31Parser):
         except_suite          ::= c_stmts POP_EXCEPT jump_except POP_TOP
         except_suite_finalize ::= SETUP_FINALLY c_stmts_opt except_var_finalize END_FINALLY
                                   _jump COME_FROM POP_TOP
-        jump_except           ::= JUMP_FORWARD COME_FROM POP_TOP
-        jump_except           ::= JUMP_ABSOLUTE COME_FROM POP_TOP
+
+        except_handler        ::= jmp_abs COME_FROM_EXCEPT except_stmts END_FINALLY
+
+        _ifstmts_jump         ::= c_stmts_opt JUMP_FORWARD COME_FROM POP_TOP
+        _ifstmts_jump         ::= c_stmts_opt come_froms POP_TOP JUMP_FORWARD _come_froms
+
+        jump_except           ::= _jump COME_FROM POP_TOP
+
         or                    ::= expr jmp_false expr jmp_true expr
         or                    ::= expr jmp_true expr
 
+        import_from ::= LOAD_CONST LOAD_CONST IMPORT_NAME importlist _come_froms POP_TOP
+
         ################################################################################
         # In many ways 3.0 is like 2.6. One similarity is there is no JUMP_IF_TRUE and
         # JUMP_IF_FALSE
         # The below rules in fact are the same or similar.
 
-        jmp_true       ::= JUMP_IF_TRUE POP_TOP
-        jmp_false      ::= JUMP_IF_FALSE _come_froms POP_TOP
+        jmp_true         ::= JUMP_IF_TRUE POP_TOP
+        jmp_true_then    ::= JUMP_IF_TRUE _come_froms POP_TOP
+        jmp_false        ::= JUMP_IF_FALSE _come_froms POP_TOP
+        jmp_false_then   ::= JUMP_IF_FALSE POP_TOP
 
+        # We don't have hacky THEN detection, so we do it
+        # in the grammar below which is also somewhat hacky.
+
+        stmt             ::= ifstmt30
+        stmt             ::= ifnotstmt30
+        ifstmt30         ::= testfalse_then _ifstmts_jump30
+        ifnotstmt30      ::= testtrue_then  _ifstmts_jump30
+
+        testfalse_then   ::= expr jmp_false_then
+        testtrue_then    ::= expr jmp_true_then
+        call_stmt        ::= expr COME_FROM
+        _ifstmts_jump30  ::= c_stmts POP_TOP
+
+        gen_comp_body    ::= expr YIELD_VALUE COME_FROM POP_TOP
+
+        except_handler   ::= jmp_abs COME_FROM_EXCEPT except_stmts
+                             COME_FROM POP_TOP END_FINALLY
+
+        or               ::= expr jmp_true_then expr come_from_opt
+        ret_or           ::= expr jmp_true_then expr come_from_opt
+        ret_and          ::= expr jump_false expr come_from_opt
+
+        ################################################################################
         for_block      ::= l_stmts_opt _come_froms POP_TOP JUMP_BACK
 
         except_handler ::= JUMP_FORWARD COME_FROM_EXCEPT except_stmts
@@ -108,12 +179,17 @@ class Python30Parser(Python31Parser):
         except_handler ::= jmp_abs COME_FROM_EXCEPT except_stmts
                            POP_TOP END_FINALLY
 
-        return_if_stmt ::= ret_expr RETURN_END_IF COME_FROM POP_TOP
-        and            ::= expr jmp_false expr come_from_opt
+        return_if_stmt ::= ret_expr RETURN_END_IF come_froms POP_TOP
+        return_if_stmt ::= ret_expr RETURN_VALUE come_froms POP_TOP
+
+        and            ::= expr jmp_false_then expr come_from_opt
+
         whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt come_from_opt
-                           JUMP_BACK COME_FROM POP_TOP POP_BLOCK COME_FROM_LOOP
+                           JUMP_BACK _come_froms POP_TOP POP_BLOCK COME_FROM_LOOP
         whilestmt      ::= SETUP_LOOP testexpr returns
                            POP_TOP POP_BLOCK COME_FROM_LOOP
+        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt come_from_opt
+                           come_froms POP_TOP POP_BLOCK COME_FROM_LOOP
 
 
         # compare_chained is like x <= y <= z
@@ -124,34 +200,177 @@ class Python30Parser(Python31Parser):
         compare_chained2 ::= expr COMPARE_OP RETURN_END_IF
         """
 
-    def customize_grammar_rules(self, tokens, customize):
-        super(Python30Parser, self).customize_grammar_rules(tokens, customize)
+
+    def remove_rules_30(self):
         self.remove_rules("""
+
+        # The were found using grammar coverage
+        while1stmt     ::= SETUP_LOOP l_stmts COME_FROM JUMP_BACK COME_FROM_LOOP
+        whileTruestmt  ::= SETUP_LOOP l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM_LOOP
+        whileelsestmt  ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK else_suitel COME_FROM_LOOP
+        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK COME_FROM_LOOP
+        whilestmt      ::= SETUP_LOOP testexpr l_stmts_opt JUMP_BACK POP_BLOCK JUMP_BACK COME_FROM_LOOP
+        whilestmt      ::= SETUP_LOOP testexpr returns POP_TOP POP_BLOCK COME_FROM_LOOP
+        withasstmt     ::= expr SETUP_WITH store suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+        withstmt       ::= expr SETUP_WITH POP_TOP suite_stmts_opt POP_BLOCK LOAD_CONST COME_FROM_WITH WITH_CLEANUP END_FINALLY
+
+        # lc_body ::= LOAD_FAST expr LIST_APPEND
+        # lc_body ::= LOAD_NAME expr LIST_APPEND
+        # lc_body ::= expr LIST_APPEND
+        # list_comp ::= BUILD_LIST_0 list_iter
+        # list_for ::= expr FOR_ITER store list_iter jb_or_c
+        # list_if ::= expr jmp_false list_iter
+        # list_if ::= expr jmp_false_then list_iter
+        # list_if_not ::= expr jmp_true list_iter
+        # list_iter ::= list_if JUMP_BACK
+        # list_iter ::= list_if JUMP_BACK _come_froms POP_TOP
+        # list_iter ::= list_if_not
+        # load_closure ::= BUILD_TUPLE_0
+        # load_genexpr ::= BUILD_TUPLE_1 LOAD_GENEXPR LOAD_STR
+
+        ##########################################################################################
+
         iflaststmtl        ::= testexpr c_stmts_opt JUMP_BACK COME_FROM_LOOP
         ifelsestmtl        ::= testexpr c_stmts_opt JUMP_BACK else_suitel
         iflaststmt         ::= testexpr c_stmts_opt JUMP_ABSOLUTE
         _ifstmts_jump      ::= c_stmts_opt JUMP_FORWARD _come_froms
+
         jump_forward_else  ::= JUMP_FORWARD ELSE
         jump_absolute_else ::= JUMP_ABSOLUTE ELSE
         whilestmt          ::= SETUP_LOOP testexpr l_stmts_opt COME_FROM JUMP_BACK POP_BLOCK
                                COME_FROM_LOOP
         whilestmt          ::= SETUP_LOOP testexpr returns
                                POP_BLOCK COME_FROM_LOOP
+
         assert             ::= assert_expr jmp_true LOAD_ASSERT RAISE_VARARGS_1
+
         return_if_lambda   ::= RETURN_END_IF_LAMBDA
         except_suite       ::= c_stmts POP_EXCEPT jump_except
         whileelsestmt      ::= SETUP_LOOP testexpr l_stmts JUMP_BACK POP_BLOCK
                                else_suitel COME_FROM_LOOP
 
-        # No JUMP_IF_FALSE_OR_POP
+        ################################################################
+        # No JUMP_IF_FALSE_OR_POP, JUMP_IF_TRUE_OR_POP,
+        # POP_JUMP_IF_FALSE, or POP_JUMP_IF_TRUE
+
+        jmp_false        ::= POP_JUMP_IF_FALSE
+        jmp_true         ::= JUMP_IF_TRUE_OR_POP POP_TOP
+        jmp_true         ::= POP_JUMP_IF_TRUE
+
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
                              compare_chained1 COME_FROM
         compare_chained1 ::= expr DUP_TOP ROT_THREE COMPARE_OP JUMP_IF_FALSE_OR_POP
                              compare_chained2 COME_FROM
+        ret_or           ::= expr JUMP_IF_TRUE_OR_POP  ret_expr_or_cond COME_FROM
+        ret_and          ::= expr JUMP_IF_FALSE_OR_POP ret_expr_or_cond COME_FROM
+        ret_cond         ::= expr POP_JUMP_IF_FALSE expr RETURN_END_IF
+                             COME_FROM ret_expr_or_cond
+        ret_expr_or_cond ::= ret_cond
+        or               ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        and              ::= expr JUMP_IF_TRUE_OR_POP expr COME_FROM
+        and              ::= expr JUMP_IF_FALSE_OR_POP expr COME_FROM
         """)
 
+    def customize_grammar_rules(self, tokens, customize):
+        super(Python30Parser, self).customize_grammar_rules(tokens, customize)
+        self.remove_rules_30()
+
+        self.check_reduce["iflaststmtl"] = "AST"
+        self.check_reduce['ifstmt'] = "AST"
+        self.check_reduce["ifelsestmtc"] = "AST"
+        self.check_reduce["ifelsestmt"] = "AST"
+        # self.check_reduce["and"] = "stmt"
         return
+
+    def reduce_is_invalid(self, rule, ast, tokens, first, last):
+        invalid = super(Python30Parser,
+                        self).reduce_is_invalid(rule, ast,
+                                                tokens, first, last)
+        if invalid:
+            return invalid
+        lhs = rule[0]
+        if (
+            lhs in ("iflaststmtl", "ifstmt",
+                        "ifelsestmt", "ifelsestmtc") and ast[0] == "testexpr"
+        ):
+            testexpr = ast[0]
+            if testexpr[0] == "testfalse":
+                testfalse = testexpr[0]
+                if lhs == "ifelsestmtc" and ast[2] == "jump_absolute_else":
+                    jump_absolute_else = ast[2]
+                    come_from = jump_absolute_else[2]
+                    return come_from == "COME_FROM" and come_from.attr < tokens[first].offset
+                    pass
+                elif lhs in ("ifelsestmt", "ifelsestmtc") and ast[2] == "jump_cf_pop":
+                    jump_cf_pop = ast[2]
+                    come_froms = jump_cf_pop[0]
+                    for come_from in come_froms:
+                        if come_from.attr < tokens[first].offset:
+                            return True
+                    come_froms = jump_cf_pop[2]
+                    if come_froms == "COME_FROM":
+                        if come_froms.attr < tokens[first].offset:
+                            return True
+                        pass
+                    elif come_froms == "_come_froms":
+                        for come_from in come_froms:
+                            if come_from.attr < tokens[first].offset:
+                                return True
+
+                    return False
+                elif testfalse[1] == "jmp_false":
+                    jmp_false = testfalse[1]
+                    if last == len(tokens):
+                        last -= 1
+                    while (isinstance(tokens[first].offset, str) and first < last):
+                        first += 1
+                    if first == last:
+                        return True
+                    while (first < last and isinstance(tokens[last].offset, str)):
+                        last -= 1
+                    if rule[0] == "iflaststmtl":
+                        return not (jmp_false[0].attr <= tokens[last].offset)
+                    else:
+                        jmp_false_target = jmp_false[0].attr
+                        if tokens[first].offset > jmp_false_target:
+                            return True
+                        return (
+                            (jmp_false_target > tokens[last].offset) and tokens[last] != "JUMP_FORWARD")
+                    pass
+                pass
+            pass
+        # elif lhs == "and":
+        #     return tokens[last+1] == "JUMP_FORWARD"
+
     pass
 
 class Python30ParserSingle(Python30Parser, PythonParserSingle):
     pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python30Parser()
+    p.remove_rules_30()
+    p.check_grammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.0:
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        ## FIXME: try this
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub(r'_\d+$', '',  t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$', '', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        import sys
+        if len(sys.argv) > 1:
+            from spark_parser.spark import rule2str
+            for rule in sorted(p.rule2name.items()):
+                print(rule2str(rule[0]))

uncompyle6/parsers/parse31.py

@@ -32,11 +32,49 @@ class Python31Parser(Python32Parser):
         load  ::= LOAD_FAST
         load  ::= LOAD_NAME
         """
+    def remove_rules_31(self):
+        self.remove_rules("""
+        # DUP_TOP_TWO is DUP_TOPX in 3.1 and earlier
+        subscript2 ::= expr expr DUP_TOP_TWO BINARY_SUBSCR
+
+        # The were found using grammar coverage
+        list_if     ::= expr jmp_false list_iter COME_FROM
+        list_if_not ::= expr jmp_true list_iter COME_FROM
+        """)
 
     def customize_grammar_rules(self, tokens, customize):
         super(Python31Parser, self).customize_grammar_rules(tokens, customize)
+        self.remove_rules_31()
         return
     pass
 
 class Python31ParserSingle(Python31Parser, PythonParserSingle):
     pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python31Parser()
+    p.remove_rules_31()
+    p.check_grammar()
+    from uncompyle6 import PYTHON_VERSION, IS_PYPY
+    if PYTHON_VERSION == 3.1:
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
+        from uncompyle6.scanner import get_scanner
+        s = get_scanner(PYTHON_VERSION, IS_PYPY)
+        opcode_set = set(s.opc.opname).union(set(
+            """JUMP_BACK CONTINUE RETURN_END_IF COME_FROM
+               LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP LOAD_CLASSNAME
+               LAMBDA_MARKER RETURN_LAST
+            """.split()))
+        ## FIXME: try this
+        remain_tokens = set(tokens) - opcode_set
+        import re
+        remain_tokens = set([re.sub(r'_\d+$', '',  t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$', '', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        print(remain_tokens)
+        import sys
+        if len(sys.argv) > 1:
+            from spark_parser.spark import rule2str
+            for rule in sorted(p.rule2name.items()):
+                print(rule2str(rule[0]))

uncompyle6/parsers/parse32.py

@@ -8,9 +8,15 @@ from uncompyle6.parser import PythonParserSingle
 from uncompyle6.parsers.parse3 import Python3Parser
 
 class Python32Parser(Python3Parser):
+    def p_30to33(self, args):
+        """
+        # Store locals is only in Python 3.0 to 3.3
+        stmt           ::= store_locals
+        store_locals   ::= LOAD_FAST STORE_LOCALS
+        """
+
     def p_32to35(self, args):
         """
-        expr        ::= conditional
         conditional ::= expr jmp_false expr jump_forward_else expr COME_FROM
 
         # compare_chained2 is used in a "chained_compare": x <= y <= z
@@ -18,10 +24,6 @@ class Python32Parser(Python3Parser):
         compare_chained2 ::= expr COMPARE_OP RETURN_VALUE
         compare_chained2 ::= expr COMPARE_OP RETURN_VALUE_LAMBDA
 
-        # Store locals is only in Python 3.0 to 3.3
-        stmt           ::= store_locals
-        store_locals   ::= LOAD_FAST STORE_LOCALS
-
         # Python < 3.5 no POP BLOCK
         whileTruestmt  ::= SETUP_LOOP l_stmts_opt JUMP_BACK COME_FROM_LOOP
 

uncompyle6/parsers/parse33.py

@@ -14,23 +14,6 @@ class Python33Parser(Python32Parser):
         # Python 3.3+ adds yield from.
         expr          ::= yield_from
         yield_from    ::= expr expr YIELD_FROM
-
-        # We do the grammar hackery below for semantics
-        # actions that want c_stmts_opt at index 1
-
-        # Python 3.5+ has jump optimization to remove the redundant
-        # jump_excepts. But in 3.3 we need them added
-
-        try_except  ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK
-                        except_handler
-                        jump_excepts come_from_except_clauses
-        """
-
-    def p_30to33(self, args):
-        """
-        # Store locals is only in Python 3.0 to 3.3
-        stmt           ::= store_locals
-        store_locals   ::= LOAD_FAST STORE_LOCALS
         """
 
     def customize_grammar_rules(self, tokens, customize):

uncompyle6/parsers/parse34.py

@@ -72,7 +72,7 @@ if __name__ == '__main__':
     p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.4:
-        lhs, rhs, tokens, right_recursive = p.check_sets()
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse35.py

@@ -144,9 +144,15 @@ class Python35Parser(Python34Parser):
         super(Python35Parser, self).customize_grammar_rules(tokens, customize)
         for i, token in enumerate(tokens):
             opname = token.kind
+            if opname == 'LOAD_ASSERT':
+                if 'PyPy' in customize:
+                    rules_str = """
+                    stmt ::= JUMP_IF_NOT_DEBUG stmts COME_FROM
+                    """
+                    self.add_unique_doc_rules(rules_str, customize)
             # FIXME: I suspect this is wrong for 3.6 and 3.5, but
             # I haven't verified what the 3.7ish fix is
-            if opname == 'BUILD_MAP_UNPACK_WITH_CALL':
+            elif opname == 'BUILD_MAP_UNPACK_WITH_CALL':
                 if self.version < 3.7:
                     self.addRule("expr ::= unmapexpr", nop_func)
                     nargs = token.attr % 256
@@ -257,7 +263,7 @@ if __name__ == '__main__':
     p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.5:
-        lhs, rhs, tokens, right_recursive = p.check_sets()
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(

uncompyle6/parsers/parse36.py

@@ -188,13 +188,7 @@ class Python36Parser(Python35Parser):
         for i, token in enumerate(tokens):
             opname = token.kind
 
-            if opname == 'LOAD_ASSERT':
-                if 'PyPy' in customize:
-                    rules_str = """
-                    stmt ::= JUMP_IF_NOT_DEBUG stmts COME_FROM
-                    """
-                    self.add_unique_doc_rules(rules_str, customize)
-            elif opname == 'FORMAT_VALUE':
+            if opname == 'FORMAT_VALUE':
                 rules_str = """
                     expr              ::= formatted_value1
                     formatted_value1  ::= expr FORMAT_VALUE
@@ -316,7 +310,7 @@ class Python36Parser(Python35Parser):
             pass
         return
 
-    def custom_classfunc_rule(self, opname, token, customize, next_token):
+    def custom_classfunc_rule(self, opname, token, customize, next_token, is_pypy):
 
         args_pos, args_kw = self.get_pos_kw(token)
 
@@ -338,10 +332,14 @@ class Python36Parser(Python35Parser):
             self.add_unique_rule('expr ::= async_call', token.kind, uniq_param, customize)
 
         if opname.startswith('CALL_FUNCTION_KW'):
-            self.addRule("expr ::= call_kw36", nop_func)
-            values = 'expr ' * token.attr
-            rule = "call_kw36 ::= expr {values} LOAD_CONST {opname}".format(**locals())
-            self.add_unique_rule(rule, token.kind, token.attr, customize)
+            if is_pypy:
+                # PYPY doesn't follow CPython 3.6 CALL_FUNCTION_KW conventions
+                super(Python36Parser, self).custom_classfunc_rule(opname, token, customize, next_token, is_pypy)
+            else:
+                self.addRule("expr ::= call_kw36", nop_func)
+                values = 'expr ' * token.attr
+                rule = "call_kw36 ::= expr {values} LOAD_CONST {opname}".format(**locals())
+                self.add_unique_rule(rule, token.kind, token.attr, customize)
         elif opname == 'CALL_FUNCTION_EX_KW':
             # Note: this doesn't exist in 3.7 and later
             self.addRule("""expr        ::= call_ex_kw4
@@ -406,7 +404,7 @@ class Python36Parser(Python35Parser):
                             """, nop_func)
             pass
         else:
-            super(Python36Parser, self).custom_classfunc_rule(opname, token, customize, next_token)
+            super(Python36Parser, self).custom_classfunc_rule(opname, token, customize, next_token, is_pypy)
 
     def reduce_is_invalid(self, rule, ast, tokens, first, last):
         invalid = super(Python36Parser,
@@ -443,7 +441,7 @@ if __name__ == '__main__':
     p.check_grammar()
     from uncompyle6 import PYTHON_VERSION, IS_PYPY
     if PYTHON_VERSION == 3.6:
-        lhs, rhs, tokens, right_recursive = p.check_sets()
+        lhs, rhs, tokens, right_recursive, dup_rhs = p.check_sets()
         from uncompyle6.scanner import get_scanner
         s = get_scanner(PYTHON_VERSION, IS_PYPY)
         opcode_set = set(s.opc.opname).union(set(