Get ready for release 2.9.11

by removing a lot of the control-flow labels of 3.3+

.gitignore

@@ -16,3 +16,5 @@
 /unpyc
 __pycache__
 build
+/.venv*
+/.idea

.travis.yml

@@ -9,6 +9,7 @@ python:
   - '3.3'
   - '3.4'
   - '3.2'
+  - '3.6'
 
 install:
 - pip install -r requirements.txt

HISTORY.md

@@ -30,7 +30,7 @@ another clever idea: using table-driven semantics routines, using
 format specifiers.
 
 The last mention of a release of SPARK from John is around 2002. As
-released, although the Early Algorithm parser was in good shape, this
+released, although the Earley Algorithm parser was in good shape, this
 code was woefully lacking as serious Python deparser.
 
 In the fall of 2000, Hartmut Goebel
@@ -44,7 +44,8 @@ it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precedence to the table
 specifiers, support for multiple versions of Python, the
 pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
-deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He says he could verify against the 
+entire Python library. However I have subsequently found small and relatively obscure bugs in the decompilation code. 
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -65,10 +66,12 @@ code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
 time, various JUMP instructions were classifed as going backwards, and
 COME FROM instructions were reintroduced.  See
-RELEASE-2.4-CHANGELOG.txt for more details here. There wasn't a public
+[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt) 
+for more details here. There wasn't a public
 release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
 supported. Dan says the Python 2.3 version could verify the entire
-python library.
+Python library. But given subsequent bugs found like simply
+recognizing complex-number constants in bytecode, decompilation wasn't perfect.
 
 Next we get to ["uncompyle" and
 PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -95,17 +98,17 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
+In `uncompyle`, decompilation of python bytecode 2.5 & 2.6 is done by
 transforming the byte code into a a pseudo 2.7 python bytecode and is
 based on code from Eloi Vanderbeken.
 
-This project, uncompyle6, abandons that approach for various
+This project, `uncompyle6`, abandons that approach for various
 reasons. However the main reason is that we need offsets in fragment
 deparsing to be exactly the same, and the transformation process can
-remove instructions.  Adding instructions with psuedo_offsets is
+remove instructions.  _Adding_ instructions with psuedo offsets is
 however okay.
 
-Uncompyle6, however owes its existence to the fork of uncompyle2 by
+`Uncompyle6` however owes its existence to the fork of `uncompyle2` by
 Myst herie (Mysterie) whose first commit picks up at
 2012. I chose this since it seemed to have been at that time the most
 actively, if briefly, worked on. Also starting around 2012 is Dark
@@ -115,9 +118,12 @@ I started working on this late 2015, mostly to add fragment support.
 In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
 while, handling Python bytecodes from Python versions 2.5+ and
 3.2+. In doing so, it has been expedient to separate this into three
-projects: load loading and disassembly (xdis), parsing and tree
-building (spark_parser), and grammar and semantic actions for
-decompiling (uncompyle6).
+projects:
+
+* bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
+* parsing and tree building ([spark_parser](https://pypi.python.org/pypi/spark_parser)),
+* this project - grammar and semantic actions for decompiling
+  ([uncompyle6](https://pypi.python.org/pypi/spark_parser)).
 
 
 Over the many years, code styles and Python features have
@@ -135,23 +141,26 @@ Hartmut a decade an a half ago:
     NB. This is not a masterpiece of software, but became more like a hack.
     Probably a complete rewrite would be sensefull. hG/2000-12-27
 
-This project deparses using an Early-algorithm parse with lots of
+This project deparses using an Earley-algorithm parse with lots of
 massaging of tokens and the grammar in the scanner
-phase. Early-algorithm parsers are context free and tend to be linear
+phase. Earley-algorithm parsers are context free and tend to be linear
 if the grammar is LR or left recursive.
 
 Another approach that doesn't use grammars is to do something like
 simulate execution symbolically and build expression trees off of
-stack results. The two important projects that work this way are
-[unpyc3](https://code.google.com/p/unpyc3/) and most especially
-[pycdc](https://github.com/zrax/pycdc) The latter project is largely
-by Michael Hansen and Darryl Pogue. If they supported getting
-source-code fragments and I could call it from Python, I'd probably
-ditch this and use that. From what I've seen, the code runs blindingly
-fast and spans all versions of Python.
+stack results. Control flow in that apprproach still needs to be
+handled somewhat ad hoc.  The two important projects that work this
+way are [unpyc3](https://code.google.com/p/unpyc3/) and most
+especially [pycdc](https://github.com/zrax/pycdc) The latter project
+is largely by Michael Hansen and Darryl Pogue. If they supported
+getting source-code fragments, did a better job in supporting Python
+more fully, and had a way I could call it from Python, I'd probably
+would have ditched this and used that. The code runs blindingly fast
+and spans all versions of Python, although more recently Python 3
+support has been lagging.
 
 Tests for the project have been, or are being, culled from all of the
 projects mentioned.
 
-NB. If you find mistakes, want corrections, or want your name added (or removed),
-please contact me.
+NB. If you find mistakes, want corrections, or want your name added
+(or removed), please contact me.

HOW-TO-REPORT-A-BUG.md

@@ -0,0 +1,63 @@
+# How to report a Bug
+
+## The difficulty of the problem
+
+There is no Python decompiler yet, that I know about that will
+decompyle everything. This one probably does the
+best job of *any* Python decompiler. But it is a constant work in progress: Python keeps changing, and so does its code generation.
+
+I have found bugs in *every* Python decompiler I have tried. Even
+those where authors/maintainers claim that they have used it on 
+the entire Python standard library. And I don't mean that
+the program doesn't come out with the same Python source instructions,
+but that the program is *semantically* not equivalent.
+
+So it is likely you'll find a mistranslation in decompiling.
+
+## What to send (minimum requirements)
+
+The basic requirement is pretty simple:
+
+* Python bytecode
+* Source text
+
+## What to send (additional helpful information)
+
+Some kind folks also give the invocation they used and the output
+which usually includes an error message produced. This is helpful. I
+can figure out what OS you are running this on and what version of
+*uncomplye6* was used. Therefore, if you don't provide the input
+command and the output from that, please give:
+
+* _uncompile6_ version used
+* OS that you used this on
+* Python interpreter version used
+
+
+### But I don't *have* the source code!
+
+Sure, I get it. No problem. There is Python assembly code on parse
+errors, so simply by hand decompile that. To get a full disassembly, use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis) package. Opcodes are described in the documentation for the [dis](https://docs.python.org/3.6/library/dis.html) module.
+
+### But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!
+
+Well, you could learn. No one is born into this world knowing how to disassemble Python bytecode. And as Richard Feynman once said, "What one fool can learn, so can another."
+
+## Narrowing the problem
+
+I don't need the entire source code base for which one file or module
+can't be decompiled. I just need that one file or module only. If
+there are several files, file a bug report for each file.
+
+Python modules can get quite large, and usually decompilation problems
+occur in a single function or maybe the main-line code but not any of
+the functions or classes. So please chop down the source code by
+removing those parts that do to decompile properly.
+
+By doing this, you'll probably have a better sense of what exactly is
+the problem. Perhaps you can find the boundary of what decompiles, and
+what doesn't. That is useful. Or maybe the same file will decompile
+properly on a neighboring version of Python. That is helpful too.
+
+In sum, the more you can isolate or narrow the problem, the more
+likley the problem will be fixed and fixed sooner.

MANIFEST.in

@@ -1,6 +1,7 @@
 include README.rst
 include ChangeLog
 include HISTORY.md
+include HOW-TO-REPORT-A-BUG.md
 include LICENSE
 include Makefile
 include requirements.txt

NEWS

@@ -1,3 +1,38 @@
+uncompyle6 2.9.11 2016-04-06
+
+- Better support for Python 3.5+ BUILD_MAP_UNPACK
+- Start 3.6 CALL_FUNCTION_EX support
+- Many decompilation bug fixes. (Many more remain). See ChangeLog
+
+uncompyle6 2.9.10 2016-02-25
+
+- Python grammar rule fixes
+- Add ability to get grammar coverage on runs
+- Handle Python 3.6 opcode BUILD_CONST_KEYMAP
+
+uncompyle6 2.9.9 2016-12-16
+
+- Remaining Python 3.5 ops handled
+  (this also means more Python 3.6 ops are handled)
+- Python 3.5 and 3.6 async and await handled
+- Python 3.0 decompilation improved
+- Python 3 annotations fixed
+- Better control-flow detection
+- Code cleanups and misc bug fixes
+
+uncompyle6 2.9.8 2016-12-16
+
+- Better control-flow detection
+- pseudo instruction THEN in 2.x
+  to disambiguate if from and
+- fix bug in --verify option
+- DRY (a little) control-flow detection
+- fix syntax in tuples with one element
+- if AST rule inheritence in Python 2.5
+- NAME_MODULE removal for Python <= 2.4
+- verifycall fixes for Python <= 2.4
+- more Python lint
+
 uncompyle6 2.9.7 2016-12-16
 
 - Start to handle 3.5/3.6 build_map_unpack_with_call

README.rst

@@ -11,8 +11,8 @@ Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 2.1 to 3.6 or
-so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
+source code. It accepts bytecodes from Python version 1.5, and 2.1 to
+3.6 or so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
 
 Why this?
 ---------
@@ -21,7 +21,8 @@ There were a number of decompyle, uncompile, uncompyle2, uncompyle3
 forks around. All of them came basically from the same code base, and
 almost all of them no were no longer actively maintained. Only one
 handled Python 3, and even there, only 3.2 or 3.3 depending on which
-code is used. This code pulls these together and moves forward. It
+code is used. This code pulls these together and moves forward. This
+project has the most complete support for Python 3.3 and above. It
 also addresses a number of open issues in the previous forks.
 
 What makes this different from other CPython bytecode decompilers?: its
@@ -43,10 +44,10 @@ information.
 Requirements
 ------------
 
-This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1. 
-Python versions 2.3-2.7 are supported in the python-2.4 branch.
+This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
+Python versions 2.4-2.7 are supported in the python-2.4 branch.
 The bytecode files it can read has been tested on Python bytecodes from
-versions 2.1-2.7, and 3.2-3.6 and the above-mentioned PyPy versions.
+versions 1.5, 2.1-2.7, and 3.0-3.6 and the above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -93,6 +94,16 @@ For usage help:
 
    $ uncompyle6 -h
 
+If you want strong verification of the correctness of the
+decompilation process, add the `--verify` option. But there are
+situations where this will indicate a failure, although the generated
+program is semantically equivalent. Using option `--weak-verify` will
+tell you if there is something definitely wrong. Generally, large
+swaths of code are decompiled correctly, if not the entire program.
+
+You can also cross compare the results with pycdc_ . Since they work
+differently, bugs here often aren't in that, and vice versa.
+
 
 Known Bugs/Restrictions
 -----------------------
@@ -102,7 +113,7 @@ with handling control flow. All of the Python decompilers I have looked
 at have the same problem. In some cases we can detect an erroneous
 decompilation and report that.
 
-About 90% of the decompilation of Python standard library packages in
+Over 98% of the decompilation of Python standard library packages in
 Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
 "weakly" verify. Python 2.6 drops down to 96% weakly verifying.
 Other versions drop off in quality too.
@@ -128,10 +139,12 @@ on the lower end Python versions which is more difficult for us to
 handle since we don't have a Python interpreter for versions 1.5, 1.6,
 and 2.0.
 
-Python 3.0 support is weak; Python 3.5 largely works, but still has
-some bugs in it.  Python 3.6 changes things drastically by using word
-codes rather than byte codes. That has been addressed, but then it also
-changes function call opcodes and its semantics.
+In the Python 3 series, Python support is is strongest around 3.4 or
+3.3 and drops off as you move further away from those versions. Python
+3.6 changes things drastically by using word codes rather than byte
+codes. That has been addressed, but then it also changes function call
+opcodes and its semantics and has more problems with control flow than
+3.5 has.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
@@ -145,13 +158,16 @@ We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
 Deobfuscator_ to unscramble the bytecode to get valid bytecode before
 trying this tool.
 
+Handling pathologically long lists of expressions or statements is
+slow.
+
 
 There is lots to do, so please dig in and help.
 
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
 * The HISTORY_ file.

__pkginfo__.py

@@ -9,7 +9,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2017 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 5 - Production/Stable',
@@ -39,8 +39,8 @@ entry_points={
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.5.1, < 1.6.0',
-                      'xdis >= 3.2.4, < 3.3.0']
+install_requires   = ['spark-parser >= 1.6.0, < 1.7.0',
+                      'xdis >= 3.3.0, < 3.4.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

pytest/test_build_const_key_map.py

@@ -0,0 +1,21 @@
+import pytest
+# uncompyle6
+from uncompyle6 import PYTHON_VERSION
+from validate import validate_uncompyle
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@pytest.mark.parametrize('text', (
+    "{0.: 'a', -1: 'b'}",   # BUILD_MAP
+    "{'a':'b'}",            # BUILD_MAP
+    "{0: 1}",               # BUILD_MAP
+    "{b'0':1, b'2':3}",     # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a': 1, 'b': 2}",     # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0.0:'b',0.1:'d'}",    # BUILD_CONST_KEY_MAP
+))
+def test_build_const_key_map(text):
+    validate_uncompyle(text)

pytest/test_function_call.py

@@ -0,0 +1,128 @@
+# std
+import string
+# 3rd party
+from hypothesis import given, assume, strategies as st
+import pytest
+# uncompyle
+from validate import validate_uncompyle
+
+
+alpha = st.sampled_from(string.ascii_lowercase)
+numbers = st.sampled_from(string.digits)
+alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
+expressions = st.sampled_from([x for x in string.ascii_lowercase + string.digits] + ['x+1'])
+
+
+@st.composite
+def function_calls(draw):
+    """
+    Strategy factory for generating function calls.
+
+    :param draw: Callable which draws examples from other strategies.
+
+    :return: The function call text.
+    """
+    list1 = st.lists(alpha, min_size=0, max_size=1)
+    list3 = st.lists(alpha, min_size=0, max_size=3)
+
+    positional_args = draw(list3)
+    named_args = [x + '=0' for x in draw(list3)]
+    star_args = ['*' + x for x in draw(list1)]
+    double_star_args = ['**' + x for x in draw(list1)]
+
+    arguments = positional_args + named_args + star_args + double_star_args
+    draw(st.randoms()).shuffle(arguments)
+    arguments = ','.join(arguments)
+
+    function_call = 'fn({arguments})'.format(arguments=arguments)
+    try:
+        # TODO: Figure out the exact rules for ordering of positional, named,
+        # star args, double star args and in which versions the various
+        # types of arguments are supported so we don't need to check that the
+        # expression compiles like this.
+        compile(function_call, '<string>', 'single')
+    except:
+        assume(False)
+    return function_call
+
+
+@pytest.mark.xfail()
+def test_CALL_FUNCTION():
+    validate_uncompyle("fn(w,m,f)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(w=0,m=0,**v)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(a=0,**g)")
+
+
+@pytest.mark.xfail()
+def test_CALL_FUNCTION_KW():
+    validate_uncompyle("fn(j=0)")
+
+
+@pytest.mark.xfail()
+def test_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(*g,**j)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_MAP_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(*z,u=0)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_TUPLE_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(**a)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(b,b,b=0,*a)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(*c,v)")
+
+
+@pytest.mark.xfail()
+def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
+    validate_uncompyle("fn(i=0,y=0,*p)")
+
+
+@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
+@given(function_calls())
+def test_function_call(function_call):
+    validate_uncompyle(function_call)
+
+
+examples = set()
+generate_examples = False
+
+
+@pytest.mark.skipif(not generate_examples, reason='not generating examples')
+@given(function_calls())
+def test_generate_hypothesis(function_call):
+    examples.add(function_call)
+
+
+@pytest.mark.skipif(not generate_examples, reason='not generating examples')
+def test_generate_examples():
+    import dis
+    example_opcodes = {}
+    for example in examples:
+        opcodes = tuple(sorted(set(
+            instruction.opname
+            for instruction in dis.Bytecode(example)
+            if instruction.opname not in ('LOAD_CONST', 'LOAD_NAME', 'RETURN_VALUE')
+        )))
+        example_opcodes[opcodes] = example
+    for k, v in example_opcodes.items():
+        print('def test_' + '_'.join(k) + '():\n    validate_uncompyle("' + v + '")\n\n')
+    return

pytest/test_grammar.py

@@ -40,7 +40,9 @@ def test_grammar():
     ignore_set = set(
             """
             JUMP_BACK CONTINUE RETURN_END_IF
-            COME_FROM COME_FROM_EXCEPT COME_FROM_LOOP COME_FROM_WITH
+            COME_FROM COME_FROM_EXCEPT
+            COME_FROM_EXCEPT_CLAUSE
+            COME_FROM_LOOP COME_FROM_WITH
             COME_FROM_FINALLY ELSE
             LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
             LAMBDA_MARKER RETURN_LAST

pytest/validate.py

@@ -0,0 +1,147 @@
+# future
+from __future__ import print_function
+# std
+import os
+import difflib
+import subprocess
+import tempfile
+import functools
+# compatability
+import six
+# uncompyle6 / xdis
+from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
+# TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
+from xdis.bytecode import Bytecode
+from xdis.main import get_opcode
+opc = get_opcode(PYTHON_VERSION, IS_PYPY)
+Bytecode = functools.partial(Bytecode, opc=opc)
+
+
+def _dis_to_text(co):
+    return Bytecode(co).dis()
+
+
+def print_diff(original, uncompyled):
+    """
+    Try and display a pretty html line difference between the original and
+    uncompyled code and bytecode if elinks and BeautifulSoup are installed
+    otherwise just show the diff.
+
+    :param original: Text describing the original code object.
+    :param uncompyled: Text describing the uncompyled code object.
+    """
+    original_lines = original.split('\n')
+    uncompyled_lines = uncompyled.split('\n')
+    args = original_lines, uncompyled_lines, 'original', 'uncompyled'
+    try:
+        from bs4 import BeautifulSoup
+        diff = difflib.HtmlDiff().make_file(*args)
+        diff = BeautifulSoup(diff, "html.parser")
+        diff.select_one('table[summary="Legends"]').extract()
+    except ImportError:
+        print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
+        diff = difflib.HtmlDiff().make_table(*args)
+
+    with tempfile.NamedTemporaryFile(delete=False) as f:
+        f.write(str(diff).encode('utf-8'))
+
+    try:
+        print()
+        html = subprocess.check_output([
+            'elinks',
+            '-dump',
+            '-no-references',
+            '-dump-color-mode',
+            '1',
+            f.name,
+        ]).decode('utf-8')
+        print(html)
+    except:
+        print('\nFor side by side diff install elinks')
+        diff = difflib.Differ().compare(original_lines, uncompyled_lines)
+        print('\n'.join(diff))
+    finally:
+        os.unlink(f.name)
+
+
+def are_instructions_equal(i1, i2):
+    """
+    Determine if two instructions are approximately equal,
+    ignoring certain fields which we allow to differ, namely:
+
+    * code objects are ignore (should probaby be checked) due to address
+    * line numbers
+
+    :param i1: left instruction to compare
+    :param i2: right instruction to compare
+
+    :return: True if the two instructions are approximately equal, otherwise False.
+    """
+    result = (1==1
+        and i1.opname == i2.opname
+        and i1.opcode == i2.opcode
+        and i1.arg == i2.arg
+        # ignore differences due to code objects
+        # TODO : Better way of ignoring address
+        and (i1.argval == i2.argval or '<code object' in str(i1.argval))
+        # TODO : Should probably recurse to check code objects
+        and (i1.argrepr == i2.argrepr or '<code object' in i1.argrepr)
+        and i1.offset == i2.offset
+        # ignore differences in line numbers
+        #and i1.starts_line
+        and i1.is_jump_target == i2.is_jump_target
+    )
+    return result
+
+
+def are_code_objects_equal(co1, co2):
+    """
+    Determine if two code objects are approximately equal,
+    see are_instructions_equal for more information.
+
+    :param i1: left code object to compare
+    :param i2: right code object to compare
+
+    :return: True if the two code objects are approximately equal, otherwise False.
+    """
+    instructions1 = Bytecode(co1)
+    instructions2 = Bytecode(co2)
+    for opcode1, opcode2 in zip(instructions1, instructions2):
+        if not are_instructions_equal(opcode1, opcode2):
+            return False
+    return True
+
+
+def validate_uncompyle(text, mode='exec'):
+    """
+    Validate decompilation of the given source code.
+
+    :param text: Source to validate decompilation of.
+    """
+    original_code = compile(text, '<string>', mode)
+    original_dis = _dis_to_text(original_code)
+    original_text = text
+
+    deparsed = deparse_code(PYTHON_VERSION, original_code,
+                            compile_mode=mode, out=six.StringIO())
+    uncompyled_text = deparsed.text
+    uncompyled_code = compile(uncompyled_text, '<string>', 'exec')
+
+    if not are_code_objects_equal(uncompyled_code, original_code):
+
+        uncompyled_dis = _dis_to_text(uncompyled_text)
+
+        def output(text, dis):
+            width = 60
+            return '\n\n'.join([
+                ' SOURCE CODE '.center(width, '#'),
+                text.strip(),
+                ' BYTECODE '.center(width, '#'),
+                dis
+            ])
+
+        original = output(original_text, original_dis)
+        uncompyled = output(uncompyled_text, uncompyled_dis)
+        print_diff(original, uncompyled)
+
+        assert 'original' == 'uncompyled'

requirements-dev.txt

@@ -1,3 +1,4 @@
 pytest
 flake8
-hypothesis
+hypothesis
+six

test/Makefile

@@ -3,7 +3,7 @@ PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clea
 GIT2CL ?= git2cl
 PYTHON ?= python
 
-PYTHON_VERSION = $(shell $(PYTHON) -V | cut -d ' ' -f 2 | cut -d'.' -f1,2)
+PYTHON_VERSION = $(shell $(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2)
 NATIVE_CHECK = check-$(PYTHON_VERSION)
 
 # Set COMPILE='--compile' to force compilation before check
@@ -16,11 +16,10 @@ check-short:
 
 # Run all tests
 check:
-	@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
-	$(MAKE) check-$$PYTHON_VERSION
+	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-2.7-ok
+check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-native-short
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
@@ -98,9 +97,24 @@ check-bytecode-2.4:
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
+#: Get grammar coverage for Python 2.5
+grammar-coverage-2.5:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+
+#: Get grammar coverage for Python 2.6
+grammar-coverage-2.6:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+
+#: Get grammar coverage for Python 2.7
+grammar-coverage-2.7:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
-	$(PYTHON) test_pythonlib.py --bytecode-2.6
+	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
 
 #: Check deparsing Python 2.7
 check-bytecode-2.7:

test/simple_source/bug22/01_ops.py

@@ -0,0 +1,18 @@
+# Statements to beef up grammar coverage rules
+# Force "inplace" ops
+y = +10  # UNARY_POSITIVE
+y /= 1   # INPLACE_DIVIDE
+y %= 4   # INPLACE_MODULO
+y **= 1  # INPLACE POWER
+y >>= 2  # INPLACE_RSHIFT
+y <<= 2  # INPLACE_LSHIFT
+y //= 1  # INPLACE_TRUE_DIVIDE
+y &= 1   # INPLACE_AND
+y ^= 1   # INPLACE_XOR
+
+`y`      # UNARY_CONVERT  - No in Python 3.x
+
+# Beef up augassign and STORE_SLICE+3
+x = [1,2,3,4,5]
+x[0:1] = 1
+x[0:3] += 1, 2, 3

test/simple_source/bug22/03_if1.py

@@ -0,0 +1,7 @@
+# From https://github.com/ToontownInfinite /src/otp/avatar/LocalAvatar.py#L364
+if 1:
+    def jumpLandAnimFix(self, jumpTime):
+        return 5
+
+    def jumpLand(self):
+        return 6

test/simple_source/bug25/02_try_else.py

@@ -1,7 +1,6 @@
 # Python 2.5 bug
 # Was turning into tryelse when there in fact is no else.
 def options(self, section):
-    """Return a list of option names for the given section name."""
     try:
         opts = self._sections[section].copy()
     except KeyError:
@@ -10,3 +9,19 @@ def options(self, section):
     if '__name__' in opts:
         del opts['__name__']
     return opts.keys()
+
+# From python2.5/distutils/command/register.py
+def post_to_server(self, urllib2):
+    try:
+        result = 5
+    except urllib2.HTTPError, e:
+        result = e.code, e.msg
+    except urllib2.URLError, e:
+        result = 500
+    else:
+        if self.show_response:
+            result = 10
+        result = 200
+    if self.show_response:
+        result = 8
+    return result

test/simple_source/bug25/03_if_for.py

@@ -0,0 +1,7 @@
+# From Python 2.6. distutils/sysconfig.py
+def get_config_vars(_config_vars, args):
+    if _config_vars:
+        if args == 1:
+            if args < 8:
+                for key in ('LDFLAGS', 'BASECFLAGS'):
+                    _config_vars[key] = 4

test/simple_source/bug26/03_double_equals.py

@@ -0,0 +1,6 @@
+# From Python 2.7 parse_starttag HTMLParser.pyc
+attrvalue = [1,2]
+while attrvalue:
+    if attrvalue[:1] == 5 or \
+       attrvalue[:1] == 2 == attrvalue[-1:]:
+        attrvalue = 10

test/simple_source/bug27+/03_if_1_else.py

@@ -0,0 +1 @@
+1 if 1 else __file__

test/simple_source/bug31/02_ifelse_comprehension.py

@@ -0,0 +1,12 @@
+# Python 2.7 sqlalchemy-1.013/sql/crud.py
+def _extend_values_for_multiparams(compiler, stmt, c):
+    c(
+        [
+            (
+                (compiler() if compiler()
+                    else compiler())
+                if c in stmt else compiler(),
+            )
+        ]
+        for i in enumerate(stmt)
+    )

test/simple_source/bug31/04_def_annotate.py

@@ -8,3 +8,23 @@ def open(file, mode = "r", buffering = None,
          encoding = None, errors = None,
          newline = None, closefd = True) -> "IOBase":
     return text
+
+def foo(x: 'an argument that defaults to 5' = 5):
+    print(x)
+
+def div(a: dict(type=float, help='the dividend'),
+        b: dict(type=float, help='the divisor (must be different than 0)')
+    ) -> dict(type=float, help='the result of dividing a by b'):
+    """Divide a by b"""
+    return a / b
+
+class TestSignatureObject(unittest.TestCase):
+    def test_signature_on_wkwonly(self):
+        def test(*, a:float, b:str) -> int:
+            pass
+
+class SupportsInt(_Protocol):
+
+    @abstractmethod
+    def __int__(self) -> int:
+        pass

test/simple_source/bug32/01_named_and_kwargs.py

@@ -18,3 +18,12 @@ def __init__(self, defaults=None, dict_type=_default_dict,
              default_section=DEFAULTSECT,
              interpolation=_UNSET):
     pass
+
+# Bug found by hypothesis in creating function calls
+# thanks to moagstar
+def fn(a, b, d):
+    return (a, b, d)
+
+b = {'b': 1,
+     'd': 2}
+fn(a=0, **b)

test/simple_source/bug32/01_try_except_raise.py

@@ -0,0 +1,9 @@
+# From 3.2 _abcoll.py
+def pop(self):
+    it = iter(self)
+    try:
+        value = next(it)
+    except StopIteration:
+        raise KeyError
+    self.discard(value)
+    return value

test/simple_source/bug32/03_if.py

@@ -0,0 +1,11 @@
+# From 3.2 shlex.py
+def _samefile(os, src, dst):
+    if hasattr(os.path, 'samefile'):
+        try:
+            return os.path.samefile(src, dst)
+        except OSError:
+            return False
+
+    # All other platforms: check for same pathname.
+    return (os.path.normcase(os.path.abspath(src)) ==
+            os.path.normcase(os.path.abspath(dst)))

test/simple_source/bug33/01_delete_deref.py

@@ -0,0 +1,4 @@
+def a():
+    del y
+    def b():
+        return y

test/simple_source/bug33/01_if_try_except.py

@@ -0,0 +1,16 @@
+# From 3.3.5 _osx_support.py
+def _get_system_version():
+    if __file__ is None:
+        try:
+            m = 5
+        except IOError:
+            pass
+        else:
+            try:
+                m = 10
+            finally:
+                m = 15
+            if m is not None:
+                m = 20
+
+    return m

test/simple_source/bug33/01_try_except.py

@@ -0,0 +1,16 @@
+# From 3.3.5 _osx_support.py
+def _get_system_version():
+    if __file__ is None:
+        try:
+            m = 5
+        except IOError:
+            pass
+        else:
+            try:
+                m = 10
+            finally:
+                m = 15
+            if m is not None:
+                m = 20
+
+    return m

test/simple_source/bug33/02_pos_args.py

@@ -1,3 +1,6 @@
 # From Python 3.3.6 hmac.py
 # Problem was getting wrong placement of positional args
 digest_cons = lambda d=b'': 5
+
+# Handle single kwarg
+lambda *, d=0: None

test/simple_source/bug33/02_while1.py

@@ -0,0 +1,14 @@
+# From Python 3.4 mailcap
+def readmailcapfile(caps):
+    while 1:
+        line = 'abc'
+        if line[0] == '#' or line == '':
+            continue
+        key, fields = (1,2)
+        if not (key and fields):
+            continue
+        if key in caps:
+            caps[key].append(fields)
+        else:
+            caps[key] = [fields]
+    return caps

test/simple_source/bug35/01_map_unpack.py

@@ -0,0 +1,9 @@
+# Python 3.5+ PEP 448 - Additional Unpacking Generalizations for dictionaries
+{**{}}
+{**{'a': 1, 'b': 2}}
+## {**{'x': 1}, **{'y': 2}}
+# {'c': 1, {'d': 2}, **{'e': 3}}
+[*[]]
+{**{0:0 for a in b}}
+## {**{}, **{}}
+## {**{}, **{}, **{}}

test/simple_source/bug35/02_async_for.py

@@ -0,0 +1,3 @@
+async def a(b, c):
+    async for b in c:
+        pass

test/simple_source/bug35/03_async_await.py

@@ -0,0 +1,32 @@
+# Python 3.5+ async and await
+async def await_test(asyncio):
+    reader, writer = await asyncio.open_connection(80)
+    await bar()
+
+async def afor_test():
+
+    async for i in [1,2,3]:
+        x = i
+
+
+async def afor_else_test():
+
+    async for i in [1,2,3]:
+        x = i
+    else:
+        z = 4
+
+
+async def awith_test():
+    async with i:
+        print(i)
+
+async def awith_as_test():
+    async with 1 as i:
+        print(i)
+
+async def f(z):
+    await z
+
+async def g(z):
+    return await z

test/simple_source/bug35/03_double_star_unpack.py

@@ -0,0 +1,9 @@
+# Bug in Python 3.5 is getting the two star'd arguments right.
+def sum(a,b,c,d):
+    return a + b + c + d
+
+args=(1,2)
+sum(*args, *args)
+
+# FIXME: this is handled incorrectly
+# (*c,) = (3,4)

test/simple_source/bug35/03_while-if-break.py

@@ -0,0 +1,14 @@
+# Python 3.5 and 3.6 break inside a
+# while True and if / break
+def display_date(loop):
+    while True:
+        if loop.time():
+            break
+        x = 5
+
+    # Another loop to test 3.5 ifelsestmtl grammar rule
+    while loop:
+        if x:
+            True
+        else:
+            True

test/simple_source/bug36/01_call_function.py

@@ -0,0 +1,5 @@
+# Python 3.6's changes for calling functions.
+# See https://github.com/rocky/python-uncompyle6/issues/58
+# CALL_FUNCTION_EX takes 2 to 3 arguments on the stack: the function, the tuple of positional arguments,
+# and optionally the dict of keyword arguments if bit 0 of oparg is 1.
+a(*[])

test/simple_source/bug36/01_extended_arg.py

@@ -0,0 +1,2 @@
+if __file__:
+    0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0+0

test/simple_source/def/10_kw+pos_args-bug.py

@@ -8,3 +8,20 @@ def __init__(self, defaults=None, dict_type=_default_dict,
              default_section=DEFAULTSECT,
              interpolation=_UNSET):
     pass
+
+# From 3.5 sqlalchemy/orm/__init__.py
+# Python 3.5 changes the stack position of where * args are (furthest down the stack)
+# Python 3.6+ replaces CALL_FUNCTION_VAR_KW with CALL_FUNCTION_EX
+def deferred(*columns, **kw):
+    return ColumnProperty(deferred=True, *columns, **kw)
+
+
+# From sqlalchemy/sql/selectable.py
+class GenerativeSelect():
+    def __init__(self,
+                 ClauseList,
+                 util,
+                 order_by=None):
+        self._order_by_clause = ClauseList(
+            *util.to_list(order_by),
+            _literal_as_text=5)

test/simple_source/exception/02_try_finally.py

@@ -0,0 +1,11 @@
+# From 2.6.9 cmd.py
+try:
+    if __file__:
+        x = 2
+    x = 3
+finally:
+    if x and __file__:
+        try:
+            x = 1
+        except:
+            pass

test/simple_source/stmts/02_while1else.py

@@ -0,0 +1,8 @@
+# From Python-3.5.2/Lib/multiprocessing/connection.py
+
+def PipeClient(address):
+    while 1:
+        z = 2
+    else:
+        raise
+    return

test/test_pyenvlib.py

@@ -30,10 +30,10 @@ from fnmatch import fnmatch
 TEST_VERSIONS=('2.3.7', '2.4.6', '2.5.6', '2.6.9',
                'pypy-2.4.0', 'pypy-2.6.1',
                'pypy-5.0.1', 'pypy-5.3.1',
-               '2.7.10', '2.7.11', '2.7.12',
+               '2.7.10', '2.7.11', '2.7.12', '2.7.13',
                '3.0.1', '3.1.5', '3.2.6',
                '3.3.5', '3.3.6',
-               '3.4.2', '3.5.1')
+               '3.4.2', '3.5.1', '3.6.0')
 
 target_base = '/tmp/py-dis/'
 lib_prefix = os.path.join(os.environ['HOME'], '.pyenv/versions')
@@ -106,28 +106,40 @@ def do_tests(src_dir, patterns, target_dir, start_with=None, do_verify=False):
 if __name__ == '__main__':
     import getopt, sys
 
-    do_verify = False
+    do_coverage = do_verify = False
     test_dirs = []
     start_with = None
 
     test_options_keys = list(test_options.keys())
     test_options_keys.sort()
     opts, args = getopt.getopt(sys.argv[1:], '',
-                               ['start-with=', 'verify', 'weak-verify', 'all', ] \
+                               ['start-with=', 'verify', 'weak-verify',
+                                'coverage', 'all', ] \
                                + test_options_keys )
+    vers = ''
     for opt, val in opts:
         if opt == '--verify':
             do_verify = True
         if opt == '--weak-verify':
             do_verify = 'weak'
+        if opt == '--coverage':
+            do_coverage = True
         elif opt == '--start-with':
             start_with = val
         elif opt[2:] in test_options_keys:
-            test_dirs.append(test_options[opt[2:]])
+            triple = test_options[opt[2:]]
+            vers = triple[-1]
+            test_dirs.append(triple)
         elif opt == '--all':
+            vers = 'all'
             for val in test_options_keys:
                 test_dirs.append(test_options[val])
 
+    if do_coverage:
+        os.environ['SPARK_PARSER_COVERAGE'] = (
+            '/tmp/spark-grammar-%s.cover' % vers
+            )
+
     for src_dir, pattern, target_dir in test_dirs:
         if os.path.exists(src_dir):
             target_dir = os.path.join(target_base, target_dir)

test/test_pythonlib.py

@@ -190,6 +190,7 @@ if __name__ == '__main__':
     test_options_keys.sort()
     opts, args = getopt.getopt(sys.argv[1:], '',
                                ['start-with=', 'verify', 'weak-verify', 'all', 'compile',
+                                'coverage',
                                 'no-rm'] \
                                + test_options_keys )
     if not opts: help()
@@ -198,7 +199,8 @@ if __name__ == '__main__':
         'do_compile': False,
         'do_verify': False,
         'start_with': None,
-        'rmtree' : True
+        'rmtree' : True,
+        'coverage' : False
         }
 
     for opt, val in opts:
@@ -217,11 +219,18 @@ if __name__ == '__main__':
         elif opt == '--all':
             for val in test_options_keys:
                 test_dirs.append(test_options[val])
+        elif opt == '--coverage':
+            test_opts['coverage'] = True
         else:
             help()
             pass
         pass
 
+    if test_opts['coverage']:
+        os.environ['SPARK_PARSER_COVERAGE'] = (
+            '/tmp/spark-grammar-python-lib%s.cover' % test_dirs[0][-1]
+            )
+
     last_compile_version = None
     for src_dir, pattern, target_dir, compiled_version in test_dirs:
         if os.path.isdir(src_dir):

uncompyle6/__init__.py

@@ -41,13 +41,18 @@ PYTHON_VERSION_STR  = "%s.%s" % (sys.version_info[0], sys.version_info[1])
 
 IS_PYPY = '__pypy__' in sys.builtin_module_names
 
-sys.setrecursionlimit(5000)
+if hasattr(sys, 'setrecursionlimit'):
+    # pyston doesn't have setrecursionlimit
+    sys.setrecursionlimit(5000)
 
 import uncompyle6.semantics.pysource
 import uncompyle6.semantics.fragments
 
 # Export some functions
-from uncompyle6.main import uncompyle_file
+from uncompyle6.main import decompile_file
+
+# For compaitility
+uncompyle_file = decompile_file
 
 # Conventience functions so you can say:
 # from uncompyle6 import deparse_code

uncompyle6/main.py

@@ -11,7 +11,7 @@ from uncompyle6.linenumbers import line_number_mapping
 
 from xdis.load import load_module
 
-def uncompyle(
+def decompile(
         bytecode_version, co, out=None, showasm=None, showast=False,
         timestamp=None, showgrammar=False, code_objects={},
         source_size=None, is_pypy=False, magic_int=None):
@@ -48,8 +48,10 @@ def uncompyle(
         # deparsing failed
         raise pysource.SourceWalkerError(str(e))
 
+# For compatiblity
+uncompyle = decompile
 
-def uncompyle_file(filename, outstream=None, showasm=None, showast=False,
+def decompile_file(filename, outstream=None, showasm=None, showast=False,
                    showgrammar=False):
     """
     decompile Python byte-code file (.pyc)
@@ -62,16 +64,20 @@ def uncompyle_file(filename, outstream=None, showasm=None, showast=False,
 
     if type(co) == list:
         for con in co:
-            uncompyle(version, con, outstream, showasm, showast,
+            decompile(version, con, outstream, showasm, showast,
                       timestamp, showgrammar, code_objects=code_objects,
                       is_pypy=is_pypy, magic_int=magic_int)
     else:
-        uncompyle(version, co, outstream, showasm, showast,
+        decompile(version, co, outstream, showasm, showast,
                   timestamp, showgrammar,
                   code_objects=code_objects, source_size=source_size,
                   is_pypy=is_pypy, magic_int=magic_int)
     co = None
 
+# For compatiblity
+uncompyle_file = decompile_file
+
+
 # FIXME: combine into an options parameter
 def main(in_base, out_base, files, codes, outfile=None,
          showasm=None, showast=False, do_verify=False,
@@ -101,12 +107,6 @@ def main(in_base, out_base, files, codes, outfile=None,
 
     tot_files = okay_files = failed_files = verify_failed_files = 0
 
-    # for code in codes:
-    #    version = sys.version[:3] # "2.5"
-    #    with open(code, "r") as f:
-    #        co = compile(f.read(), "", "exec")
-    #    uncompyle(sys.version[:3], co, sys.stdout, showasm=showasm, showast=showast)
-
     for filename in files:
         infile = os.path.join(in_base, filename)
         if not os.path.exists(infile):
@@ -126,8 +126,9 @@ def main(in_base, out_base, files, codes, outfile=None,
                     prefix = prefix[:-len('.py')]
                 junk, outfile = tempfile.mkstemp(suffix=".py",
                                              prefix=prefix)
-                # Unbuffer output
-                sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
+                # Unbuffer output if possible
+                buffering = -1 if sys.stdout.isatty() else 0
+                sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', buffering)
                 tee = subprocess.Popen(["tee", outfile], stdin=subprocess.PIPE)
                 os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
                 os.dup2(tee.stdin.fileno(), sys.stderr.fileno())
@@ -141,7 +142,7 @@ def main(in_base, out_base, files, codes, outfile=None,
 
         # Try to uncompile the input file
         try:
-            uncompyle_file(infile, outstream, showasm, showast, showgrammar)
+            decompile_file(infile, outstream, showasm, showast, showgrammar)
             tot_files += 1
         except (ValueError, SyntaxError, ParserError, pysource.SourceWalkerError) as e:
             sys.stdout.write("\n")
@@ -230,11 +231,11 @@ def status_msg(do_verify, tot_files, okay_files, failed_files,
                verify_failed_files):
     if tot_files == 1:
         if failed_files:
-            return "decompile failed"
+            return "\n# decompile failed"
         elif verify_failed_files:
-            return "decompile verify failed"
+            return "\n# decompile verify failed"
         else:
-            return "Successfully decompiled file"
+            return "\n# Successfully decompiled file"
             pass
         pass
     mess = "decompiled %i files: %i okay, %i failed" % (tot_files, okay_files, failed_files)

uncompyle6/parser.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2015-2016 Rocky Bernstein
+#  Copyright (c) 2015-2017 Rocky Bernstein
 #  Copyright (c) 2005 by Dan Pascu <dan@windowmaker.org>
 #  Copyright (c) 2000-2002 by hartmut Goebel <h.goebel@crazy-compilers.com>
 #  Copyright (c) 1999 John Aycock
@@ -28,6 +28,22 @@ nop_func = lambda self, args: None
 
 class PythonParser(GenericASTBuilder):
 
+    def __init__(self, AST, start, debug):
+        super(PythonParser, self).__init__(AST, start, debug)
+        self.collect = frozenset(
+            ['stmts', 'except_stmts', '_stmts', 'load_attrs',
+             'exprlist', 'kvlist', 'kwargs', 'come_froms', '_come_from',
+              # Python < 3
+             'print_items',
+             # PyPy:
+             'kvlist_n'])
+
+    def ast_first_offset(self, ast):
+        if hasattr(ast, 'offset'):
+            return ast.offset
+        else:
+            return self.ast_first_offset(ast[0])
+
     def add_unique_rule(self, rule, opname, count, customize):
         """Add rule to grammar, but only if it hasn't been added previously
            opname and count are used in the customize() semantic the actions
@@ -115,11 +131,7 @@ class PythonParser(GenericASTBuilder):
         return token.type
 
     def nonterminal(self, nt, args):
-        collect = ('stmts', 'exprlist', 'kvlist', '_stmts', 'print_items', 'kwargs',
-                   # PYPY:
-                   'kvlist_n')
-
-        if nt in collect and len(args) > 1:
+        if nt in self.collect and len(args) > 1:
             #
             #  Collect iterated thingies together. That is rather than
             #  stmts -> stmts stmt -> stmts stmt -> ...
@@ -389,8 +401,7 @@ class PythonParser(GenericASTBuilder):
         import_cont ::= LOAD_CONST LOAD_CONST import_as_cont
         import_as_cont ::= IMPORT_FROM designator
 
-        load_attrs ::= LOAD_ATTR
-        load_attrs ::= load_attrs LOAD_ATTR
+        load_attrs ::= LOAD_ATTR+
         """
 
     def p_list_comprehension(self, args):
@@ -497,8 +508,12 @@ class PythonParser(GenericASTBuilder):
         expr ::= conditional
         conditional ::= expr jmp_false expr JUMP_FORWARD expr COME_FROM
         conditional ::= expr jmp_false expr JUMP_ABSOLUTE expr
+
         expr ::= conditionalnot
-        conditionalnot ::= expr jmp_true expr _jump expr COME_FROM
+        conditionalnot  ::= expr jmp_true expr _jump expr COME_FROM
+
+        expr            ::= conditionalTrue
+        conditionalTrue ::= expr JUMP_FORWARD expr COME_FROM
 
         ret_expr ::= expr
         ret_expr ::= ret_and

uncompyle6/parsers/parse2.py

@@ -44,7 +44,8 @@ class Python2Parser(PythonParser):
         while1stmt     ::= SETUP_LOOP l_stmts     JUMP_BACK COME_FROM
         while1stmt     ::= SETUP_LOOP l_stmts     JUMP_BACK POP_BLOCK COME_FROM
 
-        while1elsestmt ::= SETUP_LOOP l_stmts JUMP_BACK else_suite COME_FROM
+        while1elsestmt ::= SETUP_LOOP l_stmts JUMP_BACK POP_BLOCK else_suite COME_FROM
+        while1elsestmt ::= SETUP_LOOP l_stmts JUMP_BACK           else_suite COME_FROM
 
         exec_stmt ::= expr exprlist DUP_TOP EXEC_STMT
         exec_stmt ::= expr exprlist EXEC_STMT
@@ -126,6 +127,7 @@ class Python2Parser(PythonParser):
         assert_expr_and ::= assert_expr jmp_false expr
 
         ifstmt ::= testexpr _ifstmts_jump
+        ifstmt ::= testexpr return_if_stmts COME_FROM
 
         testexpr ::= testfalse
         testexpr ::= testtrue
@@ -144,6 +146,8 @@ class Python2Parser(PythonParser):
 
         ifelsestmtr ::= testexpr return_if_stmts return_stmts
 
+        ifelsestmtr ::= testexpr return_if_stmts COME_FROM return_stmts
+
         ifelsestmtl ::= testexpr c_stmts_opt JUMP_BACK else_suitel