Administrivia: Remove six dependency..

add version of hypothesis known to work

Makefile

@@ -38,6 +38,7 @@ check-3.0 check-3.1 check-3.2 check-3.6:
 	$(MAKE) -C test $@
 
 check-3.7: pytest
+	$(MAKE) -C test check
 
 #:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
 # Skip for now

NEWS

@@ -1,3 +1,18 @@
+uncompyle6 3.2.2 2018-06-04 When I'm 64
+
+- Python 3.0 support and bug fixes
+
+uncompyle6 3.2.1 2018-06-04 MF
+
+- Python 1.4 and 1.5 bug fixes
+
+uncompyle6 3.2.0 2018-05-19 Rocket Scientist
+
+- Add rudimentary 1.4 support (still a bit buggy)
+- add --tree+ option to show formatting rule, when it is constant
+- Python 2.7.15candidate1 support (via xdis)
+- bug fixes, especially for 3.7 (but 2.7 and 3.6 and others as well)
+
 uncompyle6 3.1.3 2018-04-16
 
 - Add some Python 3.7 rules, such as for handling LOAD_METHOD (not complete)

README.rst

@@ -1,4 +1,4 @@
-|buildstatus|
+|buildstatus| |Latest Version| |Supported Python Versions|
 
 uncompyle6
 ==========
@@ -29,11 +29,11 @@ CPython bytecode decompilers is the ability to deparse just
 *fragments* of source code and give source-code information around a
 given bytecode offset.
 
-I use the tree fragments to deparse fragments of code inside my
-trepan_ debuggers_. For that, bytecode offsets are recorded and
-associated with fragments of the source code. This purpose, although
-compatible with the original intention, is yet a little bit different.
-See this_ for more information.
+I use the tree fragments to deparse fragments of code *at run time*
+inside my trepan_ debuggers_. For that, bytecode offsets are recorded
+and associated with fragments of the source code. This purpose,
+although compatible with the original intention, is yet a little bit
+different.  See this_ for more information.
 
 Python fragment deparsing given an instruction offset is useful in
 showing stack traces and can be encorporated into any program that
@@ -58,7 +58,7 @@ provides decompilation for subset of Python versions, we generally do
 demonstrably better for those as well.
 
 How can we tell? By taking Python bytecode that comes distributed with
-that version of Python and decompiling these.  Among htose that
+that version of Python and decompiling these.  Among those that
 successfully decompile, we can then make sure the resulting programs
 are syntactically correct by running the Python interpreter for that
 bytecode version.  Finally, in cases where the program has a test for
@@ -151,12 +151,12 @@ for that bytecode version. Having done this the bytecode produced
 could be compared with the original bytecode. However as Python's code
 generation got better, this is no longer feasible.
 
-There is a kind of *weak verification* that we use that doesn't check
-bytecode for equivalence but does check to see if the resulting
-decompiled source is a valid Python program by running the Python
-interpreter. Because the Python language has changed so much, for best
-results you should use the same Python version in checking as was used
-in creating the bytecode.
+There verification that we use that doesn't check bytecode for
+equivalence but does check to see if the resulting decompiled source
+is a valid Python program by running the Python interpreter. Because
+the Python language has changed so much, for best results you should
+use the same Python version in checking as was used in creating the
+bytecode.
 
 There are however an interesting class of these programs that is
 readily available give stronger verification: those programs that
@@ -174,19 +174,21 @@ that era was minimal)
 
 There is some work to do on the lower end Python versions which is
 more difficult for us to handle since we don't have a Python
-interpreter for versions 1.5, 1.6, and 2.0.
+interpreter for versions 1.6, and 2.0.
 
 In the Python 3 series, Python support is is strongest around 3.4 or
 3.3 and drops off as you move further away from those versions. Python
-3.6 changes things drastically by using word codes rather than byte
-codes. As a result, the jump offset field in a jump instruction
-argument has been reduced. This makes the `EXTENDED_ARG` instructions
-are now more prevalent in jump instruction; previously they had been
-rare.  Perhaps to compensate for the additional `EXTENDED_ARG`
-instructions, additional jump optimization has been added. So in sum
-handling control flow by ad hoc means as is currently done is worse.
+3.0 is weird in that it in some ways resembles 2.6 more than it does
+3.1 or 2.7. Python 3.6 changes things drastically by using word codes
+rather than byte codes. As a result, the jump offset field in a jump
+instruction argument has been reduced. This makes the `EXTENDED_ARG`
+instructions are now more prevalent in jump instruction; previously
+they had been rare.  Perhaps to compensate for the additional
+`EXTENDED_ARG` instructions, additional jump optimization has been
+added. So in sum handling control flow by ad hoc means as is currently
+done is worse.
 
-Also, between Python 3.5, 3.6 and 3.7 there have been major changes to the
+Between Python 3.5, 3.6 and 3.7 there have been major changes to the
 `MAKE_FUNCTION` and `CALL_FUNCTION` instructions.
 
 Currently not all Python magic numbers are supported. Specifically in
@@ -212,11 +214,12 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for Python 3 is a bit lacking though.
-* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
-* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
-* The HISTORY_ file.
+* https://github.com/zrax/pycdc : purports to support all versions of Python. It is written in C++ and is most accurate for Python versions around 2.7 and 3.3 when the code was more actively developed. Accuracy for more recent versions of Python 3 and early versions of Python are especially lacking. See its `issue tracker <https://github.com/zrax/pycdc/issues>`_ for details. Currently lightly maintained.
+* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here. Currently unmaintained.
+* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations. Currently unmaintained.
+* https://github.com/wibiti/uncompyle2 : supports Python 2.7 only, but does that fairly well. Because of it specificity it can sometimes to better than uncompyle6 which we can't do withouth breaking other 2.7 cases. Currently lightly maintained. See its issue `tracker <https://github.com/wibiti/uncompyle2/issues>`_ for more details
 * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
+* The HISTORY_ file.
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
 * https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
@@ -234,3 +237,6 @@ See Also
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
 .. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe
+.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
+.. |Latest Version| image:: https://badge.fury.io/py/uncompyle6.svg
+		 :target: https://badge.fury.io/py/uncompyle6

__pkginfo__.py

@@ -35,6 +35,7 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 2.5',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
+                'Programming Language :: Python :: 3.0',
                 'Programming Language :: Python :: 3.1',
                 'Programming Language :: Python :: 3.2',
                 'Programming Language :: Python :: 3.3',
@@ -56,7 +57,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
-                      'xdis >= 3.8.0, < 3.9.0', 'six']
+                      'xdis >= 3.8.4, < 3.9.0']
 
 license            = 'GPL3'
 mailing_list       = 'python-debugger@googlegroups.com'

admin-tools/check-newer-versions.sh

@@ -16,6 +16,7 @@ if ! source ./setup-master.sh ; then
 fi
 cd ..
 for version in $PYVERSIONS; do
+    echo --- $version ---
     if ! pyenv local $version ; then
 	exit $?
     fi
@@ -23,4 +24,5 @@ for version in $PYVERSIONS; do
     if ! make check; then
 	exit $?
     fi
+    echo === $version ===
 done

admin-tools/check-older-versions.sh

@@ -15,6 +15,7 @@ fi
 
 cd ..
 for version in $PYVERSIONS; do
+    echo --- $version ---
     if ! pyenv local $version ; then
 	exit $?
     fi
@@ -22,4 +23,5 @@ for version in $PYVERSIONS; do
     if ! make check ; then
 	exit $?
     fi
+    echo === $version ===
 done

admin-tools/setup-master.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-PYTHON_VERSION=3.6.4
+PYTHON_VERSION=3.6.5
 
 # FIXME put some of the below in a common routine
 function finish {

appveyor.yml

@@ -47,7 +47,7 @@ install:
 
   # Upgrade to the latest version of pip to avoid it displaying warnings
   # about it being out of date.
-  - "pip install --disable-pip-version-check --user --upgrade pip"
+  - "%PYTHON%\\python.exe -m pip install --disable-pip-version-check --user --upgrade pip"
 
   # Install the build dependencies of the project. If some dependencies contain
   # compiled extensions and are not provided as pre-built wheel packages,

circle.yml

@@ -6,6 +6,7 @@ machine:
 
 dependencies:
     override:
+        - pip install --upgrade setuptools
         - pip install -e .
         - pip install pytest==3.2.5 hypothesis
 test:

pytest/test_fstring.py

@@ -1,150 +1,154 @@
 # std
-import os
 # test
-import pytest
-import hypothesis
-from hypothesis import strategies as st
-# uncompyle6
 from uncompyle6 import PYTHON_VERSION, deparse_code
+import pytest
+pytestmark = pytest.mark.skipif(PYTHON_VERSION <= 2.6,
+                                reason='hypothesis needs 2.7 or later')
+if PYTHON_VERSION > 2.6:
+
+    import hypothesis
+    from hypothesis import strategies as st
+
+    # uncompyle6
 
 
-@st.composite
-def expressions(draw):
-    # todo : would be nice to generate expressions using hypothesis however
-    # this is pretty involved so for now just use a corpus of expressions
-    # from which to select.
-    return draw(st.sampled_from((
-        'abc',
-        'len(items)',
-        'x + 1',
-        'lineno',
-        'container',
-        'self.attribute',
-        'self.method()',
-        # These expressions are failing, I think these are control
-        # flow problems rather than problems with FORMAT_VALUE,
-        # however I need to confirm this...
-        #'sorted(items, key=lambda x: x.name)',
-        #'func(*args, **kwargs)',
-        #'text or default',
-        #'43 if life_the_universe and everything else None'
-    )))
+    @st.composite
+    def expressions(draw):
+        # todo : would be nice to generate expressions using hypothesis however
+        # this is pretty involved so for now just use a corpus of expressions
+        # from which to select.
+        return draw(st.sampled_from((
+            'abc',
+            'len(items)',
+            'x + 1',
+            'lineno',
+            'container',
+            'self.attribute',
+            'self.method()',
+            # These expressions are failing, I think these are control
+            # flow problems rather than problems with FORMAT_VALUE,
+            # however I need to confirm this...
+            #'sorted(items, key=lambda x: x.name)',
+            #'func(*args, **kwargs)',
+            #'text or default',
+            #'43 if life_the_universe and everything else None'
+        )))
 
 
-@st.composite
-def format_specifiers(draw):
-    """
-    Generate a valid format specifier using the rules:
+    @st.composite
+    def format_specifiers(draw):
+        """
+        Generate a valid format specifier using the rules:
 
-    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
-    fill        ::=  <any character>
-    align       ::=  "<" | ">" | "=" | "^"
-    sign        ::=  "+" | "-" | " "
-    width       ::=  integer
-    precision   ::=  integer
-    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
+        format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
+        fill        ::=  <any character>
+        align       ::=  "<" | ">" | "=" | "^"
+        sign        ::=  "+" | "-" | " "
+        width       ::=  integer
+        precision   ::=  integer
+        type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
 
-    See https://docs.python.org/2/library/string.html
+        See https://docs.python.org/2/library/string.html
 
-    :param draw: Let hypothesis draw from other strategies.
+        :param draw: Let hypothesis draw from other strategies.
 
-    :return: An example format_specifier.
-    """
-    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
-    fill = draw(st.one_of(alphabet_strategy, st.none()))
-    align = draw(st.sampled_from(list('<>=^')))
-    fill_align = (fill + align or '') if fill else ''
+        :return: An example format_specifier.
+        """
+        alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
+        fill = draw(st.one_of(alphabet_strategy, st.none()))
+        align = draw(st.sampled_from(list('<>=^')))
+        fill_align = (fill + align or '') if fill else ''
 
-    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
-    can_have_sign = type_ in 'deEfFgGnoxX%'
-    can_have_comma = type_ in 'deEfFgG%'
-    can_have_precision = type_ in 'fFgG'
-    can_have_pound = type_ in 'boxX%'
-    can_have_zero = type_ in 'oxX'
+        type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
+        can_have_sign = type_ in 'deEfFgGnoxX%'
+        can_have_comma = type_ in 'deEfFgG%'
+        can_have_precision = type_ in 'fFgG'
+        can_have_pound = type_ in 'boxX%'
+        can_have_zero = type_ in 'oxX'
 
-    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
-    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
-    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
+        sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
+        pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
+        zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
 
-    int_strategy = st.integers(min_value=1, max_value=1000)
+        int_strategy = st.integers(min_value=1, max_value=1000)
 
-    width = draw(st.one_of(int_strategy, st.none()))
-    width = str(width) if width is not None else ''
+        width = draw(st.one_of(int_strategy, st.none()))
+        width = str(width) if width is not None else ''
 
-    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
-    if can_have_precision:
-        precision = draw(st.one_of(int_strategy, st.none()))
-        precision = '.' + str(precision) if precision else ''
-    else:
-        precision = ''
+        comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
+        if can_have_precision:
+            precision = draw(st.one_of(int_strategy, st.none()))
+            precision = '.' + str(precision) if precision else ''
+        else:
+            precision = ''
 
-    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
+        return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
 
 
-@st.composite
-def fstrings(draw):
-    """
-    Generate a valid f-string.
-    See https://www.python.org/dev/peps/pep-0498/#specification
+    @st.composite
+    def fstrings(draw):
+        """
+        Generate a valid f-string.
+        See https://www.python.org/dev/peps/pep-0498/#specification
 
-    :param draw: Let hypothsis draw from other strategies.
+        :param draw: Let hypothsis draw from other strategies.
 
-    :return: A valid f-string.
-    """
-    character_strategy = st.characters(
-        blacklist_characters='\r\n\'\\s{}',
-        min_codepoint=1,
-        max_codepoint=1000,
-    )
-    is_raw = draw(st.booleans())
-    integer_strategy = st.integers(min_value=0, max_value=3)
-    expression_count = draw(integer_strategy)
-    content = []
-    for _ in range(expression_count):
-        expression = draw(expressions())
-        conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
-        has_specifier = draw(st.booleans())
-        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
-        content.append('{{{}{}}}'.format(expression, conversion, specifier))
-        content.append(draw(st.text(character_strategy)))
-    content = ''.join(content)
-    return "f{}'{}'".format('r' if is_raw else '', content)
+        :return: A valid f-string.
+        """
+        character_strategy = st.characters(
+            blacklist_characters='\r\n\'\\s{}',
+            min_codepoint=1,
+            max_codepoint=1000,
+        )
+        is_raw = draw(st.booleans())
+        integer_strategy = st.integers(min_value=0, max_value=3)
+        expression_count = draw(integer_strategy)
+        content = []
+        for _ in range(expression_count):
+            expression = draw(expressions())
+            conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
+            has_specifier = draw(st.booleans())
+            specifier = ':' + draw(format_specifiers()) if has_specifier else ''
+            content.append('{{{}{}}}'.format(expression, conversion, specifier))
+            content.append(draw(st.text(character_strategy)))
+        content = ''.join(content)
+        return "f{}'{}'".format('r' if is_raw else '', content)
 
 
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(format_specifiers())
-def test_format_specifiers(format_specifier):
-    """Verify that format_specifiers generates valid specifiers"""
-    try:
-        exec('"{:' + format_specifier + '}".format(0)')
-    except ValueError as e:
-        if 'Unknown format code' not in str(e):
-            raise
+    @pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+    @hypothesis.given(format_specifiers())
+    def test_format_specifiers(format_specifier):
+        """Verify that format_specifiers generates valid specifiers"""
+        try:
+            exec('"{:' + format_specifier + '}".format(0)')
+        except ValueError as e:
+            if 'Unknown format code' not in str(e):
+                raise
 
 
-def run_test(text):
-    hypothesis.assume(len(text))
-    hypothesis.assume("f'{" in text)
-    expr = text + '\n'
-    code = compile(expr, '<string>', 'single')
-    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
-    recompiled = compile(deparsed.text, '<string>', 'single')
-    if recompiled != code:
-        assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
+    def run_test(text):
+        hypothesis.assume(len(text))
+        hypothesis.assume("f'{" in text)
+        expr = text + '\n'
+        code = compile(expr, '<string>', 'single')
+        deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
+        recompiled = compile(deparsed.text, '<string>', 'single')
+        if recompiled != code:
+            assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
 
 
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(fstrings())
-def test_uncompyle_fstring(fstring):
-    """Verify uncompyling fstring bytecode"""
-    run_test(fstring)
+    @pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+    @hypothesis.given(fstrings())
+    def test_uncompyle_fstring(fstring):
+        """Verify uncompyling fstring bytecode"""
+        run_test(fstring)
 
 
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@pytest.mark.parametrize('fstring', [
-    "f'{abc}{abc!s}'",
-    "f'{abc}0'",
-])
-def test_uncompyle_direct(fstring):
-    """useful for debugging"""
-    run_test(fstring)
+    @pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+    @pytest.mark.parametrize('fstring', [
+        "f'{abc}{abc!s}'",
+        "f'{abc}0'",
+    ])
+    def test_uncompyle_direct(fstring):
+        """useful for debugging"""
+        run_test(fstring)

pytest/test_function_call.py

@@ -1,184 +1,185 @@
-# std
 import string
-# 3rd party
-from hypothesis import given, assume, example, settings, strategies as st
-import pytest
-# uncompyle
-from validate import validate_uncompyle
-from test_fstring import expressions
 from uncompyle6 import PYTHON_VERSION
+import pytest
+pytestmark = pytest.mark.skip(PYTHON_VERSION < 2.7,
+                              reason="need at least Python 2.7")
 
-alpha = st.sampled_from(string.ascii_lowercase)
-numbers = st.sampled_from(string.digits)
-alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
+if PYTHON_VERSION > 2.6:
+    from hypothesis import given, assume, example, settings, strategies as st
+    from validate import validate_uncompyle
+    from test_fstring import expressions
+
+    alpha = st.sampled_from(string.ascii_lowercase)
+    numbers = st.sampled_from(string.digits)
+    alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
 
 
-@st.composite
-def function_calls(draw,
-                   min_keyword_args=0, max_keyword_args=5,
-                   min_positional_args=0, max_positional_args=5,
-                   min_star_args=0, max_star_args=1,
-                   min_double_star_args=0, max_double_star_args=1):
-    """
-    Strategy factory for generating function calls.
+    @st.composite
+    def function_calls(draw,
+                       min_keyword_args=0, max_keyword_args=5,
+                       min_positional_args=0, max_positional_args=5,
+                       min_star_args=0, max_star_args=1,
+                       min_double_star_args=0, max_double_star_args=1):
+        """
+        Strategy factory for generating function calls.
 
-    :param draw: Callable which draws examples from other strategies.
+        :param draw: Callable which draws examples from other strategies.
 
-    :return: The function call text.
-    """
-    st_positional_args = st.lists(
-        alpha,
-        min_size=min_positional_args,
-        max_size=max_positional_args
-    )
-    st_keyword_args = st.lists(
-        alpha,
-        min_size=min_keyword_args,
-        max_size=max_keyword_args
-    )
-    st_star_args = st.lists(
-        alpha,
-        min_size=min_star_args,
-        max_size=max_star_args
-    )
-    st_double_star_args = st.lists(
-        alpha,
-        min_size=min_double_star_args,
-        max_size=max_double_star_args
-    )
+        :return: The function call text.
+        """
+        st_positional_args = st.lists(
+            alpha,
+            min_size=min_positional_args,
+            max_size=max_positional_args
+        )
+        st_keyword_args = st.lists(
+            alpha,
+            min_size=min_keyword_args,
+            max_size=max_keyword_args
+        )
+        st_star_args = st.lists(
+            alpha,
+            min_size=min_star_args,
+            max_size=max_star_args
+        )
+        st_double_star_args = st.lists(
+            alpha,
+            min_size=min_double_star_args,
+            max_size=max_double_star_args
+        )
 
-    positional_args = draw(st_positional_args)
-    keyword_args = draw(st_keyword_args)
-    st_values = st.lists(
-        expressions(),
-        min_size=len(keyword_args),
-        max_size=len(keyword_args)
-    )
-    keyword_args = [
-        x + '=' + e
-        for x, e in
-        zip(keyword_args, draw(st_values))
-    ]
-    star_args = ['*' + x for x in draw(st_star_args)]
-    double_star_args = ['**' + x for x in draw(st_double_star_args)]
+        positional_args = draw(st_positional_args)
+        keyword_args = draw(st_keyword_args)
+        st_values = st.lists(
+            expressions(),
+            min_size=len(keyword_args),
+            max_size=len(keyword_args)
+        )
+        keyword_args = [
+            x + '=' + e
+            for x, e in
+            zip(keyword_args, draw(st_values))
+        ]
+        star_args = ['*' + x for x in draw(st_star_args)]
+        double_star_args = ['**' + x for x in draw(st_double_star_args)]
 
-    arguments = positional_args + keyword_args + star_args + double_star_args
-    draw(st.randoms()).shuffle(arguments)
-    arguments = ','.join(arguments)
+        arguments = positional_args + keyword_args + star_args + double_star_args
+        draw(st.randoms()).shuffle(arguments)
+        arguments = ','.join(arguments)
 
-    function_call = 'fn({arguments})'.format(arguments=arguments)
-    try:
-        # TODO: Figure out the exact rules for ordering of positional, keyword,
-        # star args, double star args and in which versions the various
-        # types of arguments are supported so we don't need to check that the
-        # expression compiles like this.
-        compile(function_call, '<string>', 'single')
-    except:
-        assume(False)
-    return function_call
+        function_call = 'fn({arguments})'.format(arguments=arguments)
+        try:
+            # TODO: Figure out the exact rules for ordering of positional, keyword,
+            # star args, double star args and in which versions the various
+            # types of arguments are supported so we don't need to check that the
+            # expression compiles like this.
+            compile(function_call, '<string>', 'single')
+        except:
+            assume(False)
+        return function_call
 
 
-def test_function_no_args():
-    validate_uncompyle("fn()")
-
-@pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                    reason="need at least Python 2.7")
-def isolated_function_calls(which):
-    """
-    Returns a strategy for generating function calls, but isolated to
-    particular types of arguments, for example only positional arguments.
-
-    This can help reason about debugging errors in specific types of function
-    calls.
-
-    :param which: One of 'keyword', 'positional', 'star', 'double_star'
-
-    :return: Strategy for generating an function call isolated to specific
-             argument types.
-    """
-    kwargs = dict(
-        max_keyword_args=0,
-        max_positional_args=0,
-        max_star_args=0,
-        max_double_star_args=0,
-    )
-    kwargs['_'.join(('min', which, 'args'))] = 1
-    kwargs['_'.join(('max', which, 'args'))] = 5 if 'star' not in which else 1
-    return function_calls(**kwargs)
-
-
-with settings(max_examples=25):
+    def test_function_no_args():
+        validate_uncompyle("fn()")
 
     @pytest.mark.skipif(PYTHON_VERSION < 2.7,
                         reason="need at least Python 2.7")
-    @given(isolated_function_calls('positional'))
-    @example("fn(0)")
-    def test_function_positional_only(expr):
-        validate_uncompyle(expr)
+    def isolated_function_calls(which):
+        """
+        Returns a strategy for generating function calls, but isolated to
+        particular types of arguments, for example only positional arguments.
 
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('keyword'))
-    @example("fn(a=0)")
-    def test_function_call_keyword_only(expr):
-        validate_uncompyle(expr)
+        This can help reason about debugging errors in specific types of function
+        calls.
 
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('star'))
-    @example("fn(*items)")
-    def test_function_call_star_only(expr):
-        validate_uncompyle(expr)
+        :param which: One of 'keyword', 'positional', 'star', 'double_star'
 
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('double_star'))
-    @example("fn(**{})")
-    def test_function_call_double_star_only(expr):
-        validate_uncompyle(expr)
+        :return: Strategy for generating an function call isolated to specific
+                 argument types.
+        """
+        kwargs = dict(
+            max_keyword_args=0,
+            max_positional_args=0,
+            max_star_args=0,
+            max_double_star_args=0,
+        )
+        kwargs['_'.join(('min', which, 'args'))] = 1
+        kwargs['_'.join(('max', which, 'args'))] = 5 if 'star' not in which else 1
+        return function_calls(**kwargs)
 
 
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(w=0,m=0,**v)")
+    with settings(max_examples=25):
+
+        @pytest.mark.skipif(PYTHON_VERSION < 2.7,
+                            reason="need at least Python 2.7")
+        @given(isolated_function_calls('positional'))
+        @example("fn(0)")
+        def test_function_positional_only(expr):
+            validate_uncompyle(expr)
+
+        @pytest.mark.skipif(PYTHON_VERSION < 2.7,
+                            reason="need at least Python 2.7")
+        @given(isolated_function_calls('keyword'))
+        @example("fn(a=0)")
+        def test_function_call_keyword_only(expr):
+            validate_uncompyle(expr)
+
+        @pytest.mark.skipif(PYTHON_VERSION < 2.7,
+                            reason="need at least Python 2.7")
+        @given(isolated_function_calls('star'))
+        @example("fn(*items)")
+        def test_function_call_star_only(expr):
+            validate_uncompyle(expr)
+
+        @pytest.mark.skipif(PYTHON_VERSION < 2.7,
+                            reason="need at least Python 2.7")
+        @given(isolated_function_calls('double_star'))
+        @example("fn(**{})")
+        def test_function_call_double_star_only(expr):
+            validate_uncompyle(expr)
 
 
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(a=0,**g)")
+    @pytest.mark.xfail()
+    def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(w=0,m=0,**v)")
 
 
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*g,**j)")
+    @pytest.mark.xfail()
+    def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(a=0,**g)")
 
 
-@pytest.mark.xfail()
-def test_BUILD_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*z,u=0)")
+    @pytest.mark.xfail()
+    def test_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(*g,**j)")
 
 
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(**a)")
+    @pytest.mark.xfail()
+    def test_BUILD_MAP_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(*z,u=0)")
 
 
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(b,b,b=0,*a)")
+    @pytest.mark.xfail()
+    def test_BUILD_TUPLE_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(**a)")
 
 
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*c,v)")
+    @pytest.mark.xfail()
+    def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(b,b,b=0,*a)")
 
 
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(i=0,y=0,*p)")
+    @pytest.mark.xfail()
+    def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(*c,v)")
 
 
-@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
-@given(function_calls())
-def test_function_call(function_call):
-    validate_uncompyle(function_call)
+    @pytest.mark.xfail()
+    def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
+        validate_uncompyle("fn(i=0,y=0,*p)")
+
+
+    @pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
+    @given(function_calls())
+    def test_function_call(function_call):
+        validate_uncompyle(function_call)

pytest/test_single_compile.py

@@ -1,21 +1,22 @@
 import pytest
 from uncompyle6 import PYTHON_VERSION, deparse_code
+pytestmark = pytest.mark.skip(PYTHON_VERSION < 2.7,
+                              reason="need at least Python 2.7")
 
-@pytest.mark.skip(PYTHON_VERSION < 2.7,
-                  reason="need at least Python 2.7")
-def test_single_mode():
-    single_expressions = (
-        'i = 1',
-        'i and (j or k)',
-        'i += 1',
-        'i = j % 4',
-        'i = {}',
-        'i = []',
-        'for i in range(10):\n    i\n',
-        'for i in range(10):\n    for j in range(10):\n        i + j\n',
-        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
-    )
+if PYTHON_VERSION > 2.6:
+    def test_single_mode():
+        single_expressions = (
+            'i = 1',
+            'i and (j or k)',
+            'i += 1',
+            'i = j % 4',
+            'i = {}',
+            'i = []',
+            'for i in range(10):\n    i\n',
+            'for i in range(10):\n    for j in range(10):\n        i + j\n',
+            'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+        )
 
-    for expr in single_expressions:
-        code = compile(expr + '\n', '<string>', 'single')
-        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'
+        for expr in single_expressions:
+            code = compile(expr + '\n', '<string>', 'single')
+            assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

pytest/validate.py

@@ -6,16 +6,18 @@ import difflib
 import subprocess
 import tempfile
 import functools
-# compatability
-import six
 # uncompyle6 / xdis
-from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
+from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY, deparse_code
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
 from xdis.bytecode import Bytecode
 from xdis.main import get_opcode
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
 Bytecode = functools.partial(Bytecode, opc=opc)
 
+if PYTHON3:
+    from io import StringIO
+else:
+    from StringIO import StringIO
 
 def _dis_to_text(co):
     return Bytecode(co).dis()

requirements.txt

@@ -1,2 +1,4 @@
 # Pick up stuff from setup.py
+hypothesis==2.0.0
+pytest
 -e .

setup.py

@@ -4,12 +4,12 @@ import sys
 """Setup script for the 'uncompyle6' distribution."""
 
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 7)) or ((3, 0) <= SYS_VERSION <= (3, 0)):
-    mess = "Python Release 2.6 .. 3.7 excluding 3.0 are supported in this code branch."
+if not ((2, 6) <= SYS_VERSION  <= (3, 7)):
+    mess = "Python Release 2.6 .. 3.7 are supported in this code branch."
     if ((2, 4) <= SYS_VERSION <= (2, 7)):
         mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
                  sys.version[0:3])
-    elif SYS_VERSION < (2, 4) or (3, 0) <= SYS_VERSION:
+    elif SYS_VERSION < (2, 4):
         mess += ("\nThis package is not supported for Python version %s."
                  % sys.version[0:3])
     print(mess)

test/Makefile

@@ -1,5 +1,6 @@
 PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
-      check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
+      check-bytecode-1  check-bytecode-1.4 check-bytecode-1.5 \
+      check-bytecode-2 check-bytecode-3 \
       check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
       check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
       check-3.4 check-3.5 check-3.6 check-3.7 check-5.6 5.6 5.8 \
@@ -76,7 +77,7 @@ check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1: check-bytecode-1.5
+check-bytecode-1: check-bytecode-1.4 check-bytecode-1.5
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -93,11 +94,17 @@ check-bytecode-3:
 #: Check deparsing bytecode that works running Python 2 and Python 3
 check-bytecode: check-bytecode-3
 	$(PYTHON) test_pythonlib.py \
+	--bytecode-1.4 --bytecode-1.5 \
+        --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
 	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
         --bytecode-pypy2.7
 
 
+#: Check deparsing bytecode 1.4 only
+check-bytecode-1.4:
+	$(PYTHON) test_pythonlib.py --bytecode-1.4
+
 #: Check deparsing bytecode 1.5 only
 check-bytecode-1.5:
 	$(PYTHON) test_pythonlib.py --bytecode-1.5
@@ -263,7 +270,7 @@ check-3.4-ok:
 2.6:
 
 #: PyPy 5.0.x with Python 2.7 ...
-pypy-2.7 5.0 5.3:
+pypy-2.7 5.0 5.3 6.0:
 	$(PYTHON) test_pythonlib.py --bytecode-pypy2.7 --verify
 
 #: PyPy 2.4.x with Python 3.2 ...

test/simple_source/branching/02_ifelse_lambda.py

@@ -2,7 +2,8 @@
 # lambda's have to be more or less on a line
 
 f = lambda x: 1 if x<2 else 3
-f(5)
+assert f(3) == 3
+assert f(1) == 1
 
 # If that wasn't enough ...
 # Python will create dead code
@@ -10,10 +11,18 @@ f(5)
 # not to include the else expression
 
 g = lambda: 1 if True else 3
-g()
+assert g() == 1
 
 h = lambda: 1 if False else 3
-h()
+assert h() == 3
 
 # From 2.7 test_builtin
-lambda c: 'a' <= c <= 'z', 'Hello World'
+i = lambda c: 'a' <= c <= 'z', 'Hello World'
+assert i[0]('a') == True
+assert i[0]('A') == False
+
+# Issue #170. Bug is needing an "conditional_not_lambda" grammar rule
+# in addition the the "conditional_lambda" rule
+j = lambda a: False if not a else True
+assert j(True) == True
+assert j(False) == False

test/simple_source/bug14/00_unpack_list.py

@@ -0,0 +1,3 @@
+# Python 1.4 tzparse.py, but also appears in 1.5
+
+[tzname, delta] = __file__

test/simple_source/bug14/01_print.py

@@ -0,0 +1,6 @@
+# Python 1.4 aifc.py
+# Something weird about the final "print" and PRINT_NL_CONT followed by PRINT_NL
+def _readmark(self, markers):
+    if self._markers: print 'marker',
+    else: print 'markers',
+    print 'instead of', markers

test/simple_source/bug14/02_continue.py

@@ -0,0 +1,11 @@
+# Python 1.4 cgi.py
+# Bug was in "continue" detection.
+# 1.4 doesn't have lnotab and our CONTINUE detection is off.
+def parse_multipart(params, pdict):
+    while params:
+	if params.has_key('name'):
+	   params = None
+	else:
+	    continue
+
+    return None

test/simple_source/bug26/01_ifelse_listcomp.py

@@ -0,0 +1,4 @@
+# Bug from issue #171:  parsing "if x if a else y" inside a list comprehension on 2.7
+# This is RUNNABLE!
+assert [False, True, True, True, True] == [False if not a else True for a in range(5)]
+assert [True, False, False, False, False] == [False if a else True for a in range(5)]

test/simple_source/bug26/03_complex_and.py

@@ -0,0 +1,17 @@
+# From 2.6 test_datetime.py
+# Bug is in parsing (x is 0 or x is 1) and (y is 5 or y is 2)
+# correctly.
+
+# This code is RUNNABLE!
+result = []
+for y in (1, 2, 10):
+    x = cmp(1, y)
+    if (x is 0 or x is 1) and (y is 5 or y is 2):
+        expected = 10
+    elif y is 2:
+        expected = 2
+    else:
+        expected = 3
+    result.append(expected)
+
+assert result == [10, 2, 3]

test/simple_source/bug27+/03_for_try_raise.py

@@ -0,0 +1,12 @@
+# Code in 2.7 needing rule:
+#   try_except ::= SETUP_EXCEPT suite_stmts_opt POP_BLOCK except_handler
+# Generally we need a COME_FROM. But not in the situation below.
+
+for package in [1,2]:
+    try:
+        pass
+    except IndexError:
+        with __file__ as f:
+            pass
+    except:
+        raise

test/simple_source/bug27+/05_try_else.py

@@ -0,0 +1,11 @@
+# From Python 2.7 test_ziplib.py
+# Bug is distinguishing try from try/else.
+def testAFakeZlib(self):
+    try:
+        self.doTest()
+    except ImportError:
+        if self.compression != 3:
+            self.fail("expected test to not raise ImportError")
+    else:
+        if self.compression != 4:
+            self.fail("expected test to raise ImportError")

test/simple_source/bug275/01_while1.py

@@ -0,0 +1,9 @@
+# Issue #173. Bug is that 2.7.5 omits POP_BLOCK in
+# in later 2.7 grammar.
+#  while1stmt        ::= SETUP_LOOP l_stmts_opt JUMP_BACK COME_FROM
+#  while1stmt        ::= SETUP_LOOP l_stmts_opt CONTINUE COME_FROM
+# which is included in later code generation
+ms=0
+if ms==1:
+    while 1:
+        pass

test/simple_source/bug30/03_pop_top.py

@@ -0,0 +1,20 @@
+# From 3.0.1 __dummy_thread.py
+# bug was handling else:
+def interrupt_main():
+    """Set _interrupt flag to True to have start_new_thread raise
+    KeyboardInterrupt upon exiting."""
+    if _main:
+        raise KeyboardInterrupt
+    else:
+        global _interrupt
+        _interrupt = True
+
+# From 3.0.1 ast.py bug was mangling prototype
+# def parse(expr, filename='<unknown>', mode='exec'):
+
+# From 3.0.1 bisect
+def bisect_left(a, x, lo=0, hi=None):
+    while lo:
+        if a[mid] < x: lo = mid+1
+        else: hi = mid
+    return lo

test/simple_source/comprehension/05_dict_comp.py

@@ -1,4 +1,7 @@
+# Bug was in dictionary comprehension involving "if not"
 # Issue #162
+#
+# This code is RUNNABLE!
 def x(s):
     return {k: v
             for (k, v) in s

test/simple_source/looping/04_while1_while1.py

@@ -19,10 +19,15 @@ while 1:
 while 1:
     if __name__:
         while 1:
-            if y:
+            if __name__:
                 break
             raise RuntimeError
     elif __file__:
         x = 2
     else:
         raise RuntimeError
+
+# Degenerate case. Note: we can't run becase this causes an infinite loop.
+# Suggested in issue #172
+while 1:
+    pass

test/stdlib/compile-file-1x.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+"""byte compiles a Python 1.x program"""
+import sys
+if len(sys.argv) != 2:
+    print("Usage: compile-file.py *python-file*")
+    sys.exit(1)
+source = sys.argv[1]
+
+# assert source.endswith('.py')
+basename = source[:-3]
+
+# We do this crazy way to support Python 1.4 which
+# doesn't support version_info.
+PY_VERSION = sys.version[:3]
+
+bytecode = "%s-%s.pyc" % (basename, PY_VERSION)
+
+import py_compile
+print("# compiling %s to %s" % (source, bytecode))
+py_compile.compile(source, bytecode)
+# import os
+# os.system("../bin/uncompyle6 %s" % bytecode)

test/stdlib/compile-file.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+"""byte compiles a Python program after version 2.2 or so. Also see compile_file_1x.py"""
 import sys
 if len(sys.argv) != 2:
     print("Usage: compile-file.py *python-file*")

test/stdlib/runtests.sh

@@ -56,6 +56,8 @@ case $PYVERSION in
     2.6)
 	SKIP_TESTS=(
 	    [test_compile.py]=1  # Intermittent - sometimes works and sometimes doesn't
+	    [test_grammar.py]=1  # Need real flow control. "and" in side "or"
+	                         # "and" inside ifelse need to simulatenously work
 	    [test_grp.py]=1      # Long test - might work Control flow?
 	    [test_opcodes.py]=1
 	    [test_pwd.py]=1 # Long test - might work? Control flow?
@@ -73,12 +75,24 @@ case $PYVERSION in
 	    # .pyenv/versions/2.6.9/lib/python2.6/sre_parse.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tabnanny.pyc
 	    # .pyenv/versions/2.6.9/lib/python2.6/tarfile.pyc
+
+	    # Not getting set by bach below?
+	    [test_pprint.py]=1
+
 	)
 	if (( batch )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_aifc.py]=1
 	    SKIP_TESTS[test_array.py]=1
+
+	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+	    SKIP_TESTS[test_base64.py]=1
+
+	    # output indicates expected == output, but this fails anyway.
+	    # Maybe the underlying encoding is subtlely different so it
+	    # looks the same?
+	    SKIP_TESTS[test_pprint.py]=1
 	fi
 	;;
     2.7)
@@ -91,6 +105,7 @@ case $PYVERSION in
 	    [test_curses.py]=1  # Possibly fails on its own but not detected
 	    [test_dis.py]=1   # We change line numbers - duh!
 	    [test_doctest.py]=1 # Fails on its own
+	    [test_format.py]=1  # control flow. uncompyle2 does not have problems here
 	    [test_generators.py]=1  # control flow. uncompyle2 has problem here too
 	    [test_grammar.py]=1     # Too many stmts. Handle large stmts
 	    [test_io.py]=1 # Test takes too long to run
@@ -109,12 +124,17 @@ case $PYVERSION in
 	    [test_unicode.py]=1  # Too long to run 11 seconds
 	    [test_xpickle.py]=1 # Runs ok but takes 72 seconds
 	    [test_zipfile64.py]=1  # Runs ok but takes 204 seconds
+	    [test_zipimport.py]=1  # We can't distinguish try from try/else yet
         )
 	if (( batch )) ; then
 	    # Fails in crontab environment?
 	    # Figure out what's up here
 	    SKIP_TESTS[test_array.py]=1
 	    SKIP_TESTS[test_ast.py]=1
+	    SKIP_TESTS[test_audioop.py]=1
+
+	    # SyntaxError: Non-ASCII character '\xdd' in file test_base64.py on line 153, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
+	    SKIP_TESTS[test_base64.py]=1
 	fi
 	;;
     3.5)
@@ -171,7 +191,7 @@ if [[ -n $1 ]] ; then
 	SKIP_TESTS=()
     fi
 else
-    files=test_a*.py
+    files=test_*.py
 fi
 
 typeset -i ALL_FILES_STARTTIME=$(date +%s)

test/test_pythonlib.py

@@ -78,7 +78,7 @@ for  vers in (2.7, 3.4, 3.5, 3.6):
     test_options[key] = (os.path.join(src_dir, pythonlib), PYOC, key, vers)
     pass
 
-for  vers in (1.5,
+for  vers in (1.4, 1.5,
               2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7,
               3.0, 3.1, 3.2, 3.3,
               3.4, 3.5, 3.6, 3.7, 'pypy3.2', 'pypy2.7'):

uncompyle6/bin/uncompile.py

@@ -46,6 +46,7 @@ Debugging Options:
   --asm     -a  include byte-code         (disables --verify)
   --grammar -g  show matching grammar
   --tree    -t  include syntax tree       (disables --verify)
+  --tree++      add template rules to --tree when possible
 
 Extensions of generated files:
   '.pyc_dis' '.pyo_dis'   successfully decompiled (and verified if --verify)
@@ -61,19 +62,19 @@ from uncompyle6.version import VERSION
 
 def usage():
     print("""usage:
-   %s [--verify | --weak-verify ] [--asm] [--tree] [--grammar] [-o <path>] FILE|DIR...
+    %s [--verify | --weak-verify ] [--asm] [--tree[+]] [--grammar] [-o <path>] FILE|DIR...
    %s [--help | -h | --version | -V]
 """  % (program, program))
     sys.exit(1)
 
 
 def main_bin():
-    if not (sys.version_info[0:2] in ((2, 6), (2, 7),
+    if not (sys.version_info[0:2] in ((2, 6), (2, 7), (3, 0),
                                       (3, 1), (3, 2), (3, 3),
                                       (3, 4), (3, 5), (3, 6),
                                       (3, 7)
         )):
-        print('Error: %s requires Python 2.6-2.7, or 3.1-3.7' % program,
+        print('Error: %s requires Python 2.6-3.7' % program,
               file=sys.stderr)
         sys.exit(-1)
 
@@ -87,8 +88,10 @@ def main_bin():
 
     try:
         opts, files = getopt.getopt(sys.argv[1:], 'hagtdrVo:c:p:',
-                                    'help asm grammar linemaps recurse timestamp tree '
-                                    'fragments verify verify-run version weak-verify '
+                                    'help asm grammar linemaps recurse '
+                                    'timestamp tree tree+ '
+                                    'fragments verify verify-run version '
+                                    'weak-verify '
                                     'showgrammar'.split(' '))
     except getopt.GetoptError as e:
         print('%s: %s' % (os.path.basename(sys.argv[0]), e),  file=sys.stderr)
@@ -118,6 +121,9 @@ def main_bin():
         elif opt in ('--tree', '-t'):
             options['showast'] = True
             options['do_verify'] = None
+        elif opt in ('--tree+',):
+            options['showast'] = 'Full'
+            options['do_verify'] = None
         elif opt in ('--grammar', '-g'):
             options['showgrammar'] = True
         elif opt == '-o':

uncompyle6/disas.py

@@ -77,6 +77,19 @@ def disco_loop(disasm, queue, real_out):
             pass
         pass
 
+def disassemble_fp(fp, outstream=None):
+    """
+    disassemble Python byte-code from an open file
+    """
+    (version, timestamp, magic_int, co, is_pypy,
+     source_size) = load_from_fp(fp)
+    if type(co) == list:
+        for con in co:
+            disco(version, con, outstream)
+    else:
+        disco(version, co, outstream, is_pypy=is_pypy)
+    co = None
+
 def disassemble_file(filename, outstream=None):
     """
     disassemble Python byte-code file (.pyc)

uncompyle6/parser.py

@@ -620,18 +620,25 @@ def get_python_parser(
     # a lazy way of doing the import?
 
     if version < 3.0:
-        if version == 1.5:
-            import uncompyle6.parsers.parse15 as parse15
-            if compile_mode == 'exec':
-                p = parse15.Python15Parser(debug_parser)
-            else:
-                p = parse15.Python15ParserSingle(debug_parser)
-        elif version == 2.1:
-            import uncompyle6.parsers.parse21 as parse21
-            if compile_mode == 'exec':
-                p = parse21.Python21Parser(debug_parser)
-            else:
-                p = parse21.Python21ParserSingle(debug_parser)
+        if version < 2.2:
+            if version == 1.4:
+                import uncompyle6.parsers.parse14 as parse14
+                if compile_mode == 'exec':
+                    p = parse14.Python14Parser(debug_parser)
+                else:
+                    p = parse14.Python14ParserSingle(debug_parser)
+            elif version == 1.5:
+                import uncompyle6.parsers.parse15 as parse15
+                if compile_mode == 'exec':
+                    p = parse15.Python15Parser(debug_parser)
+                else:
+                    p = parse15.Python15ParserSingle(debug_parser)
+            elif version == 2.1:
+                import uncompyle6.parsers.parse21 as parse21
+                if compile_mode == 'exec':
+                    p = parse21.Python21Parser(debug_parser)
+                else:
+                    p = parse21.Python21ParserSingle(debug_parser)
         elif version == 2.2:
             import uncompyle6.parsers.parse22 as parse22
             if compile_mode == 'exec':

uncompyle6/parsers/parse14.py

@@ -0,0 +1,70 @@
+#  Copyright (c) 2018 Rocky Bernstein
+
+from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
+from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parsers.parse15 import Python15Parser
+
+class Python14Parser(Python15Parser):
+
+    def p_misc14(self, args):
+        """
+        # Nothing here yet, but will need to add UNARY_CALL, BINARY_CALL,
+        # RAISE_EXCEPTION, BUILD_FUNCTION, UNPACK_ARG, UNPACK_VARARG, LOAD_LOCAL,
+        # SET_FUNC_ARGS, and RESERVE_FAST
+
+        # FIXME: should check that this indeed around __doc__
+        # Possibly not strictly needed
+        stmt     ::= doc_junk
+        doc_junk ::= LOAD_CONST POP_TOP
+
+        # Not sure why later Python's omit the COME_FROM
+        jb_pop14  ::= JUMP_BACK COME_FROM POP_TOP
+
+        whileelsestmt ::= SETUP_LOOP testexpr l_stmts_opt
+                          jb_pop14
+                          POP_BLOCK else_suitel COME_FROM
+
+        print_items_nl_stmt ::= expr PRINT_ITEM_CONT print_items_opt PRINT_NEWLINE_CONT
+
+
+        # 1.4 doesn't have linenotab, and although this shouldn't
+        # be a show stopper, our CONTINUE detection is off here.
+        continue ::= JUMP_BACK
+        """
+
+    def __init__(self, debug_parser=PARSER_DEFAULT_DEBUG):
+        super(Python14Parser, self).__init__(debug_parser)
+        self.customized = {}
+
+    def customize_grammar_rules(self, tokens, customize):
+        super(Python14Parser, self).customize_grammar_rules(tokens, customize)
+        self.remove_rules("""
+        whileelsestmt ::= SETUP_LOOP testexpr l_stmts_opt
+                          jb_pop
+                          POP_BLOCK else_suitel COME_FROM
+        """)
+        self.check_reduce['doc_junk'] = 'tokens'
+
+
+    def reduce_is_invalid(self, rule, ast, tokens, first, last):
+        invalid = super(Python14Parser,
+                        self).reduce_is_invalid(rule, ast,
+                                                tokens, first, last)
+        if invalid or tokens is None:
+            return invalid
+        if rule[0] == 'doc_junk':
+            return not isinstance(tokens[first].pattr, str)
+
+
+
+class Python14ParserSingle(Python14Parser, PythonParserSingle):
+    pass
+
+if __name__ == '__main__':
+    # Check grammar
+    p = Python14Parser()
+    p.check_grammar()
+    p.dump_grammar()
+
+# local variables:
+# tab-width: 4

uncompyle6/parsers/parse15.py

@@ -2,7 +2,7 @@
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 
 from spark_parser import DEFAULT_DEBUG as PARSER_DEFAULT_DEBUG
-from uncompyle6.parser import PythonParserSingle
+from uncompyle6.parser import PythonParserSingle, nop_func
 from uncompyle6.parsers.parse21 import Python21Parser
 
 class Python15Parser(Python21Parser):
@@ -23,7 +23,18 @@ class Python15Parser(Python21Parser):
         importlist  ::= IMPORT_FROM
         """
 
-class Python15ParserSingle(Python21Parser, PythonParserSingle):
+    def customize_grammar_rules(self, tokens, customize):
+        super(Python15Parser, self).customize_grammar_rules(tokens, customize)
+        for i, token in enumerate(tokens):
+            opname = token.kind
+            opname_base = opname[:opname.rfind('_')]
+
+            if opname_base == 'UNPACK_LIST':
+                self.addRule("store ::= unpack_list", nop_func)
+
+
+
+class Python15ParserSingle(Python15Parser, PythonParserSingle):
     pass
 
 if __name__ == '__main__':

uncompyle6/parsers/parse2.py

@@ -274,12 +274,12 @@ class Python2Parser(PythonParser):
              'LOAD',      'LOOKUP',     'MAKE',      'SETUP',
              'RAISE',     'UNPACK'))
 
-        # Opcode names in the custom_ops_seen set have rules that get added
+        # Opcode names in the custom_seen_ops set have rules that get added
         # unconditionally and the rules are constant. So they need to be done
         # only once and if we see the opcode a second we don't have to consider
         # adding more rules.
         #
-        custom_ops_seen = set()
+        custom_seen_ops = set()
 
         for i, token in enumerate(tokens):
             opname = token.kind
@@ -287,14 +287,13 @@ class Python2Parser(PythonParser):
             # Do a quick breakout before testing potentially
             # each of the dozen or so instruction in if elif.
             if (opname[:opname.find('_')] not in customize_instruction_basenames
-                    or opname in custom_ops_seen):
+                    or opname in custom_seen_ops):
                 continue
 
             opname_base = opname[:opname.rfind('_')]
 
             # The order of opname listed is roughly sorted below
             if opname_base in ('BUILD_LIST', 'BUILD_SET', 'BUILD_TUPLE'):
-                v = token.attr
                 collection = opname_base[opname_base.find('_')+1:].lower()
                 rule = '%s ::= %s%s' % (collection, (token.attr * 'expr '), opname)
                 self.add_unique_rules([
@@ -355,32 +354,32 @@ class Python2Parser(PythonParser):
                        + 'expr ' * nak + opname
             elif opname == 'CONTINUE_LOOP':
                 self.addRule('continue ::= CONTINUE_LOOP', nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_ATTR':
                 self.addRule('del_stmt ::= expr DELETE_ATTR', nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_DEREF':
                 self.addRule("""
                    stmt           ::= del_deref_stmt
                    del_deref_stmt ::= DELETE_DEREF
                    """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'DELETE_SUBSCR':
                 self.addRule("""
                     del_stmt ::= delete_subscr
                     delete_subscr ::= expr expr DELETE_SUBSCR
                    """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'GET_ITER':
                 self.addRule("""
                     expr      ::= get_iter
                     attribute ::= expr GET_ITER
                     """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base in ('DUP_TOPX', 'RAISE_VARARGS'):
                 # FIXME: remove these conditions if they are not needed.
@@ -395,7 +394,6 @@ class Python2Parser(PythonParser):
                     """, nop_func)
                 continue
             elif opname == 'JUMP_IF_NOT_DEBUG':
-                v = token.attr
                 self.addRule("""
                     jmp_true_false ::= POP_JUMP_IF_TRUE
                     jmp_true_false ::= POP_JUMP_IF_FALSE
@@ -413,18 +411,18 @@ class Python2Parser(PythonParser):
                   expr      ::= attribute
                   attribute ::= expr LOAD_ATTR
                   """, nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOAD_LISTCOMP':
                 self.addRule("expr ::= listcomp", nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOAD_SETCOMP':
                 self.add_unique_rules([
                     "expr ::= set_comp",
                     "set_comp ::= LOAD_SETCOMP MAKE_FUNCTION_0 expr GET_ITER CALL_FUNCTION_1"
                     ], customize)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'LOOKUP_METHOD':
                 # A PyPy speciality - DRY with parse3
@@ -433,7 +431,7 @@ class Python2Parser(PythonParser):
                              attribute ::= expr LOOKUP_METHOD
                              """,
                              nop_func)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base == 'MAKE_FUNCTION':
                 if i > 0 and tokens[i-1] == 'LOAD_LAMBDA':
@@ -482,7 +480,7 @@ class Python2Parser(PythonParser):
                         "try_except_pypy ::= SETUP_EXCEPT suite_stmts_opt except_handler_pypy",
                         "except_handler_pypy ::= COME_FROM except_stmts END_FINALLY COME_FROM"
                         ], customize)
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname == 'SETUP_FINALLY':
                 if 'PyPy' in customize:
@@ -491,13 +489,13 @@ class Python2Parser(PythonParser):
                         tryfinallystmt_pypy ::= SETUP_FINALLY suite_stmts_opt COME_FROM_FINALLY
                                                 suite_stmts_opt END_FINALLY""", nop_func)
 
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 continue
             elif opname_base in ('UNPACK_TUPLE', 'UNPACK_SEQUENCE'):
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 rule = 'unpack ::= ' + opname + ' store' * token.attr
             elif opname_base == 'UNPACK_LIST':
-                custom_ops_seen.add(opname)
+                custom_seen_ops.add(opname)
                 rule = 'unpack_list ::= ' + opname + ' store' * token.attr
             else:
                 continue

uncompyle6/parsers/parse23.py

@@ -1,4 +1,4 @@
-#  Copyright (c) 2016-2017 Rocky Bernstein
+#  Copyright (c) 2016-2018 Rocky Bernstein
 #  Copyright (c) 2000-2002 by hartmut Goebel <hartmut@goebel.noris.de>
 #  Copyright (c) 1999 John Aycock
 
@@ -32,9 +32,6 @@ class Python23Parser(Python24Parser):
         while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        POP_TOP POP_BLOCK COME_FROM
 
-        while1stmt ::= _while1test l_stmts_opt JUMP_BACK
-                       POP_TOP POP_BLOCK
-
         while1stmt ::= _while1test l_stmts_opt JUMP_BACK
                        COME_FROM POP_TOP POP_BLOCK COME_FROM
 

uncompyle6/parsers/parse25.py

@@ -75,8 +75,12 @@ class Python25Parser(Python26Parser):
         setupwithas      ::= DUP_TOP LOAD_ATTR ROT_TWO LOAD_ATTR CALL_FUNCTION_0 setup_finally
         stmt             ::= classdefdeco
         stmt             ::= conditional_lambda
+        stmt             ::= conditional_not_lambda
         conditional_lambda ::= expr jmp_false_then expr return_if_lambda
                                return_stmt_lambda LAMBDA_MARKER
+        conditional_not_lambda
+                           ::= expr jmp_true_then expr return_if_lambda
+                               return_stmt_lambda LAMBDA_MARKER
         """)
         super(Python25Parser, self).customize_grammar_rules(tokens, customize)
         if self.version == 2.5: