Merge branch 'master' into python-2.4

in a hacky way

.gitignore

@@ -21,4 +21,3 @@
 ChangeLog
 __pycache__
 build
-nohup.out

.travis.yml

@@ -3,13 +3,7 @@ language: python
 sudo: false
 
 python:
-  - '3.5'
-  - '2.7'
-  - '2.6'
-  - '3.3'
-  - '3.4'
-  - '3.2'
-  - '3.6'
+  - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
 install:
 - pip install -e .

Makefile

@@ -27,18 +27,21 @@ check:
 check-short: pytest
 	$(MAKE) -C test check-short
 
-# Note for 2.6 use <=3.0.1 see requirements-dev.txt
 #: Tests for Python 2.7, 3.3 and 3.4
-check-2.6 check-2.7 check-3.3 check-3.4 check-3.5: pytest
+check-2.7 check-3.3 check-3.4: pytest
 	$(MAKE) -C test $@
 
 #: Tests for Python 3.2 and 3.5 - pytest doesn't work here
 # Or rather 3.5 doesn't work not on Travis
-check-3.0 check-3.1 check-3.2 check-3.6:
+check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
 	$(MAKE) -C test $@
 
 check-3.7: pytest
 
+#:Tests for Python 2.6 (doesn't have pytest)
+check-2.4 check-2.5 check-2.6:
+	$(MAKE) -C test $@
+
 #:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
 # Skip for now
 2.6 5.0 5.3 5.6 5.8:

NEWS

@@ -1,17 +1,3 @@
-uncompyle6 3.1.3 2018-04-16
-
-- Add some Python 3.7 rules, such as for handling LOAD_METHOD (not complete)
-- Fix some fragment bugs
-- small doc changes
-
-uncompyle6 3.1.2 2018-04-08 Eastern Orthodox Easter
-
-- Python 3.x subclass and call parsing fixes
-- Allow/note running on Python 3.1
-- improve 3.5+ BUILD_MAP_UNPACK
-- DRY instruction building code between 2.x and 3.x
-- expand testing
-
 uncompyle6 3.1.1 2018-04-01 Easter April Fool's
 
 Jesus on Friday's New York Times puzzle: "I'm stuck on 2A"

README.rst

@@ -52,17 +52,8 @@ You get the idea. This code pulls all of these forks together and
 *moves forward*. There is some serious refactoring and cleanup in this
 code base over those old forks.
 
-This demonstrably does the best in decompiling Python across all
-Python versions. And even when there is another project that only
-provides decompilation for subset of Python versions, we generally do
-demonstrably better for those as well.
-
-How can we tell? By taking Python bytecode that comes distributed with
-that version of Python and decompiling these.  Among htose that
-successfully decompile, we can then make sure the resulting programs
-are syntactically correct by running the Python interpreter for that
-bytecode version.  Finally, in cases where the program has a test for
-itself, we can run the check on the decompiled code.
+This project has the most complete support for Python 3.3 and above
+and the best all-around Python support.
 
 We are serious about testing, and use automated processes to find
 bugs. In the issue trackers for other decompilers, you will find a
@@ -145,26 +136,26 @@ All of the Python decompilers that I have looked at have problems
 decompiling Python's control flow. In some cases we can detect an
 erroneous decompilation and report that.
 
-In older versions of Python it was possible to verify bytecode by
-decompiling bytecode, and then compiling using the Python interpreter
-for that bytecode version. Having done this the bytecode produced
-could be compared with the original bytecode. However as Python's code
-generation got better, this is no longer feasible.
+*Verification* is the process of decompiling bytecode, compiling with
+a Python for that bytecode version, and then comparing the bytecode
+produced by the decompiled/compiled program. Some allowance is made
+for inessential differences. But other semantically equivalent
+differences are not caught. For example ``1 and 0`` is decompiled to
+the equivalent ``0``; remnants of the first true evaluation (1) is
+lost when Python compiles this. When Python next compiles ``0`` the
+resulting code is simpler.
 
-There is a kind of *weak verification* that we use that doesn't check
-bytecode for equivalence but does check to see if the resulting
-decompiled source is a valid Python program by running the Python
-interpreter. Because the Python language has changed so much, for best
-results you should use the same Python version in checking as was used
-in creating the bytecode.
+*Weak Verification*
+on the other hand doesn't check bytecode for equivalence but does
+check to see if the resulting decompiled source is a valid Python
+program by running the Python interpreter. Because the Python language
+has changed so much, for best results you should use the same Python
+Version in checking as used in the bytecode.
 
-There are however an interesting class of these programs that is
-readily available give stronger verification: those programs that
-when run check some computation, or even better themselves.
-
-And already Python has a set of programs like this: the test suite
-for the standard library that comes with Python. We have some
-code in `test/stdlib` to facilitate this kind of checking.
+Finally, we have automated running the standard Python tests after
+first compiling and decompiling the test program. Results here are a
+bit weak (if not better than most other Python decompilers). But over
+time this will probably get better.
 
 Python support is strongest in Python 2 for 2.7 and drops off as you
 get further away from that. Support is also probably pretty good for
@@ -212,7 +203,7 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for Python 3 is a bit lacking though.
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
 * https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
 * The HISTORY_ file.

__pkginfo__.py

@@ -35,8 +35,6 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 2.5',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
-                'Programming Language :: Python :: 3.1',
-                'Programming Language :: Python :: 3.2',
                 'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
@@ -56,8 +54,7 @@ entry_points       = {
     ]}
 ftp_url            = None
 install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
-                      'xdis >= 3.8.0, < 3.9.0', 'six']
-
+                      'xdis >= 3.7.0, < 3.8.0']
 license            = 'GPL3'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

admin-tools/how-to-make-a-release.txt

@@ -0,0 +1,46 @@
+git pull
+
+Change version in uncompyle6/version.py
+source uncompyle6/version.py
+echo $VERSION
+git commit -m"Get ready for release $VERSION" .
+
+Update ChangeLog:
+  make ChangeLog
+
+Update NEWS from ChangeLog
+make check
+
+git commit --amend .
+
+git push
+
+Make sure pyenv is running
+# Pyenv
+
+source admin-tools/check-newer-versions.sh
+
+
+# Switch to python-2.4 and build that first...
+source admin-tools/setup-python-2.4
+
+rm ChangeLog
+git merge master
+
+Update NEWS from master branch
+
+git commit -m"Get ready for release $VERSION" .
+
+source admin-tools/check-older-versions.sh
+source admin-tools/check-newer-versions.sh
+
+make-dist-older.sh
+
+git tag release-python-2.4-$VERSION
+
+./make-dist-newer.sh
+
+git tag release-$VERSION
+
+
+twine upload dist/uncompyle6-${VERSION}*

admin-tools/pycdc-runtests.sh

@@ -1,15 +0,0 @@
-#!/bin/bash
-# Use pycdc to run our test/bytecode* test suite
-bs=${BASH_SOURCE[0]}
-testdir=$(dirname $bs)/../test
-fulldir=$(readlink -f $testdir)
-cd $fulldir
-for dir in bytecode_* ; do
-    echo ========= $dir ================
-    cd $fulldir/$dir
-    for file in *.pyc; do
-	if ! pycdc $file > /dev/null ; then
-	    echo -----  $dir/$file ------
-	fi
-    done
-done

admin-tools/pyenv-newer-versions

@@ -5,4 +5,4 @@ if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.5 3.6.5 2.6.9 3.3.7 2.7.14 3.2.6 3.1.5 3.4.8'
+export PYVERSIONS='3.5.5 3.6.4 2.6.9 3.3.7 2.7.14 3.4.8'

admin-tools/uncompyle2-runtests.sh

@@ -1,32 +0,0 @@
-#!/bin/bash
-# Use pycdc to run our test/bytecode_2.7* test suite
-bs=${BASH_SOURCE[0]}
-topdir=$(dirname $bs)/..
-(cd $topdir && pyenv local 2.7.14)
-testdir=$topdir/test
-fulldir=$(readlink -f $testdir)
-cd $fulldir
-
-for bytecode in bytecode_2.7/*.pyc ; do
-    echo $bytecode
-    uncompyle2 $bytecode > /dev/null
-    echo ================ $bytecode rc: $? ==============
-done
-
-tmpdir=/tmp/test-2.7
-( cd bytecode_2.7_run &&
-      mkdir $tmpdir || true
-      for bytecode in *.pyc ; do
-	  shortname=$(basename $bytecode .pyc)
-	  echo $bytecode
-	  py_file=${tmpdir}/${shortname}.py
-	  typeset -i rc=0
-	  uncompyle2 $bytecode > $py_file
-	  rc=$?
-	  if (( rc == 0 )); then
-	      python $py_file
-	      rc=$?
-	  fi
-	  echo ================ $bytecode rc: $rc ==============
-      done
-)

admin-tools/update-sources.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+cd $(dirname ${BASH_SOURCE[0]})/..
+git pull

circle.yml

@@ -11,4 +11,4 @@ dependencies:
 test:
     override:
         - python ./setup.py develop && make check-2.7
-        - cd ./test/stdlib && pyenv local 2.7.10 && bash ./runtests.sh 'test_[p-z]*.py'
+        - cd ./test/stdlib && pyenv local 2.7.10 && bash ./runtests.sh 'test_[f-i]*.py'

pytest/test_deparse.py

@@ -30,7 +30,7 @@ def list_comp():
     [y for y in range(3)]
 
 def get_parsed_for_fn(fn):
-    code = fn.__code__ if PYTHON3 else fn.func_code
+    code = fn.func_code
     return deparse(code, version=PYTHON_VERSION)
 
 def check_expect(expect, parsed, fn_name):

pytest/test_docstring.py

@@ -10,7 +10,7 @@ else:
     maxint = sys.maxint
 from uncompyle6.semantics.helper import print_docstring
 
-class PrintFake():
+class PrintFake:
     def __init__(self):
         self.pending_newlines = 0
         self.f = StringIO()

pytest/test_fjt.py

@@ -1,6 +1,8 @@
 #!/usr/bin/env python
 from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
+from xdis.bytecode import Bytecode
+from array import array
 def bug(state, slotstate):
     if state:
         if slotstate is not None:
@@ -20,10 +22,17 @@ def bug_loop(disassemble, tb=None):
     disassemble(tb)
 
 def test_if_in_for():
-    code = bug.__code__
+    code = bug.func_code
     scan = get_scanner(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
-        scan.build_instructions(code)
+        n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
+        scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
+        scan.build_prev_op(n)
         fjt = scan.find_jump_targets(False)
 
         ## FIXME: the data below is wrong.
@@ -38,7 +47,14 @@ def test_if_in_for():
         #    {'start': 62, 'end': 63, 'type': 'for-else'}]
 
         code = bug_loop.__code__
-        scan.build_instructions(code)
+        n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
+        scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
+        scan.build_prev_op(n)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt = scan.find_jump_targets(False)
         assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
         assert scan.structs == [
@@ -52,7 +68,14 @@ def test_if_in_for():
             {'start': 48, 'end': 67, 'type': 'while-loop'}]
 
     elif 3.2 < PYTHON_VERSION <= 3.4:
-        scan.build_instructions(code)
+        bytecode = Bytecode(code, scan.opc)
+        scan.code = array('B', code.co_code)
+        scan.lines = scan.build_lines_data(code)
+        scan.build_prev_op()
+        scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \
@@ -62,6 +85,5 @@ def test_if_in_for():
            {'end': 59, 'type': 'for-loop', 'start': 31},
            {'end': 63, 'type': 'for-else', 'start': 62}]
     else:
-        print("FIXME: should fix for %s" % PYTHON_VERSION)
-        assert True
+        assert True, "FIXME: should note fixed"
     return

pytest/test_fstring.py

@@ -1,150 +0,0 @@
-# std
-import os
-# test
-import pytest
-import hypothesis
-from hypothesis import strategies as st
-# uncompyle6
-from uncompyle6 import PYTHON_VERSION, deparse_code
-
-
-@st.composite
-def expressions(draw):
-    # todo : would be nice to generate expressions using hypothesis however
-    # this is pretty involved so for now just use a corpus of expressions
-    # from which to select.
-    return draw(st.sampled_from((
-        'abc',
-        'len(items)',
-        'x + 1',
-        'lineno',
-        'container',
-        'self.attribute',
-        'self.method()',
-        # These expressions are failing, I think these are control
-        # flow problems rather than problems with FORMAT_VALUE,
-        # however I need to confirm this...
-        #'sorted(items, key=lambda x: x.name)',
-        #'func(*args, **kwargs)',
-        #'text or default',
-        #'43 if life_the_universe and everything else None'
-    )))
-
-
-@st.composite
-def format_specifiers(draw):
-    """
-    Generate a valid format specifier using the rules:
-
-    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
-    fill        ::=  <any character>
-    align       ::=  "<" | ">" | "=" | "^"
-    sign        ::=  "+" | "-" | " "
-    width       ::=  integer
-    precision   ::=  integer
-    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
-
-    See https://docs.python.org/2/library/string.html
-
-    :param draw: Let hypothesis draw from other strategies.
-
-    :return: An example format_specifier.
-    """
-    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
-    fill = draw(st.one_of(alphabet_strategy, st.none()))
-    align = draw(st.sampled_from(list('<>=^')))
-    fill_align = (fill + align or '') if fill else ''
-
-    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
-    can_have_sign = type_ in 'deEfFgGnoxX%'
-    can_have_comma = type_ in 'deEfFgG%'
-    can_have_precision = type_ in 'fFgG'
-    can_have_pound = type_ in 'boxX%'
-    can_have_zero = type_ in 'oxX'
-
-    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
-    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
-    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
-
-    int_strategy = st.integers(min_value=1, max_value=1000)
-
-    width = draw(st.one_of(int_strategy, st.none()))
-    width = str(width) if width is not None else ''
-
-    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
-    if can_have_precision:
-        precision = draw(st.one_of(int_strategy, st.none()))
-        precision = '.' + str(precision) if precision else ''
-    else:
-        precision = ''
-
-    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
-
-
-@st.composite
-def fstrings(draw):
-    """
-    Generate a valid f-string.
-    See https://www.python.org/dev/peps/pep-0498/#specification
-
-    :param draw: Let hypothsis draw from other strategies.
-
-    :return: A valid f-string.
-    """
-    character_strategy = st.characters(
-        blacklist_characters='\r\n\'\\s{}',
-        min_codepoint=1,
-        max_codepoint=1000,
-    )
-    is_raw = draw(st.booleans())
-    integer_strategy = st.integers(min_value=0, max_value=3)
-    expression_count = draw(integer_strategy)
-    content = []
-    for _ in range(expression_count):
-        expression = draw(expressions())
-        conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
-        has_specifier = draw(st.booleans())
-        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
-        content.append('{{{}{}}}'.format(expression, conversion, specifier))
-        content.append(draw(st.text(character_strategy)))
-    content = ''.join(content)
-    return "f{}'{}'".format('r' if is_raw else '', content)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(format_specifiers())
-def test_format_specifiers(format_specifier):
-    """Verify that format_specifiers generates valid specifiers"""
-    try:
-        exec('"{:' + format_specifier + '}".format(0)')
-    except ValueError as e:
-        if 'Unknown format code' not in str(e):
-            raise
-
-
-def run_test(text):
-    hypothesis.assume(len(text))
-    hypothesis.assume("f'{" in text)
-    expr = text + '\n'
-    code = compile(expr, '<string>', 'single')
-    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
-    recompiled = compile(deparsed.text, '<string>', 'single')
-    if recompiled != code:
-        assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(fstrings())
-def test_uncompyle_fstring(fstring):
-    """Verify uncompyling fstring bytecode"""
-    run_test(fstring)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@pytest.mark.parametrize('fstring', [
-    "f'{abc}{abc!s}'",
-    "f'{abc}0'",
-])
-def test_uncompyle_direct(fstring):
-    """useful for debugging"""
-    run_test(fstring)

pytest/test_function_call.py

@@ -1,184 +0,0 @@
-# std
-import string
-# 3rd party
-from hypothesis import given, assume, example, settings, strategies as st
-import pytest
-# uncompyle
-from validate import validate_uncompyle
-from test_fstring import expressions
-from uncompyle6 import PYTHON_VERSION
-
-alpha = st.sampled_from(string.ascii_lowercase)
-numbers = st.sampled_from(string.digits)
-alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
-
-
-@st.composite
-def function_calls(draw,
-                   min_keyword_args=0, max_keyword_args=5,
-                   min_positional_args=0, max_positional_args=5,
-                   min_star_args=0, max_star_args=1,
-                   min_double_star_args=0, max_double_star_args=1):
-    """
-    Strategy factory for generating function calls.
-
-    :param draw: Callable which draws examples from other strategies.
-
-    :return: The function call text.
-    """
-    st_positional_args = st.lists(
-        alpha,
-        min_size=min_positional_args,
-        max_size=max_positional_args
-    )
-    st_keyword_args = st.lists(
-        alpha,
-        min_size=min_keyword_args,
-        max_size=max_keyword_args
-    )
-    st_star_args = st.lists(
-        alpha,
-        min_size=min_star_args,
-        max_size=max_star_args
-    )
-    st_double_star_args = st.lists(
-        alpha,
-        min_size=min_double_star_args,
-        max_size=max_double_star_args
-    )
-
-    positional_args = draw(st_positional_args)
-    keyword_args = draw(st_keyword_args)
-    st_values = st.lists(
-        expressions(),
-        min_size=len(keyword_args),
-        max_size=len(keyword_args)
-    )
-    keyword_args = [
-        x + '=' + e
-        for x, e in
-        zip(keyword_args, draw(st_values))
-    ]
-    star_args = ['*' + x for x in draw(st_star_args)]
-    double_star_args = ['**' + x for x in draw(st_double_star_args)]
-
-    arguments = positional_args + keyword_args + star_args + double_star_args
-    draw(st.randoms()).shuffle(arguments)
-    arguments = ','.join(arguments)
-
-    function_call = 'fn({arguments})'.format(arguments=arguments)
-    try:
-        # TODO: Figure out the exact rules for ordering of positional, keyword,
-        # star args, double star args and in which versions the various
-        # types of arguments are supported so we don't need to check that the
-        # expression compiles like this.
-        compile(function_call, '<string>', 'single')
-    except:
-        assume(False)
-    return function_call
-
-
-def test_function_no_args():
-    validate_uncompyle("fn()")
-
-@pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                    reason="need at least Python 2.7")
-def isolated_function_calls(which):
-    """
-    Returns a strategy for generating function calls, but isolated to
-    particular types of arguments, for example only positional arguments.
-
-    This can help reason about debugging errors in specific types of function
-    calls.
-
-    :param which: One of 'keyword', 'positional', 'star', 'double_star'
-
-    :return: Strategy for generating an function call isolated to specific
-             argument types.
-    """
-    kwargs = dict(
-        max_keyword_args=0,
-        max_positional_args=0,
-        max_star_args=0,
-        max_double_star_args=0,
-    )
-    kwargs['_'.join(('min', which, 'args'))] = 1
-    kwargs['_'.join(('max', which, 'args'))] = 5 if 'star' not in which else 1
-    return function_calls(**kwargs)
-
-
-with settings(max_examples=25):
-
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('positional'))
-    @example("fn(0)")
-    def test_function_positional_only(expr):
-        validate_uncompyle(expr)
-
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('keyword'))
-    @example("fn(a=0)")
-    def test_function_call_keyword_only(expr):
-        validate_uncompyle(expr)
-
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('star'))
-    @example("fn(*items)")
-    def test_function_call_star_only(expr):
-        validate_uncompyle(expr)
-
-    @pytest.mark.skipif(PYTHON_VERSION < 2.7,
-                        reason="need at least Python 2.7")
-    @given(isolated_function_calls('double_star'))
-    @example("fn(**{})")
-    def test_function_call_double_star_only(expr):
-        validate_uncompyle(expr)
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(w=0,m=0,**v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(a=0,**g)")
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*g,**j)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*z,u=0)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(**a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(b,b,b=0,*a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*c,v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(i=0,y=0,*p)")
-
-
-@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
-@given(function_calls())
-def test_function_call(function_call):
-    validate_uncompyle(function_call)

pytest/test_grammar.py

@@ -20,20 +20,13 @@ def test_grammar():
     # We have custom rules that create the below
     expect_lhs = set(['pos_arg', 'get_iter', 'attribute'])
 
-    unused_rhs = set(['list', 'mkfunc',
+    unused_rhs = set(['list', 'mkfunc', 'dict',
                       'mklambda',
                       'unpack',])
     expect_right_recursive = set([('designList',
                                    ('store', 'DUP_TOP', 'designList'))])
-
-    if PYTHON_VERSION != 3.7:
-        unused_rhs.add('call')
-
-    if PYTHON_VERSION > 2.6:
-        expect_lhs.add('kvlist')
-        expect_lhs.add('kv3')
-        unused_rhs.add('dict')
-
+    expect_lhs.add('kvlist')
+    expect_lhs.add('kv3')
     if PYTHON3:
         expect_lhs.add('load_genexpr')
 
@@ -44,12 +37,16 @@ def test_grammar():
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
+            unused_rhs.add('call')
             unused_rhs.add("dict_comp")
             unused_rhs.add("classdefdeco1")
-            if PYTHON_VERSION != 3.6:
-                if PYTHON_VERSION in (3.5, 3.7):
+            if PYTHON_VERSION < 3.6:
+                # 3.6 has at least one non-custom call rule
+                # the others don't
+                unused_rhs.add('call')
+                if PYTHON_VERSION == 3.5:
                     expect_right_recursive.add((('l_stmts',
-                                                 ('lastl_stmt', 'come_froms', 'l_stmts'))))
+                                                 ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
                     pass
                 pass
             else:
@@ -60,6 +57,7 @@ def test_grammar():
         pass
     else:
         expect_lhs.add('kwarg')
+        unused_rhs.add('call')
 
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)
@@ -87,8 +85,6 @@ def test_grammar():
             """.split())
     if 2.6 <= PYTHON_VERSION <= 2.7:
         opcode_set = set(s.opc.opname).union(ignore_set)
-        if PYTHON_VERSION == 2.6:
-            opcode_set.add("THEN")
         check_tokens(tokens, opcode_set)
     elif PYTHON_VERSION == 3.4:
         ignore_set.add('LOAD_CLASSNAME')

pytest/test_pysource.py

@@ -8,12 +8,8 @@ from uncompyle6.semantics.consts import (
 
 if PYTHON3:
     from io import StringIO
-    def iteritems(d):
-        return d.items()
 else:
     from StringIO import StringIO
-    def iteritems(d):
-        return d.iteritems()
 
 from uncompyle6.semantics.pysource import SourceWalker as SourceWalker
 
@@ -30,7 +26,7 @@ def test_template_engine():
     # FIXME: and so on...
 
 from uncompyle6.semantics.consts import (
-    TABLE_DIRECT, TABLE_R,
+    TABLE_R, TABLE_DIRECT,
     )
 
 from uncompyle6.semantics.fragments import (
@@ -44,7 +40,7 @@ def test_tables():
             (TABLE_DIRECT, 'TABLE_DIRECT', False),
             (TABLE_R, 'TABLE_R', False),
             (TABLE_DIRECT_FRAGMENT, 'TABLE_DIRECT_FRAGMENT', True)):
-        for k, entry in iteritems(t):
+        for k, entry in t.iteritems():
             if k in skip_for_now:
                 continue
             fmt = entry[0]

pytest/test_single_compile.py

@@ -1,21 +1,19 @@
-import pytest
 from uncompyle6 import PYTHON_VERSION, deparse_code
 
-@pytest.mark.skip(PYTHON_VERSION < 2.7,
-                  reason="need at least Python 2.7")
-def test_single_mode():
-    single_expressions = (
-        'i = 1',
-        'i and (j or k)',
-        'i += 1',
-        'i = j % 4',
-        'i = {}',
-        'i = []',
-        'for i in range(10):\n    i\n',
-        'for i in range(10):\n    for j in range(10):\n        i + j\n',
-        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
-    )
+if PYTHON_VERSION >= 2.5:
+    def test_single_mode():
+        single_expressions = (
+            'i = 1',
+            'i and (j or k)',
+            'i += 1',
+            'i = j % 4',
+            'i = {}',
+            'i = []',
+            'for i in range(10):\n    i\n',
+            'for i in range(10):\n    for j in range(10):\n        i + j\n',
+            'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+        )
 
-    for expr in single_expressions:
-        code = compile(expr + '\n', '<string>', 'single')
-        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'
+        for expr in single_expressions:
+            code = compile(expr + '\n', '<string>', 'single')
+            assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

pytest/testdata/if-2.7.right

@@ -9,4 +9,4 @@
           12  JUMP_FORWARD          0  'to 15'
         15_0  COME_FROM            12  '12'
           15  LOAD_CONST            0  None
-          18  RETURN_VALUE     
+          18  RETURN_VALUE

pytest/testdata/ifelse-2.7.right

@@ -12,4 +12,4 @@
           18  STORE_NAME            2  'd'
         21_0  COME_FROM            12  '12'
           21  LOAD_CONST            2  None
-          24  RETURN_VALUE     
+          24  RETURN_VALUE

pytest/validate.py

@@ -1,24 +1,25 @@
-# future
-from __future__ import print_function
 # std
 import os
 import difflib
 import subprocess
 import tempfile
-import functools
-# compatability
-import six
+
+from StringIO import StringIO
 # uncompyle6 / xdis
 from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
 from xdis.bytecode import Bytecode
 from xdis.main import get_opcode
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
-Bytecode = functools.partial(Bytecode, opc=opc)
 
+try:
+    import functools
+    Bytecode = functools.partial(Bytecode, opc=opc)
+    def _dis_to_text(co):
+        return Bytecode(co).dis()
+except:
+    pass
 
-def _dis_to_text(co):
-    return Bytecode(co).dis()
 
 
 def print_diff(original, uncompyled):
@@ -42,8 +43,11 @@ def print_diff(original, uncompyled):
         print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
         diff = difflib.HtmlDiff().make_table(*args)
 
-    with tempfile.NamedTemporaryFile(delete=False) as f:
+    f = tempfile.NamedTemporaryFile(delete=False)
+    try:
         f.write(str(diff).encode('utf-8'))
+    finally:
+        f.close()
 
     try:
         print()
@@ -60,8 +64,7 @@ def print_diff(original, uncompyled):
         print('\nFor side by side diff install elinks')
         diff = difflib.Differ().compare(original_lines, uncompyled_lines)
         print('\n'.join(diff))
-    finally:
-        os.unlink(f.name)
+    os.unlink(f.name)
 
 
 def are_instructions_equal(i1, i2):
@@ -123,8 +126,9 @@ def validate_uncompyle(text, mode='exec'):
     original_text = text
 
     deparsed = deparse_code(PYTHON_VERSION, original_code,
+
                             compile_mode=mode,
-                            out=six.StringIO(),
+                            out=StringIO(),
                             is_pypy=IS_PYPY)
     uncompyled_text = deparsed.text
     uncompyled_code = compile(uncompyled_text, '<string>', 'exec')

requirements-dev.txt

@@ -1,3 +1,3 @@
-pytest>=3.0.0,<=3.0.1
+pytest>=3.0.0
 flake8
-hypothesis<=3.0.0
+hypothesis<=3.8.3

setup.py

@@ -1,16 +1,16 @@
 #!/usr/bin/env python
-import sys
-
 """Setup script for the 'uncompyle6' distribution."""
 
+import sys
+
 SYS_VERSION = sys.version_info[0:2]
-if not ((2, 6) <= SYS_VERSION  <= (3, 7)) or ((3, 0) <= SYS_VERSION <= (3, 0)):
-    mess = "Python Release 2.6 .. 3.7 excluding 3.0 are supported in this code branch."
-    if ((2, 4) <= SYS_VERSION <= (2, 7)):
-        mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
+if not ((2, 4) <= SYS_VERSION  <= (2, 7)):
+    mess = "Python Release 2.4 .. 2.7 are supported in this code branch."
+    if ((3, 2) <= SYS_VERSION  <= (3, 7)):
+        mess += ("\nFor your Python, version %s, use the master code/branch." %
                  sys.version[0:3])
-    elif SYS_VERSION < (2, 4) or (3, 0) <= SYS_VERSION:
-        mess += ("\nThis package is not supported for Python version %s."
+    else:
+        mess += ("\nThis package is not supported before Python 2.4. Your Python version is %s."
                  % sys.version[0:3])
     print(mess)
     raise Exception(mess)

test-unit/test_grammar.py

@@ -0,0 +1,55 @@
+import re
+import unittest
+from uncompyle6 import PYTHON_VERSION, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser, python_parser
+
+class TestGrammar(unittest.TestCase):
+    def test_grammar(self):
+
+        def check_tokens(tokens, opcode_set):
+            remain_tokens = set(tokens) - opcode_set
+            remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+            remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+            remain_tokens = set(remain_tokens) - opcode_set
+            self.assertEqual(remain_tokens, set([]),
+                    "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar()))
+
+        p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+        (lhs, rhs, tokens,
+         right_recursive, dup_rhs) = p.check_sets()
+        expect_lhs = set(['expr1024', 'pos_arg'])
+        unused_rhs = set(['list', 'call', 'mkfunc',
+                          'mklambda',
+                          'unpack',])
+
+        expect_right_recursive = frozenset([('designList',
+                                             ('store', 'DUP_TOP', 'designList'))])
+        expect_lhs.add('kwarg')
+
+        self.assertEqual(expect_lhs, set(lhs))
+        self.assertEqual(unused_rhs, set(rhs))
+        self.assertEqual(expect_right_recursive, right_recursive)
+
+        expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                    ('LOAD_CONST',),
+                                    ('JUMP_BACK',), ('JUMP_FORWARD',)])
+
+        reduced_dup_rhs = {}
+        for k in dup_rhs:
+            if k not in expect_dup_rhs:
+                reduced_dup_rhs[k] = dup_rhs[k]
+                pass
+            pass
+        for k in reduced_dup_rhs:
+            print(k, reduced_dup_rhs[k])
+        # assert not reduced_dup_rhs, reduced_dup_rhs
+
+    def test_dup_rule(self):
+        import inspect
+        python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
+                      is_pypy=IS_PYPY,
+                      parser_debug={
+                          'dups': True, 'transition': False, 'reduce': False,
+                          'rules': False, 'errorstack': None, 'context': True})
+if __name__ == '__main__':
+    unittest.main()

test/Makefile

@@ -2,11 +2,11 @@ PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clea
       check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
       check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
       check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
-      check-3.4 check-3.5 check-3.6 check-3.7 check-5.6 5.6 5.8 \
+      check-3.4 check-3.5 check-5.6 5.6 5.8 \
       grammar-coverage-2.5 grammar-coverage-2.6 grammar-coverage-2.7 \
       grammar-coverage-3.1 grammar-coverage-3.2 grammar-coverage-3.3 \
-      grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6 \
-      grammar-coverage-3.7
+      grammar-coverage-3.4 grammar-coverage-3.5 grammar-coverage-3.6
+
 
 GIT2CL ?= git2cl
 PYTHON ?= python
@@ -28,7 +28,7 @@ check:
 	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-native-short
+check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-native-short
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
@@ -37,7 +37,6 @@ check-3.0: check-bytecode
 #: Run working tests from Python 3.1
 check-3.1: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
-	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
 
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
@@ -63,10 +62,6 @@ check-3.5: check-bytecode
 check-3.6: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
 
-#: Run working tests from Python 3.7
-check-3.7: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify $(COMPILE)
-
 # FIXME
 #: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
 5.8 5.6:
@@ -206,7 +201,6 @@ check-bytecode-3.0:
 #: Check deparsing Python 3.1
 check-bytecode-3.1:
 	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
-	$(PYTHON) test_pythonlib.py --bytecode-3.1-run --verify-run
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
@@ -233,15 +227,15 @@ check-bytecode-3.6:
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 
-#: Check deparsing Python 3.7
-check-bytecode-3.7:
-	$(PYTHON) test_pythonlib.py --bytecode-3.7 --weak-verify
-
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION)-run --verify-run $(COMPILE)
 
+#: Run longer Python 2.6's lib files known to be okay
+check-2.4-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.4 --verify $(COMPILE)
+
 #: Run longer Python 2.6's lib files known to be okay
 check-2.6-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.6 --weak-verify $(COMPILE)

test/bytecode_2.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.4
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 2.4 interpreter, they are likely to give an error when they
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.4 interpreter, they will give an error if they
 are miscompiled.

test/bytecode_2.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.5.
 
-Furthermore the programs here are self-checking: when decompiled and
-then run again in a 2.5 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.5 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.6_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.6.
 
-Furthermore the programs here are self-checking: when decompiled and
-then run again in a 2.6 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.6 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.7_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 2.7.
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 2.7 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.7 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.0_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.0.
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 3.0 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.0 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.1_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.1.
 
-Furthrmore, the programs here are self-checking: when decompiled and
-then run again in a 3.1 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.1 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.2_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.2.
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 3.2 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.2 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.3_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.3.
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 3.3 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.3 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.4_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.4.
 
-Furthermore, the programs here are self-checking: when decompiled and
-then run again in a 3.4 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.4 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.5_run/README

@@ -1,5 +1,5 @@
 These are byte-compiled programs compiled by Python 3.5.
 
-Furthrmore, the programs here are self-checking: when decompiled and
-then run again in a 3.5 interpreter, they are likely to give an error
-when they are miscompiled.
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.5 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.6_run/README

@@ -1,5 +0,0 @@
-These are byte-compiled programs compiled by Python 3.6.
-
-Furthrmore, the programs here are self-checking: when decompiled and
-then run again in a 3.6 interpreter, they are likely to give an error
-when they are miscompiled.