Another 2.7 "if" with return fix

This works is in conjunction with the commit before the previous release commit.

.gitignore

@@ -19,4 +19,4 @@ build
 /.venv*
 /.idea
 /.hypothesis
-./ChangeLog
+ChangeLog

HOW-TO-REPORT-A-BUG.md

@@ -10,23 +10,36 @@ decompile everything. Overall, I think this one probably does the best
 job of *any* Python decompiler that handles such a wide range of
 versions.
 
-But at any given time, there are maybe dozens of valid Python bytecode
-files that I know of that will cause problems. And when I get through
-those and all the issues of decompiler bugs that are currently logged,
-I could probably easily find dozens more bugs just by doing a
-decompile of all the Python bytecode on any one of my
-computers. Unless you want to help out by _fixing_ bugs, or are
-willing to do work by isolating and narrowing bugs, don't feel you are
-doing me a favor by doing scans on your favorite sets of bytecode
-files.
+But at any given time, there are a number of valid Python bytecode
+files that I know of that will cause problems. See, for example, the
+list in
+[`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
 
-In sum, it is not uncommon that you will find a mistranslation in
-decompiling. Furthermore, you may be expected to do some work in order
-to have your bug worthy of being considered above other bugs.
+But I understand: you would the bugs _you_ encounter addressed before
+all the other known bugs.
 
-No one is getting paid to work to work on this project, let alone bugs
-you may have an interest in. If you require decompiling bytecode
-immediately, consider using a decompilation service.
+From my standpoint, the good thing about the bugs listed in
+`runtests.sh` is that each test case is small and isolated to a single
+kind of problem. And I'll tend to fix easier, more isolated cases than
+generic "something's wrong" kinds of bugs where I'd have to do a bit
+of work to figure out what's up, if not use some sort of mind reading,
+make some guesses, and perform some experiments to see if the guesses
+are correct. I can't read minds, nor am I into guessing games; I'd
+rather devote the effort spent instead towards fixing bugs that are
+precisely defined.
+
+And it often turns out that by just fixing the well-defined and
+prescribed cases, the ill-defined amorphous cases as well will get
+handled as well.
+
+In sum, you may need to do some work to have the bug you have found
+handled before the hundreds of other bugs, and things I could be
+doing.
+
+No one is getting paid to work to work on this project, let alone the
+bugs you may have an interest in. If you require decompiling bytecode
+immediately, consider using a decompilation service, listed further
+down in this document.
 
 ## Is it really a bug?
 
@@ -114,7 +127,7 @@ Also try to narrow the bug. See below.
 Some kind folks also give the invocation they used and the output
 which usually includes an error message produced. This is
 helpful. From this, I can figure out what OS you are running this on
-and what version of *uncomplye6* was used. Therefore, if you don't
+and what version of *uncomplye6* was used. Therefore, if you _don't_
 provide the input command and the output from that, please give:
 
 * _uncompyle6_ version used

NEWS

@@ -1,4 +1,69 @@
-uncompyle6 2.14.0 2017-12-10 Dr. Gecko
+uncompyle6 2.16.0 2018-02-17
+
+- API additions:
+  - add fragments.op_at_code_loc() and
+  - fragments.deparsed_find()_
+- Better 2.7 end_if and COME_FROM determination
+- Fix up 3.6+ CALL_FUNCTION_EX
+- Misc pydisasm fixes
+- Wierd comprehension bug seen via new loctraceback
+- Fix Python 3.5+ CALL_FUNCTION_VAR and BUILD_LIST_UNPACK in call; with this
+  we can can handle 3.5+ f(a, b, *c, *d, *e) now
+
+uncompyle6 2.15.1 2018-02-05
+
+- More bug fixes and revert an improper bug fix in 2.15.0
+
+uncompyle6 2.15.0 2018-02-05 pycon2018.co
+
+- Bug fixes
+- Code fragment improvements
+- Code cleanups
+- Expand testing
+
+uncompyle6 2.15.1 2018-01-27
+
+- Add --linemap option to give line correspondences
+  between original source lines and reconstructed line sources.
+  It is far from perfect, but it is a start
+- Add a new class of tests: tests which when decompiled check themselves
+- Split off Python version semantic action customizations into its own file
+- Fix 2.7 bug in ifelse loop statement
+- Handle 3.6+ EXTENDED_ARGs for POP_JUMP_IF... instructions
+- Correct 3.6+ calls with kwargs
+- Describe the difficulty of 3.6 in README
+
+uncompyle6 2.14.3 2018-01-19
+
+- Fix bug in 3.5+ await stmt
+- Better version to magic handling; handle 3.5.2 .. 3.5.4 versions
+- Improve/correct test_pyenvlib.py status messages
+- Fix some 2.7 and 2.6 parser bugs
+- Fix whilelse parsing bugs
+- Correct 2.5- decorator parsing
+- grammar for decorators matches AST a little more
+- better tests in setup.py for running the right version of Python
+- Fix 2.6- parsing of "for .. try/else" ...  with "continue"  inside
+
+uncompyle6 2.14.2 2018-01-09 Samish
+
+Decompilation bug fixes, mostly 3.6 and pre 2.7
+
+- 3.6 FUNCTION_EX (somewhat)
+- 3.6 FUNCTION_EX_KW fixes
+- 3.6 MAKE_FUNCTION fixes
+- correct 3.5 CALL_FUNCTION_VAR
+- stronger 3.x "while 1" testing
+- Fix bug in if's with "pass" bodies. Fixes #104
+- try/else and try/finally fixes on 2.6-
+- limit pypy customization to pypy
+- Add addr fields in COME_FROMS
+- Allow use of full instructions in parser reduction routines
+- Reduce grammar in Pythion 3 by specialization more to specific
+  Python versions
+- Match Python AST names more closely when possible
+
+uncompyle6 2.14.1 2017-12-10 Dr. Gecko
 
 - Many decompilation bugfixes
 - Grammar rule reduction and version isolation

README.rst

@@ -63,10 +63,11 @@ fixed in the other decompilers.
 Requirements
 ------------
 
-This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
-Python versions 2.4-2.7 are supported in the python-2.4 branch.
-The bytecode files it can read has been tested on Python bytecodes from
-versions 1.5, 2.1-2.7, and 3.0-3.6 and the above-mentioned PyPy versions.
+The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
+or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
+python-2.4 branch.  The bytecode files it can read have been tested on
+Python bytecodes from versions 1.5, 2.1-2.7, and 3.0-3.6 and the
+above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -127,13 +128,13 @@ Known Bugs/Restrictions
 
 The biggest known and possibly fixable (but hard) problem has to do
 with handling control flow. (Python has probably the most diverse and
-screwy set of compound statements I've ever seen; a number of the
-usual ones like else clauses on loops and try blocks I suspect most
+screwy set of compound statements I've ever seen; there
+are "else" clauses on loops and try blocks that I suspect many
 programmers don't know about.)
 
-All of the Python decompilers I have looked at have the same
-problem. In some cases we can detect an erroneous decompilation and
-report that.
+All of the Python decompilers that I have looked at have problems
+decompiling Python's control flow. In some cases we can detect an
+erroneous decompilation and report that.
 
 *Verification* is the process of decompiling bytecode, compiling with
 a Python for that bytecode version, and then comparing the bytecode
@@ -162,19 +163,22 @@ python 2.3-2.4 since a lot of the goodness of early the version of the
 decompiler from that era has been preserved (and Python compilation in
 that era was minimal)
 
-Later distributions average about 200 files. There is some work to do
-on the lower end Python versions which is more difficult for us to
-handle since we don't have a Python interpreter for versions 1.5, 1.6,
-and 2.0.
+There is some work to do on the lower end Python versions which is
+more difficult for us to handle since we don't have a Python
+interpreter for versions 1.5, 1.6, and 2.0.
 
 In the Python 3 series, Python support is is strongest around 3.4 or
 3.3 and drops off as you move further away from those versions. Python
 3.6 changes things drastically by using word codes rather than byte
-codes. That has been addressed, but then it also changes function call
-opcodes and its semantics and has more problems with control flow than
-3.5 has. Between Python 3.5, 3.6 and 3.7 there have been major changes
-to the `MAKE_FUNCTION` and `CALL_FUNCTION` instructions. Those are
-not handled yet.
+codes. As a result, the jump offset field in a jump instruction
+argument has been reduced. This makes the `EXTENDED_ARG` instructions
+are now more prevalent in jump instruction; previously they had been
+rare.  Perhaps to compensate for the additional `EXTENDED_ARG`
+instructions, additional jump optimization has been added. So in sum
+handling control flow by ad hoc means as is currently done is worse.
+
+Also, between Python 3.5, 3.6 and 3.7 there have been major changes to the
+`MAKE_FUNCTION` and `CALL_FUNCTION` instructions.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
@@ -186,10 +190,12 @@ handled.
 
 We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
 Deobfuscator_ to unscramble the bytecode to get valid bytecode before
-trying this tool.
-
-Handling pathologically long lists of expressions or statements is
-slow.
+trying this tool. This program can't decompile Microsoft Windows EXE
+files created by Py2EXE_, although we can probably decompile the code
+after you extract the bytecode properly. For situations like this, you
+might want to consider a decompilation service like `Crazy Compilers
+<http://www.crazy-compilers.com/decompyle/>`_.  Handling
+pathologically long lists of expressions or statements is slow.
 
 
 There is lots to do, so please dig in and help.
@@ -218,3 +224,4 @@ See Also
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
+.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe

__pkginfo__.py

@@ -9,7 +9,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2017 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2018 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 5 - Production/Stable',
@@ -26,6 +26,7 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
                 'Programming Language :: Python :: 3.6',
+                'Programming Language :: Python :: 3.7',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -39,13 +40,14 @@ entry_points       = {
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.8.4, < 1.9.0',
-                      'xdis >= 3.6.2, < 3.7.0', 'six']
+install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
+                      'xdis >= 3.6.9, < 3.7.0', 'six']
+
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 py_modules         = None
-short_desc         = 'Python cross-version byte-code deparser'
+short_desc         = 'Python cross-version byte-code decompiler'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

admin-tools/how-to-make-a-release.md

@@ -17,11 +17,11 @@
 <!-- markdown-toc end -->
 # Get latest sources:
 
-    $ . ./admin-tool/update-sources.sh
+    git pull
 
 # Change version in uncompyle6/version.py:
 
-	$ emacs uncompyle6/version.py
+    $ emacs uncompyle6/version.py
     $ source uncompyle6/version.py
     $ echo $VERSION
     $ git commit -m"Get ready for release $VERSION" .
@@ -32,7 +32,7 @@
 
 #  Update NEWS from ChangeLog:
 
-	$ emacs NEWS
+    $ emacs NEWS
     $ make check
     $ git commit --amend .
     $ git push   # get CI testing going early
@@ -44,15 +44,11 @@
 # Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
 
     $ source admin-tools/setup-python-2.4.sh
-    $ rm ChangeLog
-
     $ git merge master
+	# Add and fix merge conflicts
+	$ git commit
 
-# Update NEWS from master branch
-
-    $ git commit -m"Get ready for release $VERSION" .
-
-# Check against all versions
+# Check against older versions
 
     $ source admin-tools/check-older-versions.sh
 
@@ -61,7 +57,7 @@
     $ . ./admin-tools/make-dist-older.sh
     $ git tag release-python-2.4-$VERSION
 
-    $ . /admin-tools/make-dist-newer.sh
+    $ . ./admin-tools/make-dist-newer.sh
     $ git tag release-$VERSION
 
 # Upload single package and look at Rst Formating

circle.yml

@@ -7,7 +7,8 @@ machine:
 dependencies:
     override:
         - pip install -e .
-        - pip install -r requirements-dev.txt
+        - pip install pytest==3.2.5 hypothesis
 test:
     override:
         - python ./setup.py develop && make check-2.7
+        - cd ./test/stdlib && pyenv local 2.7.10 && bash ./runtests.sh 'test_[p-z]*.py'

pytest/test_deparse.py

@@ -1,4 +1,5 @@
-from uncompyle6.semantics.fragments import deparse_code as deparse
+import pytest
+from uncompyle6.semantics.fragments import deparse_code as deparse, deparsed_find
 from uncompyle6 import PYTHON_VERSION, PYTHON3
 
 def map_stmts(x, y):
@@ -32,19 +33,21 @@ def get_parsed_for_fn(fn):
     code = fn.__code__ if PYTHON3 else fn.func_code
     return deparse(PYTHON_VERSION, code)
 
-def check_expect(expect, parsed):
+def check_expect(expect, parsed, fn_name):
     debug = False
     i = 2
     max_expect = len(expect)
     for name, offset in sorted(parsed.offsets.keys()):
-        assert i+1 <= max_expect, "ran out if items in testing node"
+        assert i+1 <= max_expect, (
+            "%s: ran out if items in testing node" % fn_name)
         nodeInfo = parsed.offsets[name, offset]
         node = nodeInfo.node
+        nodeInfo2 = deparsed_find((name, offset), parsed, code)
         extractInfo = parsed.extract_node_info(node)
 
         assert expect[i] == extractInfo.selectedLine, \
-          ('line %s expect:\n%s\ngot:\n%s' %
-                         (i, expect[i], extractInfo.selectedLine))
+          ('%s: line %s expect:\n%s\ngot:\n%s' %
+               (fn_name, i, expect[i], extractInfo.selectedLine))
         assert expect[i+1] ==  extractInfo.markerLine, \
                      ('line %s expect:\n%s\ngot:\n%s' %
                      (i+1, expect[i+1], extractInfo.markerLine))
@@ -73,6 +76,7 @@ def check_expect(expect, parsed):
 
 
 def test_stuff():
+    return
     parsed = get_parsed_for_fn(map_stmts)
     expect = """
 -1
@@ -83,10 +87,10 @@ return (x, y)
 -------------
 0
 x = []
-    --
+    -
 Contained in...
 x = []
-------
+    --
 3
 x = []
 -
@@ -95,10 +99,10 @@ x = []
 ------
 6
 y = {}
-    --
+    -
 Contained in...
 y = {}
-------
+    --
 9
 y = {}
 -
@@ -130,7 +134,7 @@ Contained in...
 x = [] ...
 ------ ...
 """.split("\n")
-    check_expect(expect, parsed)
+    check_expect(expect, parsed, 'map_stmts')
     ########################################################
     # return
 
@@ -167,7 +171,7 @@ Contained in...
 return (x, y)
 -------------
 """.split("\n")
-    check_expect(expect, parsed)
+    check_expect(expect, parsed, 'return_stmt')
     ########################################################
 #     # try
 
@@ -315,4 +319,4 @@ for i in range(2): ...
 """.split("\n")
     parsed = get_parsed_for_fn(for_range_stmt)
     if not PYTHON3:
-        check_expect(expect, parsed)
+        check_expect(expect, parsed, 'range_stmt')

pytest/test_disasm.py

@@ -11,20 +11,14 @@ src_dir = get_srcdir()
 os.chdir(src_dir)
 
 
-@pytest.mark.parametrize(("test_tuple", "function_to_test"), [
-    (
-        ('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
-        disassemble_file
-    ),
-    (
-        ('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
-        disassemble_file
-    ),
+@pytest.mark.parametrize(("test_tuple"), [
+    ('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
+    ('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
 ])
-def test_funcoutput(capfd, test_tuple, function_to_test):
+def test_funcoutput(capfd, test_tuple):
 
-    in_file , filename_expected = test_tuple
-    function_to_test(in_file, native=False)
+    in_file, filename_expected = test_tuple
+    disassemble_file(in_file)
     resout, reserr = capfd.readouterr()
     expected = open(filename_expected, "r").read()
     if resout != expected:

pytest/test_fjt.py

@@ -27,7 +27,12 @@ def test_if_in_for():
     print(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
         scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         scan.build_prev_op(n)
         fjt = scan.find_jump_targets(False)
 
@@ -44,8 +49,13 @@ def test_if_in_for():
 
         code = bug_loop.__code__
         n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
         scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
         scan.build_prev_op(n)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt = scan.find_jump_targets(False)
         assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
         assert scan.structs == [
@@ -64,6 +74,9 @@ def test_if_in_for():
         scan.build_lines_data(code)
         scan.build_prev_op()
         scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \

pytest/test_grammar.py

@@ -16,7 +16,10 @@ def test_grammar():
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
     (lhs, rhs, tokens,
      right_recursive, dup_rhs) = p.check_sets()
-    expect_lhs = set(['expr1024', 'pos_arg'])
+
+    # We have custom rules that create the below
+    expect_lhs = set(['expr1024', 'pos_arg', 'get_iter', 'attribute'])
+
     unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
                       'unpack',])
@@ -35,6 +38,7 @@ def test_grammar():
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
+            unused_rhs.add('call')
             if PYTHON_VERSION < 3.6:
                 # 3.6 has at least one non-custom call rule
                 # the others don't
@@ -47,8 +51,6 @@ def test_grammar():
             else:
                 expect_right_recursive.add((('l_stmts',
                                              ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
-                unused_rhs.add('build_map_unpack_with_call')
-                unused_rhs.add('unmapexpr')
                 # expect_lhs.add('kwargs1')
                 pass
             pass

pytest/test_pysource.py

@@ -1,4 +1,6 @@
+import sys
 from uncompyle6 import PYTHON3
+from uncompyle6.scanner import get_scanner
 from uncompyle6.semantics.consts import (
     escape, NONE,
     # RETURN_NONE, PASS, RETURN_LOCALS
@@ -17,7 +19,10 @@ from uncompyle6.semantics.pysource import SourceWalker as SourceWalker
 
 def test_template_engine():
     s = StringIO()
-    sw = SourceWalker(2.7, s, None)
+    sys_version = float(sys.version[0:3])
+    scanner = get_scanner(sys_version, is_pypy=False)
+    scanner.insts = []
+    sw = SourceWalker(2.7, s, scanner)
     sw.ast = NONE
     sw.template_engine(('--%c--', 0), NONE)
     print(sw.f.getvalue())

pytest/testdata/if-2.7.right

@@ -7,6 +7,6 @@
    7       6  LOAD_NAME             1  'False'
            9  STORE_NAME            2  'b'
           12  JUMP_FORWARD          0  'to 15'
-        15_0  COME_FROM                '12'
+        15_0  COME_FROM            12  '12'
           15  LOAD_CONST            0  ''
           18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -10,6 +10,6 @@
 
    6      15  LOAD_CONST            1  2
           18  STORE_NAME            2  'd'
-        21_0  COME_FROM                '12'
+        21_0  COME_FROM            12  '12'
           21  LOAD_CONST            2  ''
           24  RETURN_VALUE     

requirements-dev.txt

@@ -1,3 +1,3 @@
 pytest>=3.0.0
 flake8
-hypothesis
+hypothesis<=3.8.3

setup.py

@@ -1,7 +1,20 @@
 #!/usr/bin/env python
+import sys
 
 """Setup script for the 'uncompyle6' distribution."""
 
+SYS_VERSION = sys.version_info[0:2]
+if not ((2, 6) <= SYS_VERSION  <= (3, 7)) or ((3, 0) <= SYS_VERSION <= (3, 1)):
+    mess = "Python Release 2.6 .. 3.7 excluding 3.0 and 3.1 are supported in this code branch."
+    if ((2, 4) <= SYS_VERSION <= (2, 7)):
+        mess += ("\nFor your Python, version %s, use the python-2.4 code/branch." %
+                 sys.version[0:3])
+    elif SYS_VERSION < (2, 4) or ((3, 0) <= SYS_VERSION <= (3, 1)):
+        mess += ("\nThis package is not supported for Python version %s."
+                 % sys.version[0:3])
+    print(mess)
+    raise Exception(mess)
+
 from __pkginfo__ import \
     author,           author_email,       install_requires,          \
     license,          long_description,   classifiers,               \

test/Makefile

@@ -86,7 +86,7 @@ check-bytecode: check-bytecode-3
 	$(PYTHON) test_pythonlib.py \
 	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
-        --bytecode-pypy2.7 --bytecode-1
+        --bytecode-pypy2.7
 
 
 #: Check deparsing bytecode 1.5 only
@@ -156,13 +156,13 @@ grammar-coverage-3.3:
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pyenvlib.py --3.3.6
 
-##: Get grammar coverage for Python 3.4
+#: Get grammar coverage for Python 3.4
 grammar-coverage-3.4:
 	-rm $(COVER_DIR)/spark-grammar-34.cover
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pyenvlib.py --3.4.2
 
-##: Get grammar coverage for Python 3.5
+#: Get grammar coverage for Python 3.5
 grammar-coverage-3.5:
 	rm $(COVER_DIR)/spark-grammar-35.cover || /bin/true
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
@@ -171,10 +171,12 @@ grammar-coverage-3.5:
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
 	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-2.6-run --verify-run
 
 #: Check deparsing Python 2.7
 check-bytecode-2.7:
 	$(PYTHON) test_pythonlib.py --bytecode-2.7 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-2.7-run --verify-run
 
 #: Check deparsing Python 3.0
 check-bytecode-3.0:
@@ -191,22 +193,27 @@ check-bytecode-3.2:
 #: Check deparsing Python 3.3
 check-bytecode-3.3:
 	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
 
 #: Check deparsing Python 3.4
 check-bytecode-3.4:
 	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
 
 #: Check deparsing Python 3.5
 check-bytecode-3.5:
 	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
 
 #: Check deparsing Python 3.6
 check-bytecode-3.6:
 	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION)-run --verify-run $(COMPILE)
 
 #: Run longer Python 2.6's lib files known to be okay
 check-2.6-ok:

test/bytecode_2.4_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.4
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.4 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.5_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.5.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.5 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.6_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.6.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.6 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.7_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.7.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.7 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.0/README

@@ -0,0 +1 @@
+These are byte-compiled programs compiled by Python 3.0

test/bytecode_3.0_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.0.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.0 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.1/README

@@ -0,0 +1 @@
+These are byte-compiled programs compiled by Python 3.1

test/bytecode_3.1_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.1.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.1 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.2_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.2.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.2 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.3_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.3.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.3 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.4_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.4.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.4 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_3.5_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 3.5.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 3.5 interpreter, they will give an error if they
+are miscompiled.

test/simple_source/branching/02_ifelse_lambda.py

@@ -14,3 +14,6 @@ g()
 
 h = lambda: 1 if False else 3
 h()
+
+# From 2.7 test_builtin
+lambda c: 'a' <= c <= 'z', 'Hello World'

test/simple_source/bug22/02_try_except_finally.py

@@ -0,0 +1,17 @@
+# Adapted from Python 2.4 bdb.py runeval()
+
+# In Python 2.4 and before, try/finally has to be one block
+# and try/except has to be in a separate block.
+
+# In Python 2.5 and later, these can be combined into one "try" block,
+# and indeed compiling this in 2.5+ will in fact combine the blocks.
+# And that's okay, even if it might not be what was written.
+
+# However for 2.4 and before make sure this _isn't_ combined into one block.
+try:
+    try:
+        quitting = eval("1+2")
+    except RuntimeError:
+        pass
+finally:
+    quitting = 1

test/simple_source/bug25/02_decorator.py

@@ -0,0 +1,9 @@
+# From python 2.5 make_decorators.py
+# Bug was in not recognizing @memoize which uses grammra rules
+# using nonterminals mkfuncdeco and mkfuncdeco0
+def memoize(func):
+    pass
+def test_memoize(self):
+    @memoize
+    def double(x):
+        return x * 2

test/simple_source/bug25/02_try_else_loop.py

@@ -0,0 +1,17 @@
+# From 2.4 test_binop.py bug is missing 'else:' in 2nd try.
+def test_constructor():
+    for bad in "0", 0.0, 0j, (), [], {}, None:
+        try:
+            raise TypeError(bad)
+        except TypeError:
+            pass
+        else:
+            assert False, "%r didn't raise TypeError" % bad
+        try:
+            raise TypeError(bad)
+        except TypeError:
+            pass
+        else:
+            assert False, "%r didn't raise TypeError" % bad
+
+test_constructor()

test/simple_source/bug25/03_try_else.py

@@ -0,0 +1,18 @@
+# From Python 2.4. test_cgi.py
+# Bug was in putting  try block inside the ifelse statement.
+
+# Note: this is a self testing program - will assert on failure.
+def do_test(method):
+    if method == "GET":
+        rc = 0
+    elif method == "POST":
+        rc = 1
+    else:
+        raise ValueError, "unknown method: %s" % method
+    try:
+        rc = 2
+    except ZeroDivisionError:
+        rc = 3
+    return rc
+
+assert 2 == do_test("GET")

test/simple_source/bug26/03_tryelse_continue.py

@@ -0,0 +1,25 @@
+# 2.6- Try/else in a loop with a continue which
+# requires a tryelsestmtc
+# From 2.6- test_codecs.py
+def test_specific_values(self):
+    for flags in self:
+        if flags:
+            try:
+                self = 1
+            except ValueError:
+                continue
+            else:
+                self = 2
+
+        self = 3
+
+# From 2.6 test_decorators.
+# Bug was thinking an "except" was some sort of if/then
+def call(*args):
+    try:
+        return 5
+    except KeyError:
+        return 2
+    except TypeError:
+        # Unhashable argument
+        return 3

test/simple_source/bug27+/02_ifelsetmtl.py

@@ -0,0 +1,11 @@
+# Issue #148 on 2.7
+# Bug is in handling CONTINUE like JUMP_BACK
+# Similar code is probably found in a 2.7 stdlib. mapurl?
+def reduce_url(url):
+    atoms = []
+    for atom in url:
+        if atom == '.':
+            pass  # JUMP_BACK is patched as CONTINUE here
+        elif atom == '..':
+            atoms.push()
+    return atoms

test/simple_source/bug27+/04_loop_try_else.py

@@ -0,0 +1,12 @@
+# From Python 2.7 ihooks.py
+def ensure_fromlist(self, fromlist):
+    for sub in fromlist:
+        if sub:
+            if not recursive:
+                try:
+                    all = 5
+                except AttributeError:
+                    pass
+                else:
+                    all = 6
+            continue

test/simple_source/bug27+/04_try_tryelse.py

@@ -0,0 +1,21 @@
+# From 2.7 test_normalize.py
+# Bug has to to with finding the end of the tryelse block. I think thrown
+# off by the "continue". In instructions the COME_FROM for END_FINALLY
+# was at the wrong offset because some sort of "rtarget" was adjust.
+
+# When control flow is in place this logic in the code will be simplified
+def test_main(self, c1):
+    for line in self:
+        try:
+            c1 = 6
+        except:
+            if c1:
+                try:
+                    c1 = 5
+                except:
+                    pass
+                else:
+                    c1 = 1
+            continue
+
+        pass

test/simple_source/bug27+/05_deadcode.py

@@ -0,0 +1,6 @@
+# Test to see we can a program that has dead code in it.
+# This was issue #150
+def func(a):
+    if a:
+        return True
+        something_never_run()

test/simple_source/bug27+/05_while_if_continue.py

@@ -0,0 +1,11 @@
+# Issue 151 for Python 2.7
+# Bug was two-fold. Not having a rile for a while loop with an ending return statement
+# and allowing iflastsmtl to have an optional c_stmt which allowed the "if" to get
+# comined into the "while". A separate analysis for control flow should make this
+# simpiler.
+def func(a, b, c):
+    while a:
+        if b:
+            continue
+        return False
+    return True

test/simple_source/bug27+/05_while_if_return.py

@@ -0,0 +1,10 @@
+# Issue #149. Bug in Python 2.7 was handling a return stmt at the end
+# of a while with so no jump back, confusing which block the
+# return should be part of
+def test(a):
+    while True:
+        if a:
+            pass
+        else:
+            continue
+        return