Get ready for release 2.7.1

* assert one and two-arg form
* trystmt

Simplify adding multiple grammar rules

.gitignore

@@ -1,5 +1,7 @@
+*.pyc
 *_dis
 *~
+*.pyc
 /.cache
 /.eggs
 /.python-version
@@ -8,6 +10,7 @@
 /__pkginfo__.pyc
 /dist
 /how-to-make-a-release.txt
+/nose-*.egg
 /tmp
 /uncompyle6.egg-info
 __pycache__

.travis.yml

@@ -7,7 +7,6 @@ python:
   - '2.7.11'
   - '2.6'
   - '3.4'
-  - '3.2'
 
 install:
 - pip install -r requirements.txt
@@ -15,3 +14,8 @@ install:
 
 script:
 - python ./setup.py develop && COMPILE='--compile' make check
+
+# blacklist
+branches:
+  except:
+    - data-driven-pytest

HISTORY.md

@@ -93,8 +93,18 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-This project, uncompyle6, however owes its existence to the fork of
-uncompyle2 by Myst herie (Mysterie) whose first commit picks up at
+In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
+transforming the byte code into a a pseudo 2.7 python bytecode and is
+based on code from Eloi Vanderbeken.
+
+This project, uncompyle6, abandons that approach for various
+reasons. However the main reason is that we need offsets in fragment
+deparsing to be exactly the same, and the transformation process can
+remove instructions.  Adding instructions with psuedo_offsets is
+however okay.
+
+Uncompyle6, however owes its existence to the fork of uncompyle2 by
+Myst herie (Mysterie) whose first commit picks up at
 2012. I chose this since it seemed to have been at that time the most
 actively, if briefly, worked on. Also starting around 2012 is Dark
 Fenx's uncompyle3 which I used for inspiration for Python3 support.
@@ -102,8 +112,7 @@ Fenx's uncompyle3 which I used for inspiration for Python3 support.
 I started working on this late 2015, mostly to add fragment support.
 In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
 while, handling Python bytecodes from Python versions 2.5+ and
-3.2+. (I think I could go back further, but I'd consider doing that
-only after code is better cleaned up and supports Python 3 better.)
+3.2+.
 
 Over the many years, code styles and Python features have
 changed. However brilliant the code was and still is, it hasn't really

LICENSE

@@ -1,6 +1,6 @@
-Copyright (c) 1998-2002 John Aycock
-Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
 Copyright (c) 2015 by Rocky Bernstein
+Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
+Copyright (c) 1998-2002 John Aycock
 
   Permission is hereby granted, free of charge, to any person obtaining
   a copy of this software and associated documentation files (the

Makefile

@@ -40,6 +40,14 @@ check-3.2 check-3.5:
 check-2.6:
 	$(MAKE) -C test $@
 
+#:PyPy 2.6.1 or PyPy 5.0.1
+# Skip for now
+2.6 5.0:
+
+#:PyPy pypy3-2.4.0 Python 3:
+pypy-3.2 2.4:
+	$(MAKE) -C test $@
+
 #: Run py.test tests
 pytest:
 	$(MAKE) -C pytest check

NEWS

@@ -1,4 +1,50 @@
-uncompyle6 2.4.0 2016-05-18
+uncompyle6 2.7.1 2016-07-26
+
+- PyPy bytecodes for 2.7 and 3.2 added
+- Instruction formatting improved slightly
+- 2.7 bytecode "continue" bug fixed
+
+uncompyle6 2.7.0 2016-07-15
+
+- Many Syntax and verifification bugs removed
+  tested on standard libraries from 2.3.7 to 3.5.1
+  and they all decompile and verify fine.
+  I'm sure there are more bugs though.
+
+uncompyle6 2.6.2 2016-07-11 Manhattenhenge
+
+- Extend bytecodes back to 2.3
+- Fix bugs:
+  * 3.x and 2.7 set comprehensions,
+  * while1 loops
+  * continue statements
+- DRY and segregate grammar more
+
+uncompyle6 2.6.1 2016-07-08
+
+- Go over Python 2.5 bytecode deparsing
+  all library programs now deparse
+- Fix a couple bugs in 2.6 deparsing
+
+uncompyle6 2.6.0 2016-07-07
+
+- Improve Python 2.6 bytecode deparsing:
+  stdlib now will deparse something
+- Better <2.6 vs. 2.7 grammar separation
+- Fix some 2.7 deparsing bugs
+- Fix bug in installing uncompyle6 script
+- Doc improvments
+
+uncompyle6 2.5.0 2016-06-22  Summer Solstace
+
+- Much better Python 3.2-3.5 coverage.
+  3.4.6 is probably the best;3.2 and 3.5 are weaker
+- Better AST printing with -t
+- Better error reporting
+- Better fragment offset tracking
+- Some (much-needed) code refactoring
+
+uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
 
 - Many Python 3 bugs fixed:
   * Python 3.2 to 3.5 libaries largely

PKG-INFO

@@ -5,6 +5,6 @@ Summary: Python byte-code to source-code converter
 Home-page: http://github.com/rocky/python-uncompyle6
 Author: Rocky
 Author-email: rb@dustyfeet.com
-License: GPLv3
+License: MIT
 Description: UNKNOWN
 Platform: UNKNOWN

README.rst

@@ -3,27 +3,28 @@
 uncompyle6
 ==========
 
-A native Python bytecode Disassembler, Decompiler, Fragment Decompiler
-and bytecode library. Follows in the tradition of decompyle, uncompyle, and uncompyle2.
+A native Python cross-version Decompiler and Fragment Decompiler.
+Follows in the tradition of decompyle, uncompyle, and uncompyle2.
 
 
 Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 2.5 to 3.5 or
-so. The code requires Python 2.6 or later and has been tested on Python
-running versions 2.6, 2.7, 3.2, 3.3, 3.4 and 3.5.
+source code. It accepts bytecodes from Python version 2.3 to 3.5 or
+so, including PyPy bytecode.
 
 Why this?
 ---------
 
 There were a number of decompyle, uncompile, uncompyle2, uncompyle3
-forks around. All of them come basically from the same code base, and
-almost all of them not maintained very well. This code pulls these together
-and addresses a number of open issues in those.
+forks around. All of them came basically from the same code base, and
+almost all of them no were no longer actively maintained. Only one
+handled Python 3, and even there, only 3.2. This code pulls these
+together and moves forward. It also addresses a number of open issues
+in the previous forks.
 
-What makes this different from other CPython bytecode decompilers?  Its
+What makes this different from other CPython bytecode decompilers?: its
 ability to deparse just fragments and give source-code information
 around a given bytecode offset.
 
@@ -39,12 +40,12 @@ location in more detail than just a line number.  It can be also used
 when source-code information does not exist and there is just bytecode
 information.
 
-Other parts of the library can be used inside Python for various
-bytecode-related tasks. For example you can read in bytecode,
-i.e. perform a version-independent `marshal.loads()`, and disassemble
-the bytecode using a version of Python different from the one used to
-compile the bytecode.
+Requirements
+------------
 
+This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
+The bytecode files it can read has been tested on Python bytecodes from
+versions 2.3-2.7, and 3.2-3.5 and the above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -59,7 +60,7 @@ This uses setup.py, so it follows the standard Python routine:
     # or if you have pyenv:
     python setup.py develop
 
-A GNU makefile is also provided so `make install` (possibly as root or
+A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
 
 Testing
@@ -73,7 +74,7 @@ A GNU makefile has been added to smooth over setting running the right
 command, and running tests from fastest to slowest.
 
 If you have remake_ installed, you can see the list of all tasks
-including tests via `remake --tasks`
+including tests via :code:`remake --tasks`
 
 
 Usage
@@ -83,23 +84,33 @@ Run
 
 ::
 
-     ./bin/uncompyle6 -h
-     ./bin/pydisassemble -h
+$ uncompyle6 *compiled-python-file-pyc-or-pyo*
 
-for usage help.
+For usage help:
+
+::
+
+   $ uncompyle6 -h
 
 
 Known Bugs/Restrictions
 -----------------------
 
-Python 2 deparsing decompiles about the first 140 or so of the Python
-2.7.10 and 2.7.11 standard library files and all but less that 10%
-verify. So as such, it is probably a little better than uncompyle2.
-Other Python 2 versions do worse.
+Python 2 deparsing decompiles and about 90% verifies from Python 2.3.7 to Python
+3.4.2 on the standard library packages I have on my system.
 
-Python 3 deparsing before 3.5 is okay, but even there, more work is needed to
-decompile all of its library. Python 3.5 is missing some of new
-opcodes and idioms added, but it still often works.
+(Verification is the process of decompiling bytecode, compiling with a
+Python for that byecode version, and then comparing the byetcode
+produced by the decompiled/compiled program. Some allowance is made
+for inessential differences.)
+
+Later distributions average about 200 files. At this point, 2.7
+decompilation is better than uncompyle2. A number of bugs have been
+fixed.
+
+Python 3.5 largely works, but still has some bugs in it.
+Python 3.6 changes things drastically by using word codes rather than
+byte codes, and that needs to be addressed.
 
 There is lots to do, so please dig in and help.
 
@@ -107,14 +118,12 @@ See Also
 --------
 
 * https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
-* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only
-
-The above projects use a different decompiling technique what is used here.
-
-The HISTORY file.
+* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
+* The HISTORY_ file.
 
 .. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
 .. _trepan: https://pypi.python.org/pypi/trepan
+.. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc

__pkginfo__.py

@@ -36,12 +36,13 @@ entry_points={
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.2.1']
+install_requires   = ['spark-parser >= 1.4.0',
+                      'xdis >= 2.0.3']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 py_modules         = None
-short_desc         = 'Python byte-code disassembler and source-code converter'
+short_desc         = 'Python cross-version byte-code deparser'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

pytest/test_deparse.py

@@ -20,12 +20,20 @@ def for_range_stmt():
     for i in range(2):
         i+1
 
+# # FIXME: add this test - but for Python 2.7+ only
+# def set_comp():
+#     {y for y in range(3)}
+
+# FIXME: add this test
+def list_comp():
+    [y for y in range(3)]
+
 def get_parsed_for_fn(fn):
     code = fn.__code__ if PYTHON3 else fn.func_code
     return deparse(PYTHON_VERSION, code)
 
 def check_expect(expect, parsed):
-    debug = True
+    debug = False
     i = 2
     max_expect = len(expect)
     for name, offset in sorted(parsed.offsets.keys()):
@@ -160,7 +168,7 @@ return (x, y)
 -------------
 """.split("\n")
     check_expect(expect, parsed)
-#     ########################################################
+    ########################################################
 #     # try
 
 #     expect = """
@@ -291,6 +299,12 @@ return
 Contained in...
     i + 1
     -----
+31
+return
+------
+Contained in...
+for i in range(2): ...
+------------------ ...
 34
 return
 ------

pytest/test_disasm.py

@@ -24,7 +24,7 @@ os.chdir(src_dir)
 def test_funcoutput(capfd, test_tuple, function_to_test):
 
     in_file , filename_expected = test_tuple
-    function_to_test(in_file)
+    function_to_test(in_file, native=False)
     resout, reserr = capfd.readouterr()
     expected = open(filename_expected, "r").read()
     if resout != expected:

pytest/test_fjt.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-from uncompyle6 import PYTHON_VERSION
+from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
 from array import array
 def bug(state, slotstate):
@@ -12,13 +12,12 @@ def test_if_in_for():
     code = bug.__code__
     scan = get_scanner(PYTHON_VERSION)
     print(PYTHON_VERSION)
-    if 2.7 <= PYTHON_VERSION <= 3.0:
+    if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
         scan.build_lines_data(code, n)
         scan.build_prev_op(n)
         fjt = scan.find_jump_targets()
         assert {15: [3], 69: [66], 63: [18]} == fjt
-        print(scan.structs)
         assert scan.structs == \
           [{'start': 0, 'end': 72, 'type': 'root'},
            {'start': 18, 'end': 66, 'type': 'if-then'},

pytest/test_load.py

@@ -1,13 +0,0 @@
-import os, sys
-from uncompyle6.load import load_file, check_object_path, load_module
-
-def test_load():
-    """Basic test of load_file, check_object_path and load_module"""
-    co = load_file(__file__)
-    obj_path = check_object_path(__file__)
-    if os.path.exists(obj_path):
-        version, timestamp, magic_int, co2 = load_module(obj_path)
-        assert sys.version[0:3] == str(version)
-        assert co == co2
-    else:
-        assert True, "Skipped because we can't find %s" % obj_path

pytest/test_marsh.py

@@ -1,20 +0,0 @@
-#!/usr/bin/env python
-
-import os.path
-from uncompyle6.load import load_module
-
-def get_srcdir():
-    filename = os.path.normcase(os.path.dirname(os.path.abspath(__file__)))
-    return os.path.realpath(filename)
-
-srcdir = get_srcdir()
-
-def test_load_module():
-    """Tests uncompile6.load.load_module"""
-    # We deliberately pick a bytecode that we aren't likely to be running against
-    mod_file = os.path.join(get_srcdir(), '..', 'test', 'bytecode_2.5',
-                        '02_complex.pyc')
-
-    version, timestamp, magic_int, co = load_module(mod_file)
-    assert version == 2.5, "Should have picked up Python version properly"
-    assert co.co_consts == (5j, None), "Code should have a complex constant"

pytest/testdata/if-2.7.right

@@ -1,11 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6      0	LOAD_NAME             0  'True'
-          3	POP_JUMP_IF_FALSE    15  '15'
+   6       0  LOAD_NAME             0  'True'
+           3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7      6	LOAD_NAME             1  'False'
-          9	STORE_NAME            2  'b'
-         12	JUMP_FORWARD          0  '15'
-         15	LOAD_CONST            0  ''
-         18	RETURN_VALUE             ''
+   7       6  LOAD_NAME             1  'False'
+           9  STORE_NAME            2  'b'
+          12  JUMP_FORWARD          0  'to 15'
+        15_0  COME_FROM                '12'
+          15  LOAD_CONST            0  ''
+          18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -1,14 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3      0	LOAD_NAME             0  'True'
-          3	POP_JUMP_IF_FALSE    15  '15'
+   3       0  LOAD_NAME             0  'True'
+           3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4      6	LOAD_CONST            0  1
-          9	STORE_NAME            1  'b'
-         12	JUMP_FORWARD          6  '21'
+   4       6  LOAD_CONST            0  1
+           9  STORE_NAME            1  'b'
+          12  JUMP_FORWARD          6  'to 21'
 
-   6     15	LOAD_CONST            1  2
-         18	STORE_NAME            2  'd'
-         21	LOAD_CONST            2  ''
-         24	RETURN_VALUE             ''
+   6      15  LOAD_CONST            1  2
+          18  STORE_NAME            2  'd'
+        21_0  COME_FROM                '12'
+          21  LOAD_CONST            2  ''
+          24  RETURN_VALUE     

requirements.txt

@@ -1 +1,2 @@
 spark-parser >= 1.2.1
+xdis >= 2.0.3

test/Makefile

@@ -20,7 +20,7 @@ check:
 	$(MAKE) check-$$PYTHON_VERSION
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode check-2.7-ok
+check-2.6 check-2.7: check-bytecode-2 check-bytecode-3  check-2.7-ok
 
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
@@ -30,25 +30,40 @@ check-3.2: check-bytecode
 check-3.3: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.3 --verify $(COMPILE)
 
-#: Run working tests from Python 3.5
-check-3.5: check-bytecode
-
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 	$(PYTHON) test_pythonlib.py --bytecode-3.4 --verify $(COMPILE)
 
+#: Run working tests from Python 3.5
+check-3.5: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
+
 #: Check deparsing only, but from a different Python version
 check-disasm:
 	$(PYTHON) dis-compare.py
 
-#: Check deparsing bytecode only
+#: Check deparsing bytecode 2.x only
 check-bytecode-2:
-	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7
+	$(PYTHON) test_pythonlib.py --bytecode-2.3 --bytecode-2.4 \
+        --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
 
-#: Check deparsing bytecode only
-check-bytecode:
-	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
-        --bytecode-3.2 --bytecode-3.3 --bytecode-3.4 --bytecode-3.5
+#: Check deparsing bytecode 3.x only
+check-bytecode-3:
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --bytecode-3.3 \
+       --bytecode-3.4 --bytecode-3.5 --bytecode-pypy3.2
+
+#: Check deparsing bytecode that works running Python 2 and Python 3
+check-bytecode: check-bytecode-3
+	$(PYTHON) test_pythonlib.py --bytecode-2.3 --bytecode-2.4 \
+        --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
+
+#: Check deparsing Python 2.3
+check-bytecode-2.3:
+	$(PYTHON) test_pythonlib.py --bytecode-2.3
+
+#: Check deparsing Python 2.4
+check-bytecode-2.4:
+	$(PYTHON) test_pythonlib.py --bytecode-2.4
 
 #: Check deparsing Python 2.5
 check-bytecode-2.5:
@@ -82,6 +97,10 @@ check-bytecode-3.5:
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
 
+#: Run longer Python 2.6's lib files known to be okay
+check-2.6-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.6 --verify $(COMPILE)
+
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
@@ -94,6 +113,18 @@ check-3.2-ok:
 check-3.4-ok:
 	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
 
+#: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
+# Skip for now
+2.6:
+
+#: PyPy 5.0.x with Python 2.7 ...
+pypy-2.7 5.0:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy2.7 --verify
+
+#: PyPy 2.4.x with Python 3.2 ...
+pypy-3.2 2.4:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.2 --verify
+
 clean: clean-py-dis clean-dis clean-unverified
 
 clean-dis:
@@ -102,6 +133,6 @@ clean-dis:
 clean-unverified:
 	find . -name '*_unverified' -exec rm -v '{}' ';'
 
-#: Clean temporary compile/decompile/verify direcotries in /tmp
+#: Clean temporary compile/decompile/verify directories in /tmp
 clean-py-dis:
 	rm -fr /tmp/py-dis-* || true

test/add-test.py

@@ -2,11 +2,15 @@
 """ Trivial helper program to bytecompile and run an uncompile
 """
 import os, sys, py_compile
-assert len(sys.argv) == 2
-path = sys.argv[1]
-short = os.path.basename(path)
+assert len(sys.argv) >= 2
 version = sys.version[0:3]
-cfile =  "bytecode_%s/%s" % (version, short) + 'c'
-print("byte-compiling %s to %s" % (path, cfile))
-py_compile.compile(path, cfile)
-os.system("../bin/uncompyle6 -a -t %s" % cfile)
+for path in sys.argv[1:]:
+    short = os.path.basename(path)
+    if hasattr(sys, 'pypy_version_info'):
+        cfile =  "bytecode_pypy%s/%s" % (version, short) + 'c'
+    else:
+        cfile =  "bytecode_%s/%s" % (version, short) + 'c'
+    print("byte-compiling %s to %s" % (path, cfile))
+    py_compile.compile(path, cfile)
+    if isinstance(version, str) or version >= (2, 6, 0):
+        os.system("../bin/uncompyle6 -a -t %s" % cfile)