Get ready for release 2.9.10

and some parser list nonterminal cleanup

.gitignore

@@ -1,3 +1,5 @@
+*.pyo
+*.pyc
 *_dis
 *~
 /.cache
@@ -8,7 +10,9 @@
 /__pkginfo__.pyc
 /dist
 /how-to-make-a-release.txt
+/nose-*.egg
 /tmp
 /uncompyle6.egg-info
+/unpyc
 __pycache__
 build

.travis.yml

@@ -4,8 +4,9 @@ sudo: false
 
 python:
   - '3.5'
-  - '2.7.11'
+  - '2.7.12'
   - '2.6'
+  - '3.3'
   - '3.4'
   - '3.2'
 
@@ -15,3 +16,8 @@ install:
 
 script:
 - python ./setup.py develop && COMPILE='--compile' make check
+
+# blacklist
+branches:
+  except:
+    - data-driven-pytest

DECOMPYLE-2.4-CHANGELOG.txt

@@ -1,4 +1,5 @@
-This is the changelog from *decompyle*'s release 2.4 passed on by Dan Pascu
+This is the changelog from *decompyle*'s release 2.4 and before
+passed on by Dan Pascu
 
 
 release 2.4 (Dan Pascu)
@@ -74,3 +75,57 @@ release 2.2beta1 (hartmut Goebel)
   - use a single Walker and a single Parser, thus saving time and memory
   - use augmented assign and 'print >>' internally
   - optimized 'Walker.engine', the main part of code generation
+release 0.6.0: (hartmut Goebel)
+  - extended print (Python 2.0)
+  - extended import (Python 2.0) (may not cover all cases)
+  - augmented assign (Python 2.0) (may not cover all cases)
+  - list comprehensions (Python 2.0)
+  - equivalent for 'apply' (Python 1.6)
+  - if .. elif .. else are now nested as expected
+  - assert test, data
+  - unpack list corrected (was the same as unpack tuple)
+  - fixed unpack tuple (trailing semicolon was missing)
+  - major speed up :-)
+  - reduced memory usage (pre-alpha-0.5 has increased it a lot)
+  - still missing: EXTENDED_ARG
+
+pre-alpha-0.5: (hartmut Goebel)
+  - *args, **kwargs
+  - global
+  - formal tuple parameters (eg. def a(self, (x,y,z)) )
+  - actual lambda parameters (eg.  X(lambda z: z**2) )
+  - remove last 'return None' in procedures
+  - remove last 'return locals()' in class definitions
+  - docstrings
+
+pre-alpha-0.4: (hartmut Goebel)
+  - assert
+  - try/except/finally
+  - parentheses in expressions
+  - nested expressions
+  - extracted dissassemble() from module dis and
+    removed ugly redirect of stdout, thus saved a lot of
+    ugly code and a lot of memory
+
+pre-alpha-0.3: (hartmut Goebel)
+  - keyword arguments
+  - some boolean expressions
+  - and/or
+  - complex conditions in if/while
+  - read byte-code from .pyc without importing
+  - access to the body of classes and modules
+  - class and function definitions
+  - a = b = c = xxx
+
+pre-alpha-0.1 -> pre-alpha-0.2:
+  -  SET_LINENO filtered out in lexer now
+  -  added support for subscripts (just for Christian Tismer :-)
+  -  fixed bug with handling of BUILD_{LIST,TUPLE} & CALL_FUNCTION
+  -  dict-building support
+  -  comparison support
+  -  exec support
+  -  del support
+  -  pass support
+  -  slice support
+  -  no more extraneous (albeit legal) commas
+  -  finally, it excepts try [sic] but not all 42 variations of it

HISTORY.md

@@ -29,7 +29,9 @@ augmented with pseudo instruction COME_FROM. This code introduced
 another clever idea: using table-driven semantics routines, using
 format specifiers.
 
-The last mention of a release of SPARK from John is around 2002.
+The last mention of a release of SPARK from John is around 2002. As
+released, although the Earley Algorithm parser was in good shape, this
+code was woefully lacking as serious Python deparser.
 
 In the fall of 2000, Hartmut Goebel
 [took over maintaining the code](https://groups.google.com/forum/#!searchin/comp.lang.python/hartmut$20goebel/comp.lang.python/35s3mp4-nuY/UZALti6ujnQJ). The
@@ -93,8 +95,18 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-This project, uncompyle6, however owes its existence to the fork of
-uncompyle2 by Myst herie (Mysterie) whose first commit picks up at
+In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
+transforming the byte code into a a pseudo 2.7 python bytecode and is
+based on code from Eloi Vanderbeken.
+
+This project, uncompyle6, abandons that approach for various
+reasons. However the main reason is that we need offsets in fragment
+deparsing to be exactly the same, and the transformation process can
+remove instructions.  Adding instructions with psuedo_offsets is
+however okay.
+
+Uncompyle6, however owes its existence to the fork of uncompyle2 by
+Myst herie (Mysterie) whose first commit picks up at
 2012. I chose this since it seemed to have been at that time the most
 actively, if briefly, worked on. Also starting around 2012 is Dark
 Fenx's uncompyle3 which I used for inspiration for Python3 support.
@@ -102,13 +114,18 @@ Fenx's uncompyle3 which I used for inspiration for Python3 support.
 I started working on this late 2015, mostly to add fragment support.
 In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
 while, handling Python bytecodes from Python versions 2.5+ and
-3.2+. (I think I could go back further, but I'd consider doing that
-only after code is better cleaned up and supports Python 3 better.)
+3.2+. In doing so, it has been expedient to separate this into three
+projects: load loading and disassembly (xdis), parsing and tree
+building (spark_parser), and grammar and semantic actions for
+decompiling (uncompyle6).
+
 
 Over the many years, code styles and Python features have
 changed. However brilliant the code was and still is, it hasn't really
 had a single public active maintainer. And there have been many forks
-of the code.
+of the code.  I have spent a great deal of time trying to organize and
+modularize the code so that it can handle more Python versions more
+gracefully (with still only moderate success).
 
 That it has been in need of an overhaul has been recognized by the
 Hartmut a decade an a half ago:
@@ -118,9 +135,9 @@ Hartmut a decade an a half ago:
     NB. This is not a masterpiece of software, but became more like a hack.
     Probably a complete rewrite would be sensefull. hG/2000-12-27
 
-This project deparses using an Early-algorithm parse with lots of
+This project deparses using an Earley-algorithm parse with lots of
 massaging of tokens and the grammar in the scanner
-phase. Early-algorithm parsers are context free and tend to be linear
+phase. Earley-algorithm parsers are context free and tend to be linear
 if the grammar is LR or left recursive.
 
 Another approach that doesn't use grammars is to do something like

LICENSE

@@ -1,6 +1,6 @@
-Copyright (c) 1998-2002 John Aycock
-Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
 Copyright (c) 2015 by Rocky Bernstein
+Copyright (c) 2000 by hartmut Goebel <h.goebel@crazy-compilers.com>
+Copyright (c) 1998-2002 John Aycock
 
   Permission is hereby granted, free of charge, to any person obtaining
   a copy of this software and associated documentation files (the

MANIFEST.in

@@ -2,10 +2,16 @@ include README.rst
 include ChangeLog
 include HISTORY.md
 include LICENSE
+include Makefile
+include requirements.txt
+include requirements-dev.txt
 include DECOMPYLE-2.4-CHANGELOG.txt
 include __pkginfo__.py
 recursive-include uncompyle6 *.py
 include bin/uncompyle6
 include bin/pydisassemble
+include pytest/Makefile
+include test/Makefile
 recursive-include test *.py *.pyc
 recursive-include pytest *.py
+recursive-include pytest/testdata *

Makefile

@@ -33,13 +33,21 @@ check-2.7 check-3.3 check-3.4: pytest
 
 #: Tests for Python 3.2 and 3.5 - pytest doesn't work here
 # Or rather 3.5 doesn't work not on Travis
-check-3.2 check-3.5:
+check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
 	$(MAKE) -C test $@
 
 #:Tests for Python 2.6 (doesn't have pytest)
 check-2.6:
 	$(MAKE) -C test $@
 
+#:PyPy 2.6.1 or PyPy 5.0.1
+# Skip for now
+2.6 5.0 5.3:
+
+#:PyPy pypy3-2.4.0 Python 3:
+pypy-3.2 2.4:
+	$(MAKE) -C test $@
+
 #: Run py.test tests
 pytest:
 	$(MAKE) -C pytest check

NEWS

@@ -1,4 +1,230 @@
-uncompyle6 2.4.0 2016-05-18
+uncompyle6 2.9.10 2016-02-25
+
+- Python grammar rule fixes
+- Add ability to get grammar coverage on runs
+- Handle Python 3.6 opcode BUILD_CONST_KEYMAP
+
+uncompyle6 2.9.9 2016-12-16
+
+- Remaining Python 3.5 ops handled
+  (this also means more Python 3.6 ops are handled)
+- Python 3.5 and 3.6 async and await handled
+- Python 3.0 decompilation improved
+- Python 3 annotations fixed
+- Better control-flow detection
+- Code cleanups and misc bug fixes
+
+uncompyle6 2.9.8 2016-12-16
+
+- Better control-flow detection
+- pseudo instruction THEN in 2.x
+  to disambiguate if from and
+- fix bug in --verify option
+- DRY (a little) control-flow detection
+- fix syntax in tuples with one element
+- if AST rule inheritence in Python 2.5
+- NAME_MODULE removal for Python <= 2.4
+- verifycall fixes for Python <= 2.4
+- more Python lint
+
+uncompyle6 2.9.7 2016-12-16
+
+- Start to handle 3.5/3.6 build_map_unpack_with_call
+- Some Python 3.6 bytecode to wordcode conversion fixes
+- option -g: show start-end range when possible
+- track print_docstring move to help (used in python 3.1)
+- verify: allow RETURN_VALUE to match RETURN_END_IF
+- some 3.2 compatibility
+- Better Python 3 control flow detection by adding Pseudo ELSE opcodes
+
+uncompyle6 2.9.6 2016-12-04
+
+- Shorten Python3 grammars with + and *
+  this requires spark parser 1.5.1
+- Add some AST reduction checks to improve
+  decompile accuracy. This too requires
+  spark parser 1.5.1
+
+uncompyle6 2.9.6 2016-11-20
+
+- Correct MANIFEST.in
+- More AST grammar checking
+- --linemapping option or linenumbers.line_number_mapping()
+  Shows correspondence of lines between source
+  and decompiled source
+- Some control flow adjustments in code for 2.x.
+  This is probably an improvement in 2.6 and before.
+  For 2.7 things are just shuffled around a little. Sigh.
+  Overall I think we are getting more precise in
+  or analysis even if it is not always reflected
+  in the results.
+- better control flow debugging output
+- Python 2 and 3 detect structure code is more similar
+- Handle Docstrings with embedded tiple quotes (""")
+
+uncompyle6 2.9.5 2016-11-13
+
+- Fix Python 3 bugs:
+  * improprer while 1 else
+  * docstring indent
+  * 3.3 default values in lambda expressions
+  * start 3.0 decompilation (needs newer xdis)
+- Start grammar misparse checking
+
+
+uncompyle6 2.9.4 2016-11-02
+
+- Handle Python 3.x function annotations
+- track def keywoard-parameter line-splitting in source code better
+- bump min xdis version to mask previous xdis bug
+
+uncompyle6 2.9.3 2016-10-26
+
+Release forced by incompatiblity change in xdis 3.2.0.
+
+- Python 3.1 bugs:
+  * handle "with ... as"
+  * handle "with"
+  * Start handling def (...) -> yy (has bugs still)
+
+- DRY Python 3.x via inheritance
+- Python 3.6 work (from Daniel Bradburn)
+  * Handle 3.6 buildstring
+  * Handle 3.6 handle single and multiple fstring better
+
+
+uncompyle6 2.9.2 2016-10-15
+
+- use source-code line breaks to assist in where to break
+  in tuples and maps
+- Fix Python 1.5 decompyle bugs
+- Fix some Python 2.6 and below bugs
+- DRY fragments.py code a little
+
+uncompyle6 2.9.1 2016-10-09
+
+- Improved Python 1.5 decompiling
+- Handle old-style pre Python 2.2 classes
+
+uncompyle6 2.9.0 2016-10-09
+
+- Use xdis 3.0.0 protocol load_module.
+  this Forces change in requirements.txt and _pkg_info_.py
+- Start Python 1.5 decompiling; another round of work is needed to
+  remove bugs
+- Simpify python 2.1 grammar
+- Fix bug with -t ...  Wasn't showing source text when -t option was given
+- Fix 2.1-2.6 bug in list comprehension
+
+uncompyle6 2.8.4 2016-10-08
+
+- Python 3 disassembly bug fixes
+- Python 3.6 fstring bug fixes (from moagstar)
+- Python 2.1 disassembly
+- COME_FROM suffixes added in Python3
+- use .py extension in verification disassembly
+
+uncompyle6 2.8.3 2016-09-11 live from NYC!
+
+NOTE: this is possibly the last release before a major reworking of
+control-flow structure detection is done.
+
+- Lots of bug fixes in decompilation:
+  * 3.0 .. 3.4 whileTrue bug
+  * 3.x function declaration deparsing:
+    . 3.0 .. 3.2 *args processing
+    . 3.0 .. 3.2 call name and kwargs bug
+    . 3.0 .. getting parameter of *
+    . 3.0 .. handling varible number of args
+    . 3.0 .. "if" structure bugs
+  * 3.5+ if/else bugs
+  * 2.2-2.6 bugs
+    . try/except control flow
+    . a == b == c -like detection
+    . generator detection
+    . "while .. and" statement bugs
+    . handle "except <cond>, <var>"
+    . use older raise format in 2.x
+- scanner "disassemble" is now "ingest". True disassembly is done by xdis
+- Start accepting Python 3.1 bytecode
+- Add --weak-verify option on test_pyenvlib and test_pythonlib. This
+  catches more bugs more easily
+- bump xdis requirement so we can deparse dropbox 2.5 code
+- Added H. Goebel's changes before 2.4 in DECOMPYLE-2.4-CHANGELOG.txt
+
+uncompyle6 2.8.2 2016-08-29
+
+- Handle Python 3.6 format string conversions !r, !s, !a
+- Start to handle 3.1 bytecode
+- Fix some PyPy translation bugs
+- We now only handle 3.6.0a3+ since that is incompatible with 3.6 before that
+
+uncompyle6 2.8.1 2016-08-20
+
+- Add Python 2.2 decompilation
+
+- Fix bugs
+ * PyPy LOOKUP_METHOD bug
+ * Python 3.6 FORMAT_VALUE handles expressions now
+
+uncompyle6 2.8.0 2016-08-03
+
+- Start Python 3.6 support (moagstar)
+  more work on PEP 498 needed
+- tidy bytecode/word output
+- numerous decompiling bugs fixed
+- grammar testing started
+- show magic number in deparsed output
+- better grammar and semantic action segregation based
+  on python bytecode version
+
+uncompyle6 2.7.1 2016-07-26
+
+- PyPy bytecodes for 2.7 and 3.2 added
+- Instruction formatting improved slightly
+- 2.7 bytecode "continue" bug fixed
+
+uncompyle6 2.7.0 2016-07-15
+
+- Many Syntax and verifification bugs removed
+  tested on standard libraries from 2.3.7 to 3.5.1
+  and they all decompile and verify fine.
+  I'm sure there are more bugs though.
+
+uncompyle6 2.6.2 2016-07-11 Manhattenhenge
+
+- Extend bytecodes back to 2.3
+- Fix bugs:
+  * 3.x and 2.7 set comprehensions,
+  * while1 loops
+  * continue statements
+- DRY and segregate grammar more
+
+uncompyle6 2.6.1 2016-07-08
+
+- Go over Python 2.5 bytecode deparsing
+  all library programs now deparse
+- Fix a couple bugs in 2.6 deparsing
+
+uncompyle6 2.6.0 2016-07-07
+
+- Improve Python 2.6 bytecode deparsing:
+  stdlib now will deparse something
+- Better <2.6 vs. 2.7 grammar separation
+- Fix some 2.7 deparsing bugs
+- Fix bug in installing uncompyle6 script
+- Doc improvments
+
+uncompyle6 2.5.0 2016-06-22  Summer Solstace
+
+- Much better Python 3.2-3.5 coverage.
+  3.4.6 is probably the best;3.2 and 3.5 are weaker
+- Better AST printing with -t
+- Better error reporting
+- Better fragment offset tracking
+- Some (much-needed) code refactoring
+
+uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
 
 - Many Python 3 bugs fixed:
   * Python 3.2 to 3.5 libaries largely

PKG-INFO

@@ -5,6 +5,6 @@ Summary: Python byte-code to source-code converter
 Home-page: http://github.com/rocky/python-uncompyle6
 Author: Rocky
 Author-email: rb@dustyfeet.com
-License: GPLv3
+License: MIT
 Description: UNKNOWN
 Platform: UNKNOWN

README.rst

@@ -1,29 +1,30 @@
-|buildstatus|
+|buildstatus| |Supported Python Versions|
 
 uncompyle6
 ==========
 
-A native Python bytecode Disassembler, Decompiler, Fragment Decompiler
-and bytecode library. Follows in the tradition of decompyle, uncompyle, and uncompyle2.
+A native Python cross-version Decompiler and Fragment Decompiler.
+Follows in the tradition of decompyle, uncompyle, and uncompyle2.
 
 
 Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 2.5 to 3.5 or
-so. The code requires Python 2.6 or later and has been tested on Python
-running versions 2.6, 2.7, 3.2, 3.3, 3.4 and 3.5.
+source code. It accepts bytecodes from Python version 2.1 to 3.6 or
+so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
 
 Why this?
 ---------
 
 There were a number of decompyle, uncompile, uncompyle2, uncompyle3
-forks around. All of them come basically from the same code base, and
-almost all of them not maintained very well. This code pulls these together
-and addresses a number of open issues in those.
+forks around. All of them came basically from the same code base, and
+almost all of them no were no longer actively maintained. Only one
+handled Python 3, and even there, only 3.2 or 3.3 depending on which
+code is used. This code pulls these together and moves forward. It
+also addresses a number of open issues in the previous forks.
 
-What makes this different from other CPython bytecode decompilers?  Its
+What makes this different from other CPython bytecode decompilers?: its
 ability to deparse just fragments and give source-code information
 around a given bytecode offset.
 
@@ -39,12 +40,13 @@ location in more detail than just a line number.  It can be also used
 when source-code information does not exist and there is just bytecode
 information.
 
-Other parts of the library can be used inside Python for various
-bytecode-related tasks. For example you can read in bytecode,
-i.e. perform a version-independent `marshal.loads()`, and disassemble
-the bytecode using a version of Python different from the one used to
-compile the bytecode.
+Requirements
+------------
 
+This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
+Python versions 2.4-2.7 are supported in the python-2.4 branch.
+The bytecode files it can read has been tested on Python bytecodes from
+versions 1.5, 2.1-2.7, and 3.2-3.6 and the above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -59,7 +61,7 @@ This uses setup.py, so it follows the standard Python routine:
     # or if you have pyenv:
     python setup.py develop
 
-A GNU makefile is also provided so `make install` (possibly as root or
+A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
 
 Testing
@@ -73,7 +75,7 @@ A GNU makefile has been added to smooth over setting running the right
 command, and running tests from fastest to slowest.
 
 If you have remake_ installed, you can see the list of all tasks
-including tests via `remake --tasks`
+including tests via :code:`remake --tasks`
 
 
 Usage
@@ -83,23 +85,79 @@ Run
 
 ::
 
-     ./bin/uncompyle6 -h
-     ./bin/pydisassemble -h
+$ uncompyle6 *compiled-python-file-pyc-or-pyo*
 
-for usage help.
+For usage help:
+
+::
+
+   $ uncompyle6 -h
+
+If you want strong verification of the correctness of the
+decompilation process, add the `--verify` option. But there are
+situations where this will indicate a failure, although the generated
+program is semantically equivalent. Using option `--weak-verify` will
+tell you if there is something definitely wrong. Generally, large
+swaths of code are decompiled correctly, if not the entire program.
+
+You can also cross compare the results with pycdc_ . Since they work
+differently, bugs here often aren't in that, and vice versa.
 
 
 Known Bugs/Restrictions
 -----------------------
 
-Python 2 deparsing decompiles about the first 140 or so of the Python
-2.7.10 and 2.7.11 standard library files and all but less that 10%
-verify. So as such, it is probably a little better than uncompyle2.
-Other Python 2 versions do worse.
+The biggest known and possibly fixable (but hard) problem has to do
+with handling control flow. All of the Python decompilers I have looked
+at have the same problem. In some cases we can detect an erroneous
+decompilation and report that.
+
+About 90% of the decompilation of Python standard library packages in
+Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
+"weakly" verify. Python 2.6 drops down to 96% weakly verifying.
+Other versions drop off in quality too.
+
+*Verification* is the process of decompiling bytecode, compiling with
+a Python for that bytecode version, and then comparing the bytecode
+produced by the decompiled/compiled program. Some allowance is made
+for inessential differences. But other semantically equivalent
+differences are not caught. For example ``1 and 0`` is decompiled to
+the equivalent ``0``; remnants of the first true evaluation (1) is
+lost when Python compiles this. When Python next compiles ``0`` the
+resulting code is simpler.
+
+*Weak Verification*
+on the other hand doesn't check bytecode for equivalence but does
+check to see if the resulting decompiled source is a valid Python
+program by running the Python interpreter. Because the Python language
+has changed so much, for best results you should use the same Python
+Version in checking as used in the bytecode.
+
+Later distributions average about 200 files. There is some work to do
+on the lower end Python versions which is more difficult for us to
+handle since we don't have a Python interpreter for versions 1.5, 1.6,
+and 2.0.
+
+In the Python 3 series, Python support is is strongest around 3.4 or
+3.3 and drops off as you move further away from those versions. Python
+3.5 largely works, but still has some bugs in it and is missing some
+opcodes.  Python 3.6 changes things drastically by using word codes
+rather than byte codes. That has been addressed, but then it also
+changes function call opcodes and its semantics and has more problems
+with control flow than 3.5 has.
+
+Currently not all Python magic numbers are supported. Specifically in
+some versions of Python, notably Python 3.6, the magic number has
+changes several times within a version. We support only the released
+magic. There are also customized Python interpreters, notably Dropbox,
+which use their own magic and encrypt bytcode. With the exception of
+the Dropbox's old Python 2.5 interpreter this kind of thing is not
+handled.
+
+We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
+Deobfuscator_ to unscramble the bytecode to get valid bytecode before
+trying this tool.
 
-Python 3 deparsing before 3.5 is okay, but even there, more work is needed to
-decompile all of its library. Python 3.5 is missing some of new
-opcodes and idioms added, but it still often works.
 
 There is lots to do, so please dig in and help.
 
@@ -107,17 +165,20 @@ See Also
 --------
 
 * https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
-* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only
-
-The above projects use a different decompiling technique what is used here.
-
-The HISTORY file.
+* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
+* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
+* The HISTORY_ file.
 
 .. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
 .. _trepan: https://pypi.python.org/pypi/trepan
+.. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
 .. _pycdc: https://github.com/zrax/pycdc
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
+.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
+   :target: https://pypi.python.org/pypi/uncompyle6/
+.. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
+.. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator

__pkginfo__.py

@@ -9,20 +9,23 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2017 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
-classifiers =  ['Development Status :: 4 - Beta',
+classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Intended Audience :: Developers',
                 'Operating System :: OS Independent',
                 'Programming Language :: Python',
-                'Programming Language :: Python :: 2',
+                'Programming Language :: Python :: 2.4',
+                'Programming Language :: Python :: 2.5',
                 'Programming Language :: Python :: 2.6',
                 'Programming Language :: Python :: 2.7',
-                'Programming Language :: Python :: 3',
+                'Programming Language :: Python :: 3.1',
+                'Programming Language :: Python :: 3.2',
                 'Programming Language :: Python :: 3.3',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
+                'Programming Language :: Python :: 3.6',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -36,12 +39,13 @@ entry_points={
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.2.1']
+install_requires   = ['spark-parser >= 1.6.0, < 1.7.0',
+                      'xdis >= 3.2.4, < 3.3.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 py_modules         = None
-short_desc         = 'Python byte-code disassembler and source-code converter'
+short_desc         = 'Python cross-version byte-code deparser'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

circle.yml

@@ -1,6 +1,6 @@
 machine:
     python:
-        version: 2.7.8
+        version: 2.7.10
     environment:
         COMPILE: --compile
 

pytest/.gitignore

@@ -1 +1,2 @@
+/.hypothesis
 /__pycache__

pytest/test_build_const_key_map.py

@@ -0,0 +1,21 @@
+import pytest
+# uncompyle6
+from uncompyle6 import PYTHON_VERSION
+from validate import validate_uncompyle
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@pytest.mark.parametrize('text', (
+    "{0.: 'a', -1: 'b'}",   # BUILD_MAP
+    "{'a':'b'}",            # BUILD_MAP
+    "{0: 1}",               # BUILD_MAP
+    "{b'0':1, b'2':3}",     # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a': 1, 'b': 2}",     # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0.0:'b',0.1:'d'}",    # BUILD_CONST_KEY_MAP
+))
+def test_build_const_key_map(text):
+    validate_uncompyle(text)

pytest/test_deparse.py

@@ -20,12 +20,20 @@ def for_range_stmt():
     for i in range(2):
         i+1
 
+# # FIXME: add this test - but for Python 2.7+ only
+# def set_comp():
+#     {y for y in range(3)}
+
+# FIXME: add this test
+def list_comp():
+    [y for y in range(3)]
+
 def get_parsed_for_fn(fn):
     code = fn.__code__ if PYTHON3 else fn.func_code
     return deparse(PYTHON_VERSION, code)
 
 def check_expect(expect, parsed):
-    debug = True
+    debug = False
     i = 2
     max_expect = len(expect)
     for name, offset in sorted(parsed.offsets.keys()):
@@ -160,7 +168,7 @@ return (x, y)
 -------------
 """.split("\n")
     check_expect(expect, parsed)
-#     ########################################################
+    ########################################################
 #     # try
 
 #     expect = """
@@ -291,6 +299,12 @@ return
 Contained in...
     i + 1
     -----
+31
+return
+------
+Contained in...
+for i in range(2): ...
+------------------ ...
 34
 return
 ------

pytest/test_disasm.py

@@ -24,7 +24,7 @@ os.chdir(src_dir)
 def test_funcoutput(capfd, test_tuple, function_to_test):
 
     in_file , filename_expected = test_tuple
-    function_to_test(in_file)
+    function_to_test(in_file, native=False)
     resout, reserr = capfd.readouterr()
     expected = open(filename_expected, "r").read()
     if resout != expected:

pytest/test_docstring.py

@@ -0,0 +1,78 @@
+import sys
+from uncompyle6 import PYTHON3
+if PYTHON3:
+    from io import StringIO
+    minint = -sys.maxsize-1
+    maxint = sys.maxsize
+else:
+    from StringIO import StringIO
+    minint = -sys.maxint-1
+    maxint = sys.maxint
+from uncompyle6.semantics.helper import print_docstring
+
+class PrintFake():
+    def __init__(self):
+        self.pending_newlines = 0
+        self.f = StringIO()
+
+    def write(self, *data):
+        if (len(data) == 0) or (len(data) == 1 and data[0] == ''):
+            return
+        out = ''.join((str(j) for j in data))
+        n = 0
+        for i in out:
+            if i == '\n':
+                n += 1
+                if n == len(out):
+                    self.pending_newlines = max(self.pending_newlines, n)
+                    return
+            elif n:
+                self.pending_newlines = max(self.pending_newlines, n)
+                out = out[n:]
+                break
+            else:
+                break
+
+        if self.pending_newlines > 0:
+            self.f.write('\n'*self.pending_newlines)
+            self.pending_newlines = 0
+
+        for i in out[::-1]:
+            if i == '\n':
+                self.pending_newlines += 1
+            else:
+                break
+
+        if self.pending_newlines:
+            out = out[:-self.pending_newlines]
+        self.f.write(out)
+    def println(self, *data):
+        if data and not(len(data) == 1 and data[0] ==''):
+            self.write(*data)
+        self.pending_newlines = max(self.pending_newlines, 1)
+        return
+    pass
+
+def test_docstring():
+
+    for doc, expect in (
+            ("Now is the time",
+             '  """Now is the time"""'),
+           ("""
+Now is the time
+""",
+            '''  """
+  Now is the time
+  """''')
+
+            # (r'''func placeholder - ' and with ("""\nstring\n  """)''',
+            #  """  r'''func placeholder - ' and with (\"\"\"\nstring\n\"\"\")'''"""),
+            # (r"""func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" """,
+            # """  r\"\"\"func placeholder - ' and with ('''\nstring\n''') and \"\"\"\nstring\n\"\"\" \"\"\"""")
+            ):
+
+        o = PrintFake()
+        # print(doc)
+        # print(expect)
+        print_docstring(o, '  ', doc)
+        assert expect == o.f.getvalue()

pytest/test_fjt.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-from uncompyle6 import PYTHON_VERSION
+from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
 from array import array
 def bug(state, slotstate):
@@ -8,27 +8,55 @@ def bug(state, slotstate):
             for key, value in slotstate.items():
                 setattr(state, key, 2)
 
+# From 2.7 disassemble
+# Problem is not getting while, because
+# COME_FROM not added
+def bug_loop(disassemble, tb=None):
+    if tb:
+        try:
+            tb = 5
+        except AttributeError:
+            raise RuntimeError
+        while tb: tb = tb.tb_next
+    disassemble(tb)
+
 def test_if_in_for():
     code = bug.__code__
     scan = get_scanner(PYTHON_VERSION)
     print(PYTHON_VERSION)
-    if 2.7 <= PYTHON_VERSION <= 3.0:
+    if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
         scan.build_lines_data(code, n)
         scan.build_prev_op(n)
-        fjt = scan.find_jump_targets()
+        fjt = scan.find_jump_targets(False)
         assert {15: [3], 69: [66], 63: [18]} == fjt
-        print(scan.structs)
         assert scan.structs == \
           [{'start': 0, 'end': 72, 'type': 'root'},
-           {'start': 18, 'end': 66, 'type': 'if-then'},
+           {'start': 15, 'end': 66, 'type': 'if-then'},
            {'start': 31, 'end': 59, 'type': 'for-loop'},
            {'start': 62, 'end': 63, 'type': 'for-else'}]
+
+        code = bug_loop.__code__
+        n = scan.setup_code(code)
+        scan.build_lines_data(code, n)
+        scan.build_prev_op(n)
+        fjt = scan.find_jump_targets(False)
+        assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
+        assert scan.structs == [
+            {'start': 0, 'end': 80, 'type': 'root'},
+            {'start': 3, 'end': 64, 'type': 'if-then'},
+            {'start': 6, 'end': 15, 'type': 'try'},
+            {'start': 19, 'end': 38, 'type': 'except'},
+            {'start': 45, 'end': 67, 'type': 'while-loop'},
+            {'start': 70, 'end': 64, 'type': 'while-else'},
+            # previous bug was not mistaking while-loop for if-then
+            {'start': 48, 'end': 67, 'type': 'while-loop'}]
+
     elif 3.2 < PYTHON_VERSION <= 3.4:
         scan.code = array('B', code.co_code)
         scan.build_lines_data(code)
         scan.build_prev_op()
-        fjt  = scan.find_jump_targets()
+        fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \
           [{'end': 72, 'type': 'root', 'start': 0},

pytest/test_fstring.py

@@ -0,0 +1,150 @@
+# std
+import os
+# test
+import pytest
+import hypothesis
+from hypothesis import strategies as st
+# uncompyle6
+from uncompyle6 import PYTHON_VERSION, deparse_code
+
+
+@st.composite
+def expressions(draw):
+    # todo : would be nice to generate expressions using hypothesis however
+    # this is pretty involved so for now just use a corpus of expressions
+    # from which to select.
+    return draw(st.sampled_from((
+        'abc',
+        'len(items)',
+        'x + 1',
+        'lineno',
+        'container',
+        'self.attribute',
+        'self.method()',
+        # These expressions are failing, I think these are control
+        # flow problems rather than problems with FORMAT_VALUE,
+        # however I need to confirm this...
+        #'sorted(items, key=lambda x: x.name)',
+        #'func(*args, **kwargs)',
+        #'text or default',
+        #'43 if life_the_universe and everything else None'
+    )))
+
+
+@st.composite
+def format_specifiers(draw):
+    """
+    Generate a valid format specifier using the rules:
+
+    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
+    fill        ::=  <any character>
+    align       ::=  "<" | ">" | "=" | "^"
+    sign        ::=  "+" | "-" | " "
+    width       ::=  integer
+    precision   ::=  integer
+    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
+
+    See https://docs.python.org/2/library/string.html
+
+    :param draw: Let hypothesis draw from other strategies.
+
+    :return: An example format_specifier.
+    """
+    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
+    fill = draw(st.one_of(alphabet_strategy, st.none()))
+    align = draw(st.sampled_from(list('<>=^')))
+    fill_align = (fill + align or '') if fill else ''
+
+    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
+    can_have_sign = type_ in 'deEfFgGnoxX%'
+    can_have_comma = type_ in 'deEfFgG%'
+    can_have_precision = type_ in 'fFgG'
+    can_have_pound = type_ in 'boxX%'
+    can_have_zero = type_ in 'oxX'
+
+    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
+    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
+    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
+
+    int_strategy = st.integers(min_value=1, max_value=1000)
+
+    width = draw(st.one_of(int_strategy, st.none()))
+    width = str(width) if width is not None else ''
+
+    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
+    if can_have_precision:
+        precision = draw(st.one_of(int_strategy, st.none()))
+        precision = '.' + str(precision) if precision else ''
+    else:
+        precision = ''
+
+    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
+
+
+@st.composite
+def fstrings(draw):
+    """
+    Generate a valid f-string.
+    See https://www.python.org/dev/peps/pep-0498/#specification
+
+    :param draw: Let hypothsis draw from other strategies.
+
+    :return: A valid f-string.
+    """
+    character_strategy = st.characters(
+        blacklist_characters='\r\n\'\\s{}',
+        min_codepoint=1,
+        max_codepoint=1000,
+    )
+    is_raw = draw(st.booleans())
+    integer_strategy = st.integers(min_value=0, max_value=3)
+    expression_count = draw(integer_strategy)
+    content = []
+    for _ in range(expression_count):
+        expression = draw(expressions())
+        conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
+        has_specifier = draw(st.booleans())
+        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
+        content.append('{{{}{}}}'.format(expression, conversion, specifier))
+        content.append(draw(st.text(character_strategy)))
+    content = ''.join(content)
+    return "f{}'{}'".format('r' if is_raw else '', content)
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@hypothesis.given(format_specifiers())
+def test_format_specifiers(format_specifier):
+    """Verify that format_specifiers generates valid specifiers"""
+    try:
+        exec('"{:' + format_specifier + '}".format(0)')
+    except ValueError as e:
+        if 'Unknown format code' not in str(e):
+            raise
+
+
+def run_test(text):
+    hypothesis.assume(len(text))
+    hypothesis.assume("f'{" in text)
+    expr = text + '\n'
+    code = compile(expr, '<string>', 'single')
+    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
+    recompiled = compile(deparsed.text, '<string>', 'single')
+    if recompiled != code:
+        assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@hypothesis.given(fstrings())
+def test_uncompyle_fstring(fstring):
+    """Verify uncompyling fstring bytecode"""
+    run_test(fstring)
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@pytest.mark.parametrize('fstring', [
+    "f'{abc}{abc!s}'",
+    "f'{abc}0'",
+])
+def test_uncompyle_direct(fstring):
+    """useful for debugging"""
+    run_test(fstring)

pytest/test_grammar.py

@@ -0,0 +1,63 @@
+import re
+from uncompyle6 import PYTHON_VERSION, PYTHON3, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser, python_parser
+from uncompyle6.scanner import get_scanner
+
+def test_grammar():
+
+    def check_tokens(tokens, opcode_set):
+        remain_tokens = set(tokens) - opcode_set
+        remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+        remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+        remain_tokens = set(remain_tokens) - opcode_set
+        assert remain_tokens == set([]), \
+            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dumpGrammar())
+
+    p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+    lhs, rhs, tokens, right_recursive = p.checkSets()
+    expect_lhs = set(['expr1024', 'pos_arg'])
+    unused_rhs = set(['build_list', 'call_function', 'mkfunc',
+                      'mklambda',
+                      'unpack', 'unpack_list'])
+    expect_right_recursive = [['designList', ('designator', 'DUP_TOP', 'designList')]]
+    if PYTHON3:
+        expect_lhs.add('load_genexpr')
+
+        unused_rhs = unused_rhs.union(set("""
+        except_pop_except genexpr classdefdeco2 listcomp
+        """.split()))
+        if 3.0 <= PYTHON_VERSION:
+            expect_lhs.add("annotate_arg")
+            expect_lhs.add("annotate_tuple")
+            unused_rhs.add("mkfunc_annotate")
+            pass
+    else:
+        expect_lhs.add('kwarg')
+    assert expect_lhs == set(lhs)
+    assert unused_rhs == set(rhs)
+    assert expect_right_recursive == right_recursive
+    s = get_scanner(PYTHON_VERSION, IS_PYPY)
+    ignore_set = set(
+            """
+            JUMP_BACK CONTINUE RETURN_END_IF
+            COME_FROM COME_FROM_EXCEPT COME_FROM_LOOP COME_FROM_WITH
+            COME_FROM_FINALLY ELSE
+            LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
+            LAMBDA_MARKER RETURN_LAST
+            """.split())
+    if 2.6 <= PYTHON_VERSION <= 2.7:
+        opcode_set = set(s.opc.opname).union(ignore_set)
+        check_tokens(tokens, opcode_set)
+    elif PYTHON_VERSION == 3.4:
+        ignore_set.add('LOAD_CLASSNAME')
+        ignore_set.add('STORE_LOCALS')
+        opcode_set = set(s.opc.opname).union(ignore_set)
+        check_tokens(tokens, opcode_set)
+
+def test_dup_rule():
+    import inspect
+    python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
+                  is_pypy=IS_PYPY,
+                  parser_debug={
+                      'dups': True, 'transition': False, 'reduce': False,
+                      'rules': False, 'errorstack': None, 'context': True})

pytest/test_load.py

@@ -1,13 +0,0 @@
-import os, sys
-from uncompyle6.load import load_file, check_object_path, load_module
-
-def test_load():
-    """Basic test of load_file, check_object_path and load_module"""
-    co = load_file(__file__)
-    obj_path = check_object_path(__file__)
-    if os.path.exists(obj_path):
-        version, timestamp, magic_int, co2 = load_module(obj_path)
-        assert sys.version[0:3] == str(version)
-        assert co == co2
-    else:
-        assert True, "Skipped because we can't find %s" % obj_path

pytest/test_marsh.py

@@ -1,20 +0,0 @@
-#!/usr/bin/env python
-
-import os.path
-from uncompyle6.load import load_module
-
-def get_srcdir():
-    filename = os.path.normcase(os.path.dirname(os.path.abspath(__file__)))
-    return os.path.realpath(filename)
-
-srcdir = get_srcdir()
-
-def test_load_module():
-    """Tests uncompile6.load.load_module"""
-    # We deliberately pick a bytecode that we aren't likely to be running against
-    mod_file = os.path.join(get_srcdir(), '..', 'test', 'bytecode_2.5',
-                        '02_complex.pyc')
-
-    version, timestamp, magic_int, co = load_module(mod_file)
-    assert version == 2.5, "Should have picked up Python version properly"
-    assert co.co_consts == (5j, None), "Code should have a complex constant"

pytest/test_single_compile.py

@@ -9,8 +9,6 @@ def test_single_mode():
         'i = j % 4',
         'i = {}',
         'i = []',
-        'while i < 1 or stop:\n    i\n',
-        'while i < 1 or stop:\n    print%s\n' % ('(i)' if PYTHON3 else ' i'),
         'for i in range(10):\n    i\n',
         'for i in range(10):\n    for j in range(10):\n        i + j\n',
         'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'

pytest/testdata/if-2.7.right

@@ -1,11 +1,12 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_if.py
 
-   6      0	LOAD_NAME             0  'True'
-          3	POP_JUMP_IF_FALSE    15  '15'
+   6       0  LOAD_NAME             0  'True'
+           3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   7      6	LOAD_NAME             1  'False'
-          9	STORE_NAME            2  'b'
-         12	JUMP_FORWARD          0  '15'
-         15	LOAD_CONST            0  ''
-         18	RETURN_VALUE             ''
+   7       6  LOAD_NAME             1  'False'
+           9  STORE_NAME            2  'b'
+          12  JUMP_FORWARD          0  'to 15'
+        15_0  COME_FROM                '12'
+          15  LOAD_CONST            0  ''
+          18  RETURN_VALUE     

pytest/testdata/ifelse-2.7.right

@@ -1,14 +1,15 @@
 # Python 2.7
 # Embedded file name: simple_source/branching/05_ifelse.py
 
-   3      0	LOAD_NAME             0  'True'
-          3	POP_JUMP_IF_FALSE    15  '15'
+   3       0  LOAD_NAME             0  'True'
+           3  POP_JUMP_IF_FALSE    15  'to 15'
 
-   4      6	LOAD_CONST            0  1
-          9	STORE_NAME            1  'b'
-         12	JUMP_FORWARD          6  '21'
+   4       6  LOAD_CONST            0  1
+           9  STORE_NAME            1  'b'
+          12  JUMP_FORWARD          6  'to 21'
 
-   6     15	LOAD_CONST            1  2
-         18	STORE_NAME            2  'd'
-         21	LOAD_CONST            2  ''
-         24	RETURN_VALUE             ''
+   6      15  LOAD_CONST            1  2
+          18  STORE_NAME            2  'd'
+        21_0  COME_FROM                '12'
+          21  LOAD_CONST            2  ''
+          24  RETURN_VALUE     

pytest/validate.py

@@ -0,0 +1,143 @@
+# future
+from __future__ import print_function
+# std
+import os
+import dis
+import difflib
+import subprocess
+import tempfile
+# compatability
+import six
+# uncompyle6 / xdis
+from uncompyle6 import PYTHON_VERSION, deparse_code
+
+
+def _dis_to_text(co):
+    return dis.Bytecode(co).dis()
+
+
+def print_diff(original, uncompyled):
+    """
+    Try and display a pretty html line difference between the original and
+    uncompyled code and bytecode if elinks and BeautifulSoup are installed
+    otherwise just show the diff.
+
+    :param original: Text describing the original code object.
+    :param uncompyled: Text describing the uncompyled code object.
+    """
+    original_lines = original.split('\n')
+    uncompyled_lines = uncompyled.split('\n')
+    args = original_lines, uncompyled_lines, 'original', 'uncompyled'
+    try:
+        from bs4 import BeautifulSoup
+        diff = difflib.HtmlDiff().make_file(*args)
+        diff = BeautifulSoup(diff, "html.parser")
+        diff.select_one('table[summary="Legends"]').extract()
+    except ImportError:
+        print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
+        diff = difflib.HtmlDiff().make_table(*args)
+
+    with tempfile.NamedTemporaryFile(delete=False) as f:
+        f.write(str(diff).encode('utf-8'))
+
+    try:
+        print()
+        html = subprocess.check_output([
+            'elinks',
+            '-dump',
+            '-no-references',
+            '-dump-color-mode',
+            '1',
+            f.name,
+        ]).decode('utf-8')
+        print(html)
+    except:
+        print('\nFor side by side diff install elinks')
+        diff = difflib.Differ().compare(original_lines, uncompyled_lines)
+        print('\n'.join(diff))
+    finally:
+        os.unlink(f.name)
+
+
+def are_instructions_equal(i1, i2):
+    """
+    Determine if two instructions are approximately equal,
+    ignoring certain fields which we allow to differ, namely:
+
+    * code objects are ignore (should probaby be checked) due to address
+    * line numbers
+
+    :param i1: left instruction to compare
+    :param i2: right instruction to compare
+
+    :return: True if the two instructions are approximately equal, otherwise False.
+    """
+    result = (1==1
+        and i1.opname == i2.opname
+        and i1.opcode == i2.opcode
+        and i1.arg == i2.arg
+        # ignore differences due to code objects
+        # TODO : Better way of ignoring address
+        and (i1.argval == i2.argval or '<code object' in str(i1.argval))
+        # TODO : Should probably recurse to check code objects
+        and (i1.argrepr == i2.argrepr or '<code object' in i1.argrepr)
+        and i1.offset == i2.offset
+        # ignore differences in line numbers
+        #and i1.starts_line
+        and i1.is_jump_target == i2.is_jump_target
+    )
+    return result
+
+
+def are_code_objects_equal(co1, co2):
+    """
+    Determine if two code objects are approximately equal,
+    see are_instructions_equal for more information.
+
+    :param i1: left code object to compare
+    :param i2: right code object to compare
+
+    :return: True if the two code objects are approximately equal, otherwise False.
+    """
+    # TODO : Use xdis for python2 compatability
+    instructions1 = dis.Bytecode(co1)
+    instructions2 = dis.Bytecode(co2)
+    for opcode1, opcode2 in zip(instructions1, instructions2):
+        if not are_instructions_equal(opcode1, opcode2):
+            return False
+    return True
+
+
+def validate_uncompyle(text, mode='exec'):
+    """
+    Validate decompilation of the given source code.
+
+    :param text: Source to validate decompilation of.
+    """
+    original_code = compile(text, '<string>', mode)
+    original_dis = _dis_to_text(original_code)
+    original_text = text
+
+    deparsed = deparse_code(PYTHON_VERSION, original_code,
+                            compile_mode=mode, out=six.StringIO())
+    uncompyled_text = deparsed.text
+    uncompyled_code = compile(uncompyled_text, '<string>', 'exec')
+
+    if not are_code_objects_equal(uncompyled_code, original_code):
+
+        uncompyled_dis = _dis_to_text(uncompyled_text)
+
+        def output(text, dis):
+            width = 60
+            return '\n\n'.join([
+                ' SOURCE CODE '.center(width, '#'),
+                text.strip(),
+                ' BYTECODE '.center(width, '#'),
+                dis
+            ])
+
+        original = output(original_text, original_dis)
+        uncompyled = output(uncompyled_text, uncompyled_dis)
+        print_diff(original, uncompyled)
+
+        assert 'original' == 'uncompyled'

requirements-dev.txt

@@ -1,2 +1,4 @@
 pytest
 flake8
+hypothesis
+six

requirements.txt

@@ -1 +1,2 @@
-spark-parser >= 1.2.1
+# Pick up stuff from setup.py
+-e .

setup.py

@@ -24,6 +24,6 @@ setup(
        py_modules         = py_modules,
        test_suite         = 'nose.collector',
        url                = web,
-       setup_requires     = ['nose>=1.0'],
+       tests_require     = ['nose>=1.0'],
        version            = VERSION,
        zip_safe           = zip_safe)

test/Makefile

@@ -20,47 +20,114 @@ check:
 	$(MAKE) check-$$PYTHON_VERSION
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode check-2.7-ok
+check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-2.7-ok
+
+#: Run working tests from Python 3.0
+check-3.0: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify $(COMPILE)
+
+#: Run working tests from Python 3.1
+check-3.1: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.2
 check-3.2: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.2 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
 	$(PYTHON) test_pythonlib.py --bytecode-3.3 --verify $(COMPILE)
 
-#: Run working tests from Python 3.5
-check-3.5: check-bytecode
-
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 	$(PYTHON) test_pythonlib.py --bytecode-3.4 --verify $(COMPILE)
 
+#: Run working tests from Python 3.5
+check-3.5: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
+
+#: Run working tests from Python 3.6
+check-3.6: check-bytecode
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --verify $(COMPILE)
+
 #: Check deparsing only, but from a different Python version
 check-disasm:
 	$(PYTHON) dis-compare.py
 
-#: Check deparsing bytecode only
-check-bytecode-2:
-	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7
+#: Check deparsing bytecode 1.x only
+check-bytecode-1:
+	$(PYTHON) test_pythonlib.py --bytecode-1.5
 
-#: Check deparsing bytecode only
-check-bytecode:
-	$(PYTHON) test_pythonlib.py --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
-        --bytecode-3.2 --bytecode-3.3 --bytecode-3.4 --bytecode-3.5
+#: Check deparsing bytecode 2.x only
+check-bytecode-2:
+	$(PYTHON) test_pythonlib.py \
+	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
+        --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 --bytecode-pypy2.7
+
+#: Check deparsing bytecode 3.x only
+check-bytecode-3:
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
+       --bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
+       --bytecode-3.4 --bytecode-3.5 --bytecode-pypy3.2
+
+#: Check deparsing bytecode that works running Python 2 and Python 3
+check-bytecode: check-bytecode-3
+	$(PYTHON) test_pythonlib.py \
+	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
+        --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
+        --bytecode-pypy2.7 --bytecode-1
+
+
+#: Check deparsing Python 2.1
+check-bytecode-2.1:
+	$(PYTHON) test_pythonlib.py --bytecode-2.1
+
+#: Check deparsing Python 2.2
+check-bytecode-2.2:
+	$(PYTHON) test_pythonlib.py --bytecode-2.2
+
+#: Check deparsing Python 2.3
+check-bytecode-2.3:
+	$(PYTHON) test_pythonlib.py --bytecode-2.3
+
+#: Check deparsing Python 2.4
+check-bytecode-2.4:
+	$(PYTHON) test_pythonlib.py --bytecode-2.4
 
 #: Check deparsing Python 2.5
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
+#: Get grammar coverage for Python 2.5
+grammar-coverage-2.5:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+
+#: Get grammar coverage for Python 2.6
+grammar-coverage-2.6:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+
+#: Get grammar coverage for Python 2.7
+grammar-coverage-2.7:
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
-	$(PYTHON) test_pythonlib.py --bytecode-2.6
+	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
 
 #: Check deparsing Python 2.7
 check-bytecode-2.7:
-	$(PYTHON) test_pythonlib.py --bytecode-2.7
+	$(PYTHON) test_pythonlib.py --bytecode-2.7 --verify
+
+#: Check deparsing Python 3.0
+check-bytecode-3.0:
+	$(PYTHON) test_pythonlib.py --bytecode-3.0
+
+#: Check deparsing Python 3.1
+check-bytecode-3.1:
+	$(PYTHON) test_pythonlib.py --bytecode-3.1
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
@@ -78,10 +145,18 @@ check-bytecode-3.4:
 check-bytecode-3.5:
 	$(PYTHON) test_pythonlib.py --bytecode-3.5
 
+#: Check deparsing Python 3.6
+check-bytecode-3.6:
+	$(PYTHON) test_pythonlib.py --bytecode-3.6
+
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
 
+#: Run longer Python 2.6's lib files known to be okay
+check-2.6-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.6 --verify $(COMPILE)
+
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
@@ -94,6 +169,18 @@ check-3.2-ok:
 check-3.4-ok:
 	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
 
+#: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
+# Skip for now
+2.6:
+
+#: PyPy 5.0.x with Python 2.7 ...
+pypy-2.7 5.0 5.3:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy2.7 --verify
+
+#: PyPy 2.4.x with Python 3.2 ...
+pypy-3.2 2.4:
+	$(PYTHON) test_pythonlib.py --bytecode-pypy3.2 --verify
+
 clean: clean-py-dis clean-dis clean-unverified
 
 clean-dis:
@@ -102,6 +189,6 @@ clean-dis:
 clean-unverified:
 	find . -name '*_unverified' -exec rm -v '{}' ';'
 
-#: Clean temporary compile/decompile/verify direcotries in /tmp
+#: Clean temporary compile/decompile/verify directories in /tmp
 clean-py-dis:
 	rm -fr /tmp/py-dis-* || true

test/add-test.py

@@ -2,11 +2,15 @@
 """ Trivial helper program to bytecompile and run an uncompile
 """
 import os, sys, py_compile
-assert len(sys.argv) == 2
-path = sys.argv[1]
-short = os.path.basename(path)
+assert len(sys.argv) >= 2
 version = sys.version[0:3]
-cfile =  "bytecode_%s/%s" % (version, short) + 'c'
-print("byte-compiling %s to %s" % (path, cfile))
-py_compile.compile(path, cfile)
-os.system("../bin/uncompyle6 -a -t %s" % cfile)
+for path in sys.argv[1:]:
+    short = os.path.basename(path)
+    if hasattr(sys, 'pypy_version_info'):
+        cfile =  "bytecode_pypy%s/%s" % (version, short) + 'c'
+    else:
+        cfile =  "bytecode_%s/%s" % (version, short) + 'c'
+    print("byte-compiling %s to %s" % (path, cfile))
+    py_compile.compile(path, cfile)
+    if isinstance(version, str) or version >= (2, 6, 0):
+        os.system("../bin/uncompyle6 -a -t %s" % cfile)