Merge branch 'master' into python-2.4

and in verification status message

.gitignore

@@ -19,3 +19,4 @@ build
 /.venv*
 /.idea
 /.hypothesis
+ChangeLog

.travis.yml

@@ -3,13 +3,7 @@ language: python
 sudo: false
 
 python:
-  - '3.5'
-  - '2.7.12'
-  - '2.6'
-  - '3.3'
-  - '3.4'
-  - '3.2'
-  - '3.6'
+  - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
 install:
 - pip install -e .

HISTORY.md

@@ -64,14 +64,17 @@ success that his good work deserves.
 Dan Pascu did a bit of work from late 2004 to early 2006 to get this
 code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
-time, various JUMP instructions were classifed as going backwards, and
-COME FROM instructions were reintroduced.  See
+time, various JUMP instructions were classified to assist parsing For
+example, due to the way that code generation and line number table
+work, jump instructions to an earlier offset must be looping jumps,
+such as those found in a "continue" statement; "COME FROM"
+instructions were reintroduced.  See
 [RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt)
-for more details here. There wasn't a public
-release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
-supported. Dan says the Python 2.3 version could verify the entire
-Python library. But given subsequent bugs found like simply
-recognizing complex-number constants in bytecode, decompilation wasn't perfect.
+for more details here. There wasn't a public release of RELEASE-2.4
+and bytecodes other than Python 2.4 weren't supported. Dan says the
+Python 2.3 version could verify the entire Python library. But given
+subsequent bugs found like simply recognizing complex-number constants
+in bytecode, decompilation wasn't perfect.
 
 Next we get to ["uncompyle" and
 PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -109,15 +112,26 @@ Given this, perhaps it is not surprising that subsequent changes
 tended to shy away from using the built-in compiler technology
 mechanisms and addressed problems and extensions by some other means.
 
-Specifically, in `uncompyle`, decompilation of python bytecode 2.5 & 2.6
-is done by transforming the byte code into a pseudo-2.7 Python
-bytecode and is based on code from Eloi Vanderbeken.
+Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
+2.6 is done by transforming the byte code into a pseudo-2.7 Python
+bytecode and is based on code from Eloi Vanderbeken. A bit of this
+could have bene easily added by modifying grammar rules.
 
 This project, `uncompyle6`, abandons that approach for various
-reasons. However the main reason is that we need offsets in fragment
-deparsing to be exactly the same, and the transformation process can
-remove instructions.  _Adding_ instructions with psuedo offsets is
-however okay.
+reasons. Having a grammar per Python version is much cleaner and it
+scales indefinitely. That said, we don't have entire copies of the
+grammar, but work off of differences from some neighboring version.
+
+Should there be a desire to rebase or start a new base version to work
+off of, say for some future Python version, that can be done by
+dumping a grammar for a specific version after it has been loaded
+incrementally. You can get a full dump of the grammar by profiling the
+grammar on a large body of Python source code.
+
+Another problem with pseudo-2.7 bytecode is that that we need offsets
+in fragment deparsing to be exactly the same as the bytecode; the
+transformation process can remove instructions.  _Adding_ instructions
+with psuedo offsets is however okay.
 
 `Uncompyle6` however owes its existence to the fork of `uncompyle2` by
 Myst herie (Mysterie) whose first commit picks up at
@@ -155,25 +169,44 @@ Hartmut a decade an a half ago:
 This project deparses using an Earley-algorithm parse with lots of
 massaging of tokens and the grammar in the scanner
 phase. Earley-algorithm parsers are context free and tend to be linear
-if the grammar is LR or left recursive.
+if the grammar is LR or left recursive. There is a technique for
+improving LL right recursion, but our parser doesn't have that yet.
 
-Another approach that doesn't use grammars is to do something like
-simulate execution symbolically and build expression trees off of
-stack results. Control flow in that apprproach still needs to be
-handled somewhat ad hoc.  The two important projects that work this
-way are [unpyc3](https://code.google.com/p/unpyc3/) and most
-especially [pycdc](https://github.com/zrax/pycdc) The latter project
-is largely by Michael Hansen and Darryl Pogue. If they supported
-getting source-code fragments, did a better job in supporting Python
-more fully, and had a way I could call it from Python, I'd probably
-would have ditched this and used that. The code runs blindingly fast
-and spans all versions of Python, although more recently Python 3
-support has been lagging.
+Another approach to decompiling, and one that doesn't use grammars is
+to do something like simulate execution symbolically and build
+expression trees off of stack results. Control flow in that approach
+still needs to be handled somewhat ad hoc.  The two important projects
+that work this way are [unpyc3](https://code.google.com/p/unpyc3/) and
+most especially [pycdc](https://github.com/zrax/pycdc) The latter
+project is largely by Michael Hansen and Darryl Pogue. If they
+supported getting source-code fragments, did a better job in
+supporting Python more fully, and had a way I could call it from
+Python, I'd probably would have ditched this and used that. The code
+runs blindingly fast and spans all versions of Python, although more
+recently Python 3 support has been lagging. The code is impressive for
+its smallness given that it covers many versions of Python. However, I
+think it has reached a scalability issue, same as all the other
+efforts. To handle Python versions more accurately, I think that code
+base will need to have a lot more code specially which specializes for
+Python versions. And then it will run into a modularity problem.
 
 Tests for the project have been, or are being, culled from all of the
-projects mentioned.
+projects mentioned. Quite a few have been added to improve grammar
+coverage and to address the numerous bugs that have been encountered.
 
-For a little bit of the history of changes to the Early-algorithm parser,
+If you think, as I am sure will happen in the future, "hey, I can just
+write a decompiler from scratch and not have to deal with all all of
+the complexity here", think again. What is likely to happen is that
+you'll get at best a 90% solution working for a single Python release
+that will be obsolete in about a year, and more obsolete each
+subsequent year.  Writing a decompiler for Python gets harder as it
+Python progresses, so writing one for Python 3.7 isn't as easy as it
+was for Python 2.2.  That said, if you still feel you want to write a
+single version decompiler, look at the test cases in this project and
+talk to me. I may have some ideas.
+
+
+For a little bit of the history of changes to the Earley-algorithm parser,
 see the file [NEW-FEATURES.rst](https://github.com/rocky/python-spark/blob/master/NEW-FEATURES.rst) in the [python-spark github repository](https://github.com/rocky/python-spark).
 
 NB. If you find mistakes, want corrections, or want your name added

HOW-TO-REPORT-A-BUG.md

@@ -10,23 +10,36 @@ decompile everything. Overall, I think this one probably does the best
 job of *any* Python decompiler that handles such a wide range of
 versions.
 
-But at any given time, there are maybe dozens of valid Python bytecode
-files that I know of that will cause problems. And when I get through
-those and all the issues of decompiler bugs that are currently logged,
-I could probably easily find dozens more bugs just by doing a
-decompile of all the Python bytecode on any one of my
-computers. Unless you want to help out by _fixing_ bugs, or are
-willing to do work by isolating and narrowing bugs, don't feel you are
-doing me a favor by doing scans on your favorite sets of bytecode
-files.
+But at any given time, there are a number of valid Python bytecode
+files that I know of that will cause problems. See, for example, the
+list in
+[`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
 
-In sum, it is not uncommon that you will find a mistranslation in
-decompiling. Furthermore, you may be expected to do some work in order
-to have your bug worthy of being considered above other bugs.
+But I understand: you would the bugs _you_ encounter addressed before
+all the other known bugs.
 
-No one is getting paid to work to work on this project, let alone bugs
-you may have an interest in. If you require decompiling bytecode
-immediately, consider using a decompilation service.
+From my standpoint, the good thing about the bugs listed in
+`runtests.sh` is that each test case is small and isolated to a single
+kind of problem. And I'll tend to fix easier, more isolated cases than
+generic "something's wrong" kinds of bugs where I'd have to do a bit
+of work to figure out what's up, if not use some sort of mind reading,
+make some guesses, and perform some experiments to see if the guesses
+are correct. I can't read minds, nor am I into guessing games; I'd
+rather devote the effort spent instead towards fixing bugs that are
+precisely defined.
+
+And it often turns out that by just fixing the well-defined and
+prescribed cases, the ill-defined amorphous cases as well will get
+handled as well.
+
+In sum, you may need to do some work to have the bug you have found
+handled before the hundreds of other bugs, and things I could be
+doing.
+
+No one is getting paid to work to work on this project, let alone the
+bugs you may have an interest in. If you require decompiling bytecode
+immediately, consider using a decompilation service, listed further
+down in this document.
 
 ## Is it really a bug?
 
@@ -40,7 +53,7 @@ obfuscation.
 Checking if bytecode is valid is pretty simple: disassemble the code.
 Python comes with a disassembly module called `dis`. A prerequisite
 module for this package, `xdis` has a cross-python version
-disassembler.
+disassembler called `pydisasm`.
 
 ### Semantic equivalence vs. exact source code
 
@@ -114,7 +127,7 @@ Also try to narrow the bug. See below.
 Some kind folks also give the invocation they used and the output
 which usually includes an error message produced. This is
 helpful. From this, I can figure out what OS you are running this on
-and what version of *uncomplye6* was used. Therefore, if you don't
+and what version of *uncomplye6* was used. Therefore, if you _don't_
 provide the input command and the output from that, please give:
 
 * _uncompyle6_ version used
@@ -126,7 +139,7 @@ provide the input command and the output from that, please give:
 
 Sure, I get it. No problem. There is Python assembly code on parse
 errors, so simply by hand decompile that. To get a full disassembly,
-use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis)
+use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
 package. Opcodes are described in the documentation for
 the [dis](https://docs.python.org/3.6/library/dis.html) module.
 

Makefile

@@ -39,7 +39,7 @@ check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
 check-3.7: pytest
 
 #:Tests for Python 2.6 (doesn't have pytest)
-check-2.6:
+check-2.4 check-2.5 check-2.6:
 	$(MAKE) -C test $@
 
 #:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0

NEWS

@@ -1,3 +1,41 @@
+uncompyle6 2.14.3 2017-01-19
+
+- Fix bug in 3.5+ await stmt
+- Better version to magic handling; handle 3.5.2 .. 3.5.4 versions
+- Improve/correct test_pyenvlib.py status messages
+- Fix some 2.7 and 2.6 parser bugs
+- Fix whilelse parsing bugs
+- Correct 2.5- decorator parsing
+- grammar for decorators matchies AST more a little better
+- better tests in setup.py for running the right version of Python
+- Fix 2.6- parsing of "for .. try/else" ...  with "continue"  inside
+
+uncompyle6 2.14.2 2017-01-09 Samish
+
+Decompilation bug fixes, mostly 3.6 and pre 2.7
+
+- 3.6 FUNCTION_EX (somewhat)
+- 3.6 FUNCTION_EX_KW fixes
+- 3.6 MAKE_FUNCTION fixes
+- correct 3.5 CALL_FUNCTION_VAR
+- stronger 3.x "while 1" testing
+- Fix bug in if's with "pass" bodies. Fixes #104
+- try/else and try/finally fixes on 2.6-
+- limit pypy customization to pypy
+- Add addr fields in COME_FROMS
+- Allow use of full instructions in parser reduction routines
+- Reduce grammar in Pythion 3 by specialization more to specific
+  Python versions
+- Match Python AST names more closely when possible
+
+uncompyle6 2.14.1 2017-12-10 Dr. Gecko
+
+- Many decompilation bugfixes
+- Grammar rule reduction and version isolation
+- Match higher-level nonterminal names more closely
+  with Python AST
+- Start automated Python stdlib testing - full round trip
+
 uncompyle6 2.14.0 2017-11-26 johnnybamazing
 
 - Start to isolate grammar rules between versions

README.rst

@@ -3,7 +3,7 @@
 uncompyle6
 ==========
 
-A native Python cross-version Decompiler and Fragment Decompiler.
+A native Python cross-version decompiler and fragment decompiler.
 The successor to decompyle, uncompyle, and uncompyle2.
 
 
@@ -17,11 +17,40 @@ source code. It accepts bytecodes from Python version 1.5, and 2.1 to
 Why this?
 ---------
 
-There were a number of decompyle, uncompyle, uncompyle2, uncompyle3
-forks around. All of them came basically from the same code base, and
-almost all of them no were no longer actively maintained. Only one
-handled Python 3, and even there, only 3.2 or 3.3 depending on which
-code is used. This code pulls these together and moves forward.
+Ok, I'll say it: this software is amazing. It is more than your
+normal hacky decompiler. Using compiler_ technology, the program
+creates a parse tree of the program from the instructions; nodes at
+the upper levels that look a little like what might come from a Python
+AST. So we can really classify and understand what's going on in
+sections of Python bytecode.
+
+Building on this, another thing that makes this different from other
+CPython bytecode decompilers is the ability to deparse just
+*fragments* of source code and give source-code information around a
+given bytecode offset.
+
+I use the tree fragments to deparse fragments of code inside my
+trepan_ debuggers_. For that, bytecode offsets are recorded and
+associated with fragments of the source code. This purpose, although
+compatible with the original intention, is yet a little bit different.
+See this_ for more information.
+
+Python fragment deparsing given an instruction offset is useful in
+showing stack traces and can be encorporated into any program that
+wants to show a location in more detail than just a line number at
+runtime.  This code can be also used when source-code information does
+not exist and there is just bytecode. Again, my debuggers make use of
+this.
+
+There were (and still are) a number of decompyle, uncompyle,
+uncompyle2, uncompyle3 forks around. Almost all of them come basically
+from the same code base, and (almost?) all of them are no longer
+actively maintained. One was really good at decompiling Python 1.5-2.3
+or so, another really good at Python 2.7, but that only. Another
+handles Python 3.2 only; another patched that and handled only 3.3.
+You get the idea. This code pulls all of these forks together and
+*moves forward*. There is some serious refactoring and cleanup in this
+code base over those old forks.
 
 This project has the most complete support for Python 3.3 and above
 and the best all-around Python support.
@@ -29,31 +58,16 @@ and the best all-around Python support.
 We are serious about testing, and use automated processes to find
 bugs. In the issue trackers for other decompilers, you will find a
 number of bugs we've found along the way. Very few to none of them are
-not fixed in the other decompilers.
-
-Another thing that makes this different from other CPython bytecode
-decompilers is the ability to deparse just fragments and give
-source-code information around a given bytecode offset.
-
-I use this to deparse fragments of code inside my trepan_
-debuggers_. For that, I need to record text fragments for all
-bytecode offsets (of interest). This purpose although largely
-compatible with the original intention is yet a little bit different.
-See this_ for more information.
-
-The idea of Python fragment deparsing given an instruction offset can
-be used in showing stack traces or any program that wants to show a
-location in more detail than just a line number.  It can be also used
-when source-code information does not exist and there is just bytecode
-information.
+fixed in the other decompilers.
 
 Requirements
 ------------
 
-This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1.
-Python versions 2.4-2.7 are supported in the python-2.4 branch.
-The bytecode files it can read has been tested on Python bytecodes from
-versions 1.5, 2.1-2.7, and 3.0-3.6 and the above-mentioned PyPy versions.
+The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
+or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
+python-2.4 branch.  The bytecode files it can read have been tested on
+Python bytecodes from versions 1.5, 2.1-2.7, and 3.0-3.6 and the
+above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -62,11 +76,9 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -e .
-    pip install -r requirements-dev.txt
+    pip install -e .  # set up to run from source tree
+                      # Or if you want to install instead
     python setup.py install # may need sudo
-    # or if you have pyenv:
-    python setup.py develop
 
 A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
@@ -116,13 +128,13 @@ Known Bugs/Restrictions
 
 The biggest known and possibly fixable (but hard) problem has to do
 with handling control flow. (Python has probably the most diverse and
-screwy set of compound statements I've ever seen; a number of the
-usual ones like else clauses on loops and try blocks I suspect most
-programmers don't know aobut.)
+screwy set of compound statements I've ever seen; there
+are "else" clauses on loops and try blocks that I suspect many
+programmers don't know about.)
 
-All of the Python decompilers I have looked at have the same
-problem. In some cases we can detect an erroneous decompilation and
-report that.
+All of the Python decompilers that I have looked at have problems
+decompiling Python's control flow. In some cases we can detect an
+erroneous decompilation and report that.
 
 *Verification* is the process of decompiling bytecode, compiling with
 a Python for that bytecode version, and then comparing the bytecode
@@ -151,10 +163,9 @@ python 2.3-2.4 since a lot of the goodness of early the version of the
 decompiler from that era has been preserved (and Python compilation in
 that era was minimal)
 
-Later distributions average about 200 files. There is some work to do
-on the lower end Python versions which is more difficult for us to
-handle since we don't have a Python interpreter for versions 1.5, 1.6,
-and 2.0.
+There is some work to do on the lower end Python versions which is
+more difficult for us to handle since we don't have a Python
+interpreter for versions 1.5, 1.6, and 2.0.
 
 In the Python 3 series, Python support is is strongest around 3.4 or
 3.3 and drops off as you move further away from those versions. Python
@@ -175,10 +186,12 @@ handled.
 
 We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
 Deobfuscator_ to unscramble the bytecode to get valid bytecode before
-trying this tool.
-
-Handling pathologically long lists of expressions or statements is
-slow.
+trying this tool. This program can't decompile Microsoft Windows EXE
+files created by Py2EXE_, although we can probably decompile the code
+after you extract the bytecode properly. For situations like this, you
+might want to consider a decompilation service like `Crazy Compilers
+<http://www.crazy-compilers.com/decompyle/>`_.  Handling
+pathologically long lists of expressions or statements is slow.
 
 
 There is lots to do, so please dig in and help.
@@ -193,9 +206,11 @@ See Also
 * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
+* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2
+.. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
@@ -205,3 +220,4 @@ See Also
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
+.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe

__pkginfo__.py

@@ -9,7 +9,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015-2017 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2018 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 5 - Production/Stable',
@@ -26,6 +26,7 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
                 'Programming Language :: Python :: 3.6',
+                'Programming Language :: Python :: 3.7',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -39,13 +40,13 @@ entry_points       = {
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.8.0, < 1.9.0',
-                      'xdis >= 3.6.1, < 3.7.0', 'six']
+install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
+                      'xdis >= 3.6.6, < 3.7.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 py_modules         = None
-short_desc         = 'Python cross-version byte-code deparser'
+short_desc         = 'Python cross-version byte-code decompiler'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

admin-tools/check-older-versions.sh

@@ -13,8 +13,6 @@ if ! source ./setup-python-2.4.sh ; then
     exit $?
 fi
 
-PYVERSIONS='2.7.14 2.6.9 3.3.6 3.4.2 3.5.4 3.6.3'
-
 cd ..
 for version in $PYVERSIONS; do
     if ! pyenv local $version ; then

admin-tools/how-to-make-a-release.md

@@ -2,9 +2,9 @@
 **Table of Contents**
 
 - [Get latest sources:](#get-latest-sources)
-- [Change version in uncompyle6/version.py. Then:](#change-version-in-uncompyle6versionpy-then)
+- [Change version in uncompyle6/version.py](#change-version-in-uncompyle6versionpy)
 - [Update ChangeLog:](#update-changelog)
-- [Update NEWS from ChangeLog. Then:](#update-news-from-changelog-then)
+- [Update NEWS from ChangeLog:](#update-news-from-changelog)
 - [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
 - [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
 - [Update NEWS from master branch](#update-news-from-master-branch)
@@ -19,9 +19,9 @@
 
     $ . ./admin-tool/update-sources.sh
 
-# Change version in uncompyle6/version.py. Then:
+# Change version in uncompyle6/version.py:
 
-	$ emacs uncompyle6/version.py
+    $ emacs uncompyle6/version.py
     $ source uncompyle6/version.py
     $ echo $VERSION
     $ git commit -m"Get ready for release $VERSION" .
@@ -30,9 +30,9 @@
 
     $ make ChangeLog
 
-#  Update NEWS from ChangeLog. Then:
+#  Update NEWS from ChangeLog:
 
-	$ emacs NEWS
+    $ emacs NEWS
     $ make check
     $ git commit --amend .
     $ git push   # get CI testing going early
@@ -44,25 +44,18 @@
 # Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
 
     $ source admin-tools/setup-python-2.4.sh
-    $ rm ChangeLog
+    $ git merge master
 
-    # $ git merge master ?
-
-# Update NEWS from master branch
-
-    $ git commit -m"Get ready for release $VERSION" .
-
-# Check against all versions
+# Check against older versions
 
     $ source admin-tools/check-older-versions.sh
-    $ source admin-tools/check-newer-versions.sh
 
 # Make packages and tag
 
-    $ admin-tools/make-dist-older.sh
+    $ . ./admin-tools/make-dist-older.sh
     $ git tag release-python-2.4-$VERSION
 
-    $ admin-tools/make-dist-newer.sh
+    $ . ./admin-tools/make-dist-newer.sh
     $ git tag release-$VERSION
 
 # Upload single package and look at Rst Formating

admin-tools/how-to-make-a-release.txt

@@ -0,0 +1,46 @@
+git pull
+
+Change version in uncompyle6/version.py
+source uncompyle6/version.py
+echo $VERSION
+git commit -m"Get ready for release $VERSION" .
+
+Update ChangeLog:
+  make ChangeLog
+
+Update NEWS from ChangeLog
+make check
+
+git commit --amend .
+
+git push
+
+Make sure pyenv is running
+# Pyenv
+
+source admin-tools/check-newer-versions.sh
+
+
+# Switch to python-2.4 and build that first...
+source admin-tools/setup-python-2.4
+
+rm ChangeLog
+git merge master
+
+Update NEWS from master branch
+
+git commit -m"Get ready for release $VERSION" .
+
+source admin-tools/check-older-versions.sh
+source admin-tools/check-newer-versions.sh
+
+make-dist-older.sh
+
+git tag release-python-2.4-$VERSION
+
+./make-dist-newer.sh
+
+git tag release-$VERSION
+
+
+twine upload dist/uncompyle6-${VERSION}*

admin-tools/pyenv-all-versions

@@ -0,0 +1,19 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv versions we have
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+
+olddir=$(pwd)
+mydir=$(dirname ${BASH_SOURCE[0]})
+cd $mydir
+
+all=""
+for file in pyenv-{olde{st,r},newer}-versions ; do
+    . $mydir/$file
+    all="$all $PYVERSIONS"
+done
+
+PYVERSIONS="$all"
+cd $olddir

admin-tools/pyenv-newer-versions

@@ -1,6 +1,8 @@
 # -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the master branch.
 if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.2 3.6.2 2.6.9 3.3.6 2.7.14 3.4.2'
+export PYVERSIONS='3.5.3 3.6.3 2.6.9 3.3.6 2.7.14 3.4.2'

admin-tools/pyenv-older-versions

@@ -1,4 +1,7 @@
 # -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the python-2.4 branch.
+
 if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1

admin-tools/pyenv-oldest-versions

@@ -0,0 +1,9 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv the oldest versions we have.
+# These are not covered (yet) by uncompyle6, although
+# some programs do work here.
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='2.1.3 2.2.3 2.3.7'

admin-tools/update-sources.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+cd $(dirname ${BASH_SOURCE[0]})/..
+git pull

circle.yml

@@ -7,7 +7,7 @@ machine:
 dependencies:
     override:
         - pip install -e .
-        - pip install -r requirements-dev.txt
+        - pip install pytest==3.2.5 hypothesis
 test:
     override:
-        - python ./setup.py develop && make check-2.7
+        - python ./setup.py develop && make check-2.6

pytest/test_deparse.py

@@ -29,7 +29,7 @@ def list_comp():
     [y for y in range(3)]
 
 def get_parsed_for_fn(fn):
-    code = fn.__code__ if PYTHON3 else fn.func_code
+    code = fn.func_code
     return deparse(PYTHON_VERSION, code)
 
 def check_expect(expect, parsed):

pytest/test_docstring.py

@@ -10,7 +10,7 @@ else:
     maxint = sys.maxint
 from uncompyle6.semantics.helper import print_docstring
 
-class PrintFake():
+class PrintFake:
     def __init__(self):
         self.pending_newlines = 0
         self.f = StringIO()

pytest/test_fjt.py

@@ -22,12 +22,13 @@ def bug_loop(disassemble, tb=None):
     disassemble(tb)
 
 def test_if_in_for():
-    code = bug.__code__
+    code = bug.func_code
     scan = get_scanner(PYTHON_VERSION)
-    print(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
         scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
         scan.build_prev_op(n)
         fjt = scan.find_jump_targets(False)
 
@@ -64,6 +65,9 @@ def test_if_in_for():
         scan.build_lines_data(code)
         scan.build_prev_op()
         scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \

pytest/test_fstring.py

@@ -1,150 +0,0 @@
-# std
-import os
-# test
-import pytest
-import hypothesis
-from hypothesis import strategies as st
-# uncompyle6
-from uncompyle6 import PYTHON_VERSION, deparse_code
-
-
-@st.composite
-def expressions(draw):
-    # todo : would be nice to generate expressions using hypothesis however
-    # this is pretty involved so for now just use a corpus of expressions
-    # from which to select.
-    return draw(st.sampled_from((
-        'abc',
-        'len(items)',
-        'x + 1',
-        'lineno',
-        'container',
-        'self.attribute',
-        'self.method()',
-        # These expressions are failing, I think these are control
-        # flow problems rather than problems with FORMAT_VALUE,
-        # however I need to confirm this...
-        #'sorted(items, key=lambda x: x.name)',
-        #'func(*args, **kwargs)',
-        #'text or default',
-        #'43 if life_the_universe and everything else None'
-    )))
-
-
-@st.composite
-def format_specifiers(draw):
-    """
-    Generate a valid format specifier using the rules:
-
-    format_spec ::=  [[fill]align][sign][#][0][width][,][.precision][type]
-    fill        ::=  <any character>
-    align       ::=  "<" | ">" | "=" | "^"
-    sign        ::=  "+" | "-" | " "
-    width       ::=  integer
-    precision   ::=  integer
-    type        ::=  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"
-
-    See https://docs.python.org/2/library/string.html
-
-    :param draw: Let hypothesis draw from other strategies.
-
-    :return: An example format_specifier.
-    """
-    alphabet_strategy = st.characters(min_codepoint=ord('a'), max_codepoint=ord('z'))
-    fill = draw(st.one_of(alphabet_strategy, st.none()))
-    align = draw(st.sampled_from(list('<>=^')))
-    fill_align = (fill + align or '') if fill else ''
-
-    type_ = draw(st.sampled_from('bcdeEfFgGnosxX%'))
-    can_have_sign = type_ in 'deEfFgGnoxX%'
-    can_have_comma = type_ in 'deEfFgG%'
-    can_have_precision = type_ in 'fFgG'
-    can_have_pound = type_ in 'boxX%'
-    can_have_zero = type_ in 'oxX'
-
-    sign = draw(st.sampled_from(list('+- ') + [''])) if can_have_sign else ''
-    pound = draw(st.sampled_from(('#', '',))) if can_have_pound else ''
-    zero = draw(st.sampled_from(('0', '',))) if can_have_zero else ''
-
-    int_strategy = st.integers(min_value=1, max_value=1000)
-
-    width = draw(st.one_of(int_strategy, st.none()))
-    width = str(width) if width is not None else ''
-
-    comma = draw(st.sampled_from((',', '',))) if can_have_comma else ''
-    if can_have_precision:
-        precision = draw(st.one_of(int_strategy, st.none()))
-        precision = '.' + str(precision) if precision else ''
-    else:
-        precision = ''
-
-    return ''.join((fill_align, sign, pound, zero, width, comma, precision, type_,))
-
-
-@st.composite
-def fstrings(draw):
-    """
-    Generate a valid f-string.
-    See https://www.python.org/dev/peps/pep-0498/#specification
-
-    :param draw: Let hypothsis draw from other strategies.
-
-    :return: A valid f-string.
-    """
-    character_strategy = st.characters(
-        blacklist_characters='\r\n\'\\s{}',
-        min_codepoint=1,
-        max_codepoint=1000,
-    )
-    is_raw = draw(st.booleans())
-    integer_strategy = st.integers(min_value=0, max_value=3)
-    expression_count = draw(integer_strategy)
-    content = []
-    for _ in range(expression_count):
-        expression = draw(expressions())
-        conversion = draw(st.sampled_from(('', '!s', '!r', '!a',)))
-        has_specifier = draw(st.booleans())
-        specifier = ':' + draw(format_specifiers()) if has_specifier else ''
-        content.append('{{{}{}}}'.format(expression, conversion, specifier))
-        content.append(draw(st.text(character_strategy)))
-    content = ''.join(content)
-    return "f{}'{}'".format('r' if is_raw else '', content)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(format_specifiers())
-def test_format_specifiers(format_specifier):
-    """Verify that format_specifiers generates valid specifiers"""
-    try:
-        exec('"{:' + format_specifier + '}".format(0)')
-    except ValueError as e:
-        if 'Unknown format code' not in str(e):
-            raise
-
-
-def run_test(text):
-    hypothesis.assume(len(text))
-    hypothesis.assume("f'{" in text)
-    expr = text + '\n'
-    code = compile(expr, '<string>', 'single')
-    deparsed = deparse_code(PYTHON_VERSION, code, compile_mode='single')
-    recompiled = compile(deparsed.text, '<string>', 'single')
-    if recompiled != code:
-        assert 'dis(' + deparsed.text.strip('\n') + ')' == 'dis(' + expr.strip('\n') + ')'
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@hypothesis.given(fstrings())
-def test_uncompyle_fstring(fstring):
-    """Verify uncompyling fstring bytecode"""
-    run_test(fstring)
-
-
-@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
-@pytest.mark.parametrize('fstring', [
-    "f'{abc}{abc!s}'",
-    "f'{abc}0'",
-])
-def test_uncompyle_direct(fstring):
-    """useful for debugging"""
-    run_test(fstring)

pytest/test_function_call.py

@@ -1,175 +0,0 @@
-# std
-import string
-# 3rd party
-from hypothesis import given, assume, example, settings, strategies as st
-import pytest
-# uncompyle
-from validate import validate_uncompyle
-from test_fstring import expressions
-
-
-alpha = st.sampled_from(string.ascii_lowercase)
-numbers = st.sampled_from(string.digits)
-alphanum = st.sampled_from(string.ascii_lowercase + string.digits)
-
-
-@st.composite
-def function_calls(draw,
-                   min_keyword_args=0, max_keyword_args=5,
-                   min_positional_args=0, max_positional_args=5,
-                   min_star_args=0, max_star_args=1,
-                   min_double_star_args=0, max_double_star_args=1):
-    """
-    Strategy factory for generating function calls.
-
-    :param draw: Callable which draws examples from other strategies.
-
-    :return: The function call text.
-    """
-    st_positional_args = st.lists(
-        alpha,
-        min_size=min_positional_args,
-        max_size=max_positional_args
-    )
-    st_keyword_args = st.lists(
-        alpha,
-        min_size=min_keyword_args,
-        max_size=max_keyword_args
-    )
-    st_star_args = st.lists(
-        alpha,
-        min_size=min_star_args,
-        max_size=max_star_args
-    )
-    st_double_star_args = st.lists(
-        alpha,
-        min_size=min_double_star_args,
-        max_size=max_double_star_args
-    )
-
-    positional_args = draw(st_positional_args)
-    keyword_args = draw(st_keyword_args)
-    st_values = st.lists(
-        expressions(),
-        min_size=len(keyword_args),
-        max_size=len(keyword_args)
-    )
-    keyword_args = [
-        x + '=' + e
-        for x, e in
-        zip(keyword_args, draw(st_values))
-    ]
-    star_args = ['*' + x for x in draw(st_star_args)]
-    double_star_args = ['**' + x for x in draw(st_double_star_args)]
-
-    arguments = positional_args + keyword_args + star_args + double_star_args
-    draw(st.randoms()).shuffle(arguments)
-    arguments = ','.join(arguments)
-
-    function_call = 'fn({arguments})'.format(arguments=arguments)
-    try:
-        # TODO: Figure out the exact rules for ordering of positional, keyword,
-        # star args, double star args and in which versions the various
-        # types of arguments are supported so we don't need to check that the
-        # expression compiles like this.
-        compile(function_call, '<string>', 'single')
-    except:
-        assume(False)
-    return function_call
-
-
-def test_function_no_args():
-    validate_uncompyle("fn()")
-
-
-def isolated_function_calls(which):
-    """
-    Returns a strategy for generating function calls, but isolated to 
-    particular types of arguments, for example only positional arguments.
-
-    This can help reason about debugging errors in specific types of function
-    calls.
-
-    :param which: One of 'keyword', 'positional', 'star', 'double_star'
-
-    :return: Strategy for generating an function call isolated to specific
-             argument types.
-    """
-    kwargs = dict(
-        max_keyword_args=0,
-        max_positional_args=0,
-        max_star_args=0,
-        max_double_star_args=0,
-    )
-    kwargs['_'.join(('min', which, 'args'))] = 1
-    kwargs['_'.join(('max', which, 'args'))] = 5 if 'star' not in which else 1
-    return function_calls(**kwargs)
-
-
-with settings(max_examples=25):
-
-    @given(isolated_function_calls('positional'))
-    @example("fn(0)")
-    def test_function_positional_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('keyword'))
-    @example("fn(a=0)")
-    def test_function_call_keyword_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('star'))
-    @example("fn(*items)")
-    def test_function_call_star_only(expr):
-        validate_uncompyle(expr)
-
-    @given(isolated_function_calls('double_star'))
-    @example("fn(**{})")
-    def test_function_call_double_star_only(expr):
-        validate_uncompyle(expr)
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(w=0,m=0,**v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_MAP_UNPACK_WITH_CALL_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(a=0,**g)")
-
-
-@pytest.mark.xfail()
-def test_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*g,**j)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*z,u=0)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(**a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_MAP_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(b,b,b=0,*a)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_TUPLE_BUILD_TUPLE_UNPACK_WITH_CALL_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(*c,v)")
-
-
-@pytest.mark.xfail()
-def test_BUILD_CONST_KEY_MAP_CALL_FUNCTION_EX():
-    validate_uncompyle("fn(i=0,y=0,*p)")
-
-
-@pytest.mark.skip(reason='skipping property based test until all individual tests are passing')
-@given(function_calls())
-def test_function_call(function_call):
-    validate_uncompyle(function_call)

pytest/test_grammar.py

@@ -16,32 +16,48 @@ def test_grammar():
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
     (lhs, rhs, tokens,
      right_recursive, dup_rhs) = p.check_sets()
-    expect_lhs = set(['expr1024', 'pos_arg'])
-    unused_rhs = set(['build_list', 'call_function', 'mkfunc',
+
+    # We have custom rules that create the below
+    expect_lhs = set(['expr1024', 'pos_arg', 'get_iter', 'attribute'])
+
+    unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
                       'unpack',])
-
-    expect_right_recursive = frozenset([('designList',
-                                         ('designator', 'DUP_TOP', 'designList'))])
+    expect_right_recursive = set([('designList',
+                                   ('store', 'DUP_TOP', 'designList'))])
     if PYTHON3:
         expect_lhs.add('load_genexpr')
+        expect_lhs.add('kvlist')
+        expect_lhs.add('kv3')
 
         unused_rhs = unused_rhs.union(set("""
-        except_pop_except genexpr classdefdeco2 listcomp
+        except_pop_except generator_exp classdefdeco2
+        dict
         """.split()))
-        if 3.0 <= PYTHON_VERSION:
+        if PYTHON_VERSION >= 3.0:
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
+            unused_rhs.add('call')
+            if PYTHON_VERSION < 3.6:
+                # 3.6 has at least one non-custom call rule
+                # the others don't
+                unused_rhs.add('call')
+                if PYTHON_VERSION == 3.5:
+                    expect_right_recursive.add((('l_stmts',
+                                                 ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                    pass
+                pass
+            else:
+                expect_right_recursive.add((('l_stmts',
+                                             ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                # expect_lhs.add('kwargs1')
+                pass
             pass
+        pass
     else:
         expect_lhs.add('kwarg')
-
-    # FIXME: grammar remove_rule on
-    #  kv3 ::= expr expr STORE_MAP
-    # doesn't currently work in grammar, so we have this extraneous kv3 around.
-    if 3.3 <= PYTHON_VERSION <= 3.4:
-        expect_lhs.add('kv3')
+        unused_rhs.add('call')
 
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)
@@ -50,9 +66,9 @@ def test_grammar():
     expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
                                 ('LOAD_CONST',),
                                 ('JUMP_BACK',), ('JUMP_FORWARD',)])
-    reduced_dup_rhs = {k: dup_rhs[k] for k in dup_rhs if k not in expect_dup_rhs}
-    for k in reduced_dup_rhs:
-        print(k, reduced_dup_rhs[k])
+    # reduced_dup_rhs = {k: dup_rhs[k] for k in dup_rhs if k not in expect_dup_rhs}
+    # for k in reduced_dup_rhs:
+    #     print(k, reduced_dup_rhs[k])
     # assert not reduced_dup_rhs, reduced_dup_rhs
 
     s = get_scanner(PYTHON_VERSION, IS_PYPY)

pytest/test_pysource.py

@@ -1,4 +1,6 @@
+import sys
 from uncompyle6 import PYTHON3
+from uncompyle6.scanner import get_scanner
 from uncompyle6.semantics.consts import (
     escape, NONE,
     # RETURN_NONE, PASS, RETURN_LOCALS
@@ -6,18 +8,17 @@ from uncompyle6.semantics.consts import (
 
 if PYTHON3:
     from io import StringIO
-    def iteritems(d):
-        return d.items()
 else:
     from StringIO import StringIO
-    def iteritems(d):
-        return d.iteritems()
 
 from uncompyle6.semantics.pysource import SourceWalker as SourceWalker
 
 def test_template_engine():
     s = StringIO()
-    sw = SourceWalker(2.7, s, None)
+    sys_version = float(sys.version[0:3])
+    scanner = get_scanner(sys_version, is_pypy=False)
+    scanner.insts = []
+    sw = SourceWalker(2.7, s, scanner)
     sw.ast = NONE
     sw.template_engine(('--%c--', 0), NONE)
     print(sw.f.getvalue())
@@ -25,7 +26,7 @@ def test_template_engine():
     # FIXME: and so on...
 
 from uncompyle6.semantics.consts import (
-    TABLE_DIRECT, TABLE_R,
+    TABLE_R, TABLE_DIRECT,
     )
 
 from uncompyle6.semantics.fragments import (
@@ -39,7 +40,7 @@ def test_tables():
             (TABLE_DIRECT, 'TABLE_DIRECT', False),
             (TABLE_R, 'TABLE_R', False),
             (TABLE_DIRECT_FRAGMENT, 'TABLE_DIRECT_FRAGMENT', True)):
-        for k, entry in iteritems(t):
+        for k, entry in t.iteritems():
             if k in skip_for_now:
                 continue
             fmt = entry[0]

pytest/test_single_compile.py

@@ -1,19 +1,19 @@
-import pytest
-from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
+from uncompyle6 import PYTHON_VERSION, deparse_code
 
-def test_single_mode():
-    single_expressions = (
-        'i = 1',
-        'i and (j or k)',
-        'i += 1',
-        'i = j % 4',
-        'i = {}',
-        'i = []',
-        'for i in range(10):\n    i\n',
-        'for i in range(10):\n    for j in range(10):\n        i + j\n',
-        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
-    )
+if PYTHON_VERSION >= 2.5:
+    def test_single_mode():
+        single_expressions = (
+            'i = 1',
+            'i and (j or k)',
+            'i += 1',
+            'i = j % 4',
+            'i = {}',
+            'i = []',
+            'for i in range(10):\n    i\n',
+            'for i in range(10):\n    for j in range(10):\n        i + j\n',
+            'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+        )
 
-    for expr in single_expressions:
-        code = compile(expr + '\n', '<string>', 'single')
-        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'
+        for expr in single_expressions:
+            code = compile(expr + '\n', '<string>', 'single')
+            assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

pytest/testdata/if-2.7.right

@@ -7,6 +7,6 @@
    7       6  LOAD_NAME             1  'False'
            9  STORE_NAME            2  'b'
           12  JUMP_FORWARD          0  'to 15'
-        15_0  COME_FROM                '12'
+        15_0  COME_FROM            12  '12'
           15  LOAD_CONST            0  ''
-          18  RETURN_VALUE     
+          18  RETURN_VALUE

pytest/testdata/ifelse-2.7.right

@@ -10,6 +10,6 @@
 
    6      15  LOAD_CONST            1  2
           18  STORE_NAME            2  'd'
-        21_0  COME_FROM                '12'
+        21_0  COME_FROM            12  '12'
           21  LOAD_CONST            2  ''
-          24  RETURN_VALUE     
+          24  RETURN_VALUE

pytest/validate.py

@@ -1,24 +1,25 @@
-# future
-from __future__ import print_function
 # std
 import os
 import difflib
 import subprocess
 import tempfile
-import functools
-# compatability
-import six
+
+from StringIO import StringIO
 # uncompyle6 / xdis
 from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
 # TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
 from xdis.bytecode import Bytecode
 from xdis.main import get_opcode
 opc = get_opcode(PYTHON_VERSION, IS_PYPY)
-Bytecode = functools.partial(Bytecode, opc=opc)
 
+try:
+    import functools
+    Bytecode = functools.partial(Bytecode, opc=opc)
+    def _dis_to_text(co):
+        return Bytecode(co).dis()
+except:
+    pass
 
-def _dis_to_text(co):
-    return Bytecode(co).dis()
 
 
 def print_diff(original, uncompyled):
@@ -42,8 +43,11 @@ def print_diff(original, uncompyled):
         print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
         diff = difflib.HtmlDiff().make_table(*args)
 
-    with tempfile.NamedTemporaryFile(delete=False) as f:
+    f = tempfile.NamedTemporaryFile(delete=False)
+    try:
         f.write(str(diff).encode('utf-8'))
+    finally:
+        f.close()
 
     try:
         print()
@@ -60,8 +64,7 @@ def print_diff(original, uncompyled):
         print('\nFor side by side diff install elinks')
         diff = difflib.Differ().compare(original_lines, uncompyled_lines)
         print('\n'.join(diff))
-    finally:
-        os.unlink(f.name)
+    os.unlink(f.name)
 
 
 def are_instructions_equal(i1, i2):
@@ -123,8 +126,9 @@ def validate_uncompyle(text, mode='exec'):
     original_text = text
 
     deparsed = deparse_code(PYTHON_VERSION, original_code,
+
                             compile_mode=mode,
-                            out=six.StringIO(),
+                            out=StringIO(),
                             is_pypy=IS_PYPY)
     uncompyled_text = deparsed.text
     uncompyled_code = compile(uncompyled_text, '<string>', 'exec')

setup.py

@@ -1,7 +1,20 @@
 #!/usr/bin/env python
-
 """Setup script for the 'uncompyle6' distribution."""
 
+import sys
+
+SYS_VERSION = sys.version_info[0:2]
+if not ((2, 4) <= SYS_VERSION  <= (2, 7)):
+    mess = "Python Release 2.4 .. 2.7 are supported in this code branch."
+    if ((3, 2) <= SYS_VERSION  <= (3, 7)):
+        mess += ("\nFor your Python, version %s, use the master code/branch." %
+                 sys.version[0:3])
+    else:
+        mess += ("\nThis package is not supported before Python 2.4. Your Python version is %s."
+                 % sys.version[0:3])
+    print(mess)
+    raise Exception(mess)
+
 from __pkginfo__ import \
     author,           author_email,       install_requires,          \
     license,          long_description,   classifiers,               \

test-unit/test_grammar.py

@@ -0,0 +1,55 @@
+import re
+import unittest
+from uncompyle6 import PYTHON_VERSION, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser, python_parser
+
+class TestGrammar(unittest.TestCase):
+    def test_grammar(self):
+
+        def check_tokens(tokens, opcode_set):
+            remain_tokens = set(tokens) - opcode_set
+            remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+            remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+            remain_tokens = set(remain_tokens) - opcode_set
+            self.assertEqual(remain_tokens, set([]),
+                    "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar()))
+
+        p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+        (lhs, rhs, tokens,
+         right_recursive, dup_rhs) = p.check_sets()
+        expect_lhs = set(['expr1024', 'pos_arg'])
+        unused_rhs = set(['list', 'call', 'mkfunc',
+                          'mklambda',
+                          'unpack',])
+
+        expect_right_recursive = frozenset([('designList',
+                                             ('store', 'DUP_TOP', 'designList'))])
+        expect_lhs.add('kwarg')
+
+        self.assertEqual(expect_lhs, set(lhs))
+        self.assertEqual(unused_rhs, set(rhs))
+        self.assertEqual(expect_right_recursive, right_recursive)
+
+        expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                    ('LOAD_CONST',),
+                                    ('JUMP_BACK',), ('JUMP_FORWARD',)])
+
+        reduced_dup_rhs = {}
+        for k in dup_rhs:
+            if k not in expect_dup_rhs:
+                reduced_dup_rhs[k] = dup_rhs[k]
+                pass
+            pass
+        for k in reduced_dup_rhs:
+            print(k, reduced_dup_rhs[k])
+        # assert not reduced_dup_rhs, reduced_dup_rhs
+
+    def test_dup_rule(self):
+        import inspect
+        python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
+                      is_pypy=IS_PYPY,
+                      parser_debug={
+                          'dups': True, 'transition': False, 'reduce': False,
+                          'rules': False, 'errorstack': None, 'context': True})
+if __name__ == '__main__':
+    unittest.main()

test/Makefile

@@ -1,4 +1,12 @@
-PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests
+PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
+      check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
+      check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
+      check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
+      check-3.4 check-3.5 check-5.6 5.6 5.8 \
+      grammar-coverage-2.5 grammar-coverage-2.6 grammarcoverage-2.7 \
+      grammar-coverage-3.1 grammar-coverage-3.2 grammarcoverage-3.3 \
+      grammar-coverage-3.4 grammar-coverage-3.5 grammarcoverage-3.6
+
 
 GIT2CL ?= git2cl
 PYTHON ?= python
@@ -20,7 +28,7 @@ check:
 	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1 check-native-short
+check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-native-short
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
@@ -44,7 +52,7 @@ check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.6
 check-3.6: check-bytecode
@@ -59,8 +67,7 @@ check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1:
-	$(PYTHON) test_pythonlib.py --bytecode-1.5
+check-bytecode-1: check-bytecode-1.5
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -82,6 +89,10 @@ check-bytecode: check-bytecode-3
         --bytecode-pypy2.7 --bytecode-1
 
 
+#: Check deparsing bytecode 1.5 only
+check-bytecode-1.5:
+	$(PYTHON) test_pythonlib.py --bytecode-1.5
+
 #: Check deparsing Python 2.1
 check-bytecode-2.1:
 	$(PYTHON) test_pythonlib.py --bytecode-2.1
@@ -102,6 +113,12 @@ check-bytecode-2.4:
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
+#: Get grammar coverage for Python 2.4
+grammar-coverage-2.4:
+	-rm $(COVER_DIR)/spark-grammar-24.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
+
 #: Get grammar coverage for Python 2.5
 grammar-coverage-2.5:
 	-rm $(COVER_DIR)/spark-grammar-25.cover
@@ -120,6 +137,19 @@ grammar-coverage-2.7:
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
 
+#: Get grammar coverage for Python 3.0
+grammar-coverage-3.0:
+	-rm $(COVER_DIR)/spark-grammar-30.cover
+
+SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pyenvlib.py --3.0.1
+
+#: Get grammar coverage for Python 3.1
+grammar-coverage-3.1:
+	-rm $(COVER_DIR)/spark-grammar-31.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pyenvlib.py --3.1.5
+
 #: Get grammar coverage for Python 3.2
 grammar-coverage-3.2:
 	-rm $(COVER_DIR)/spark-grammar-32.cover
@@ -132,20 +162,20 @@ grammar-coverage-3.3:
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pyenvlib.py --3.3.6
 
-##: Get grammar coverage for Python 3.4
+#: Get grammar coverage for Python 3.4
 grammar-coverage-3.4:
 	-rm $(COVER_DIR)/spark-grammar-34.cover
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pyenvlib.py --3.4.2
 
-##: Get grammar coverage for Python 3.5
+#: Get grammar coverage for Python 3.5
 grammar-coverage-3.5:
 	rm $(COVER_DIR)/spark-grammar-35.cover || /bin/true
 	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
-	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pyenvlib.py --3.5.2
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pyenvlib.py --3.5.3
 
 #: Check deparsing Python 2.6
-pcheck-bytecode-2.6:
+check-bytecode-2.6:
 	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
 
 #: Check deparsing Python 2.7
@@ -154,19 +184,19 @@ check-bytecode-2.7:
 
 #: Check deparsing Python 3.0
 check-bytecode-3.0:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify
 
 #: Check deparsing Python 3.1
 check-bytecode-3.1:
-	$(PYTHON) test_pythonlib.py --bytecode-3.1
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
-	$(PYTHON) test_pythonlib.py --bytecode-3.2
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
 
 #: Check deparsing Python 3.3
 check-bytecode-3.3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.3
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
 
 #: Check deparsing Python 3.4
 check-bytecode-3.4:
@@ -174,31 +204,35 @@ check-bytecode-3.4:
 
 #: Check deparsing Python 3.5
 check-bytecode-3.5:
-	$(PYTHON) test_pythonlib.py --bytecode-3.5
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
 
 #: Check deparsing Python 3.6
 check-bytecode-3.6:
-	$(PYTHON) test_pythonlib.py --bytecode-3.6
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 
 #: short tests for bytecodes only for this version of Python
 check-native-short:
 	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)
 
+#: Run longer Python 2.6's lib files known to be okay
+check-2.4-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.4 --verify $(COMPILE)
+
 #: Run longer Python 2.6's lib files known to be okay
 check-2.6-ok:
 	$(PYTHON) test_pythonlib.py --ok-2.6 --weak-verify $(COMPILE)
 
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:
-	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-2.7 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.2's lib files known to be okay
 check-3.2-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.2 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.4's lib files known to be okay
 check-3.4-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.4 --weak-verify $(COMPILE)
 
 #: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
 # Skip for now