Merge branch 'master' into python-2.4

Improve 3.4 coverage and note a 3.5 while bug

.gitignore

@@ -19,3 +19,4 @@ build
 /.venv*
 /.idea
 /.hypothesis
+./ChangeLog

HISTORY.md

@@ -64,14 +64,17 @@ success that his good work deserves.
 Dan Pascu did a bit of work from late 2004 to early 2006 to get this
 code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
-time, various JUMP instructions were classifed as going backwards, and
-COME FROM instructions were reintroduced.  See
+time, various JUMP instructions were classified to assist parsing For
+example, due to the way that code generation and line number table
+work, jump instructions to an earlier offset must be looping jumps,
+such as those found in a "continue" statement; "COME FROM"
+instructions were reintroduced.  See
 [RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt)
-for more details here. There wasn't a public
-release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
-supported. Dan says the Python 2.3 version could verify the entire
-Python library. But given subsequent bugs found like simply
-recognizing complex-number constants in bytecode, decompilation wasn't perfect.
+for more details here. There wasn't a public release of RELEASE-2.4
+and bytecodes other than Python 2.4 weren't supported. Dan says the
+Python 2.3 version could verify the entire Python library. But given
+subsequent bugs found like simply recognizing complex-number constants
+in bytecode, decompilation wasn't perfect.
 
 Next we get to ["uncompyle" and
 PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -109,15 +112,26 @@ Given this, perhaps it is not surprising that subsequent changes
 tended to shy away from using the built-in compiler technology
 mechanisms and addressed problems and extensions by some other means.
 
-Specifically, in `uncompyle`, decompilation of python bytecode 2.5 & 2.6
-is done by transforming the byte code into a pseudo-2.7 Python
-bytecode and is based on code from Eloi Vanderbeken.
+Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
+2.6 is done by transforming the byte code into a pseudo-2.7 Python
+bytecode and is based on code from Eloi Vanderbeken. A bit of this
+could have bene easily added by modifying grammar rules.
 
 This project, `uncompyle6`, abandons that approach for various
-reasons. However the main reason is that we need offsets in fragment
-deparsing to be exactly the same, and the transformation process can
-remove instructions.  _Adding_ instructions with psuedo offsets is
-however okay.
+reasons. Having a grammar per Python version is much cleaner and it
+scales indefinitely. That said, we don't have entire copies of the
+grammar, but work off of differences from some neighboring version.
+
+Should there be a desire to rebase or start a new base version to work
+off of, say for some future Python version, that can be done by
+dumping a grammar for a specific version after it has been loaded
+incrementally. You can get a full dump of the grammar by profiling the
+grammar on a large body of Python source code.
+
+Another problem with pseudo-2.7 bytecode is that that we need offsets
+in fragment deparsing to be exactly the same as the bytecode; the
+transformation process can remove instructions.  _Adding_ instructions
+with psuedo offsets is however okay.
 
 `Uncompyle6` however owes its existence to the fork of `uncompyle2` by
 Myst herie (Mysterie) whose first commit picks up at
@@ -155,25 +169,44 @@ Hartmut a decade an a half ago:
 This project deparses using an Earley-algorithm parse with lots of
 massaging of tokens and the grammar in the scanner
 phase. Earley-algorithm parsers are context free and tend to be linear
-if the grammar is LR or left recursive.
+if the grammar is LR or left recursive. There is a technique for
+improving LL right recursion, but our parser doesn't have that yet.
 
-Another approach that doesn't use grammars is to do something like
-simulate execution symbolically and build expression trees off of
-stack results. Control flow in that apprproach still needs to be
-handled somewhat ad hoc.  The two important projects that work this
-way are [unpyc3](https://code.google.com/p/unpyc3/) and most
-especially [pycdc](https://github.com/zrax/pycdc) The latter project
-is largely by Michael Hansen and Darryl Pogue. If they supported
-getting source-code fragments, did a better job in supporting Python
-more fully, and had a way I could call it from Python, I'd probably
-would have ditched this and used that. The code runs blindingly fast
-and spans all versions of Python, although more recently Python 3
-support has been lagging.
+Another approach to decompiling, and one that doesn't use grammars is
+to do something like simulate execution symbolically and build
+expression trees off of stack results. Control flow in that approach
+still needs to be handled somewhat ad hoc.  The two important projects
+that work this way are [unpyc3](https://code.google.com/p/unpyc3/) and
+most especially [pycdc](https://github.com/zrax/pycdc) The latter
+project is largely by Michael Hansen and Darryl Pogue. If they
+supported getting source-code fragments, did a better job in
+supporting Python more fully, and had a way I could call it from
+Python, I'd probably would have ditched this and used that. The code
+runs blindingly fast and spans all versions of Python, although more
+recently Python 3 support has been lagging. The code is impressive for
+its smallness given that it covers many versions of Python. However, I
+think it has reached a scalability issue, same as all the other
+efforts. To handle Python versions more accurately, I think that code
+base will need to have a lot more code specially which specializes for
+Python versions. And then it will run into a modularity problem.
 
 Tests for the project have been, or are being, culled from all of the
-projects mentioned.
+projects mentioned. Quite a few have been added to improve grammar
+coverage and to address the numerous bugs that have been encountered.
 
-For a little bit of the history of changes to the Early-algorithm parser,
+If you think, as I am sure will happen in the future, "hey, I can just
+write a decompiler from scratch and not have to deal with all all of
+the complexity here", think again. What is likely to happen is that
+you'll get at best a 90% solution working for a single Python release
+that will be obsolete in about a year, and more obsolete each
+subsequent year.  Writing a decompiler for Python gets harder as it
+Python progresses, so writing one for Python 3.7 isn't as easy as it
+was for Python 2.2.  That said, if you still feel you want to write a
+single version decompiler, look at the test cases in this project and
+talk to me. I may have some ideas.
+
+
+For a little bit of the history of changes to the Earley-algorithm parser,
 see the file [NEW-FEATURES.rst](https://github.com/rocky/python-spark/blob/master/NEW-FEATURES.rst) in the [python-spark github repository](https://github.com/rocky/python-spark).
 
 NB. If you find mistakes, want corrections, or want your name added

HOW-TO-REPORT-A-BUG.md

@@ -40,7 +40,7 @@ obfuscation.
 Checking if bytecode is valid is pretty simple: disassemble the code.
 Python comes with a disassembly module called `dis`. A prerequisite
 module for this package, `xdis` has a cross-python version
-disassembler.
+disassembler called `pydisasm`.
 
 ### Semantic equivalence vs. exact source code
 
@@ -74,10 +74,10 @@ else:
 
 ```
 
-may out as `elif`.
+may come out as `elif`.
 
 
-As mentioned in the README. It is possible that Python changes what
+As mentioned in the README, It is possible that Python changes what
 you write to be more efficient. For example, for:
 
 
@@ -126,7 +126,7 @@ provide the input command and the output from that, please give:
 
 Sure, I get it. No problem. There is Python assembly code on parse
 errors, so simply by hand decompile that. To get a full disassembly,
-use pydisasm from the [xdis](https://pypi.python.org/pypi/xdis)
+use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
 package. Opcodes are described in the documentation for
 the [dis](https://docs.python.org/3.6/library/dis.html) module.
 
@@ -144,9 +144,10 @@ there are others around let me know and I'll list them here.)
 
 ## Narrowing the problem
 
-I don't need or want the entire source code base for which one file or
-module can't be decompiled. I just need those file(s) or module(s).
-If there are several files, file a bug report for each file.
+I don't need or want the entire source code base for the file(s) or
+module(s) can't be decompiled. I just need those file(s) or module(s).
+If there are problems in several files, file a bug report for each
+file.
 
 Python modules can get quite large, and usually decompilation problems
 occur in a single function or maybe the main-line code but not any of
@@ -165,8 +166,22 @@ likley the problem will be fixed and fixed sooner.
 
 When you report a bug, you are giving up confidentiality to the source
 code and the byte code. However, I would imagine that if you have
-narrowed the problem sufficiently, confidentiality little that
+narrowed the problem sufficiently, confidentiality of the little that
 remains would not be an issue.
 
 However feel free to remove any commments, and modify variable names
 or constants in the source code.
+
+## Ethics
+
+I do not condone using this program for unethical or illegal purposes.
+More detestful, at least to me, is asking for help to assist you in
+something that might not legitimate.
+
+Don't use the issue tracker for such solicitations. To try to stave
+off illegitimate behavior, you should note that the issue tracker, the
+code, and bugs mentioned in that are in the open: there is no
+confidentiality. You may be asked about the authorship or claimed
+ownership of the bytecode. If I think something is not quite right, I
+may label the issue questionable which may make the it easier those
+who are looking for illegal activity.

NEWS

@@ -1,3 +1,19 @@
+uncompyle6 2.14.0 2017-12-10 Dr. Gecko
+
+- Many decompilation bugfixes
+- Grammar rule reduction and version isolation
+- Match higher-level nonterminal names more closely
+  with Python AST
+- Start automated Python stdlib testing - full round trip
+
+uncompyle6 2.14.0 2017-11-26 johnnybamazing
+
+- Start to isolate grammar rules between versions
+  and remove used grammar rules
+- Fix a number of bytecode decompile problems
+  (many more remain)
+- Add stdlib/runtests.sh for even more rigourous testing
+
 uncompyle6 2.13.3 2017-11-13
 
 Overall: better 3.6 decompiling and some much needed code refactoring and cleanup

README.rst

@@ -3,7 +3,7 @@
 uncompyle6
 ==========
 
-A native Python cross-version Decompiler and Fragment Decompiler.
+A native Python cross-version decompiler and fragment decompiler.
 The successor to decompyle, uncompyle, and uncompyle2.
 
 
@@ -17,29 +17,48 @@ source code. It accepts bytecodes from Python version 1.5, and 2.1 to
 Why this?
 ---------
 
-There were a number of decompyle, uncompile, uncompyle2, uncompyle3
-forks around. All of them came basically from the same code base, and
-almost all of them no were no longer actively maintained. Only one
-handled Python 3, and even there, only 3.2 or 3.3 depending on which
-code is used. This code pulls these together and moves forward. This
-project has the most complete support for Python 3.3 and above. It
-also addresses a number of open issues in the previous forks.
+Ok, I'll say it: this software is amazing. It is more than your
+normal hacky decompiler. Using compiler_ technology, the program
+creates a parse tree of the program from the instructions; nodes at
+the upper levels that look a little like what might come from a Python
+AST. So we can really classify and understand what's going on in
+sections of Python bytecode.
 
-What makes this different from other CPython bytecode decompilers?: its
-ability to deparse just fragments and give source-code information
-around a given bytecode offset.
+Building on this, another thing that makes this different from other
+CPython bytecode decompilers is the ability to deparse just
+*fragments* of source code and give source-code information around a
+given bytecode offset.
 
-I use this to deparse fragments of code inside my trepan_
-debuggers_. For that, I need to record text fragments for all
-bytecode offsets (of interest). This purpose although largely
-compatible with the original intention is yet a little bit different.
+I use the tree fragments to deparse fragments of code inside my
+trepan_ debuggers_. For that, bytecode offsets are recorded and
+associated with fragments of the source code. This purpose, although
+compatible with the original intention, is yet a little bit different.
 See this_ for more information.
 
-The idea of Python fragment deparsing given an instruction offset can
-be used in showing stack traces or any program that wants to show a
-location in more detail than just a line number.  It can be also used
-when source-code information does not exist and there is just bytecode
-information.
+Python fragment deparsing given an instruction offset is useful in
+showing stack traces and can be encorporated into any program that
+wants to show a location in more detail than just a line number at
+runtime.  This code can be also used when source-code information does
+not exist and there is just bytecode. Again, my debuggers make use of
+this.
+
+There were (and still are) a number of decompyle, uncompyle,
+uncompyle2, uncompyle3 forks around. Almost all of them come basically
+from the same code base, and (almost?) all of them are no longer
+actively maintained. One was really good at decompiling Python 1.5-2.3
+or so, another really good at Python 2.7, but that only. Another
+handles Python 3.2 only; another patched that and handled only 3.3.
+You get the idea. This code pulls all of these forks together and
+*moves forward*. There is some serious refactoring and cleanup in this
+code base over those old forks.
+
+This project has the most complete support for Python 3.3 and above
+and the best all-around Python support.
+
+We are serious about testing, and use automated processes to find
+bugs. In the issue trackers for other decompilers, you will find a
+number of bugs we've found along the way. Very few to none of them are
+fixed in the other decompilers.
 
 Requirements
 ------------
@@ -56,11 +75,9 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -e .
-    pip install -r requirements-dev.txt
+    pip install -e .  # set up to run from source tree
+                      # Or if you want to install instead
     python setup.py install # may need sudo
-    # or if you have pyenv:
-    python setup.py develop
 
 A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
@@ -109,14 +126,14 @@ Known Bugs/Restrictions
 -----------------------
 
 The biggest known and possibly fixable (but hard) problem has to do
-with handling control flow. All of the Python decompilers I have looked
-at have the same problem. In some cases we can detect an erroneous
-decompilation and report that.
+with handling control flow. (Python has probably the most diverse and
+screwy set of compound statements I've ever seen; a number of the
+usual ones like else clauses on loops and try blocks I suspect most
+programmers don't know about.)
 
-Over 98% of the decompilation of Python standard library packages in
-Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
-"weakly" verify. Python 2.6 drops down to 96% weakly verifying.
-Other versions drop off in quality too.
+All of the Python decompilers I have looked at have the same
+problem. In some cases we can detect an erroneous decompilation and
+report that.
 
 *Verification* is the process of decompiling bytecode, compiling with
 a Python for that bytecode version, and then comparing the bytecode
@@ -134,6 +151,17 @@ program by running the Python interpreter. Because the Python language
 has changed so much, for best results you should use the same Python
 Version in checking as used in the bytecode.
 
+Finally, we have automated running the standard Python tests after
+first compiling and decompiling the test program. Results here are a
+bit weak (if not better than most other Python decompilers). But over
+time this will probably get better.
+
+Python support is strongest in Python 2 for 2.7 and drops off as you
+get further away from that. Support is also probably pretty good for
+python 2.3-2.4 since a lot of the goodness of early the version of the
+decompiler from that era has been preserved (and Python compilation in
+that era was minimal)
+
 Later distributions average about 200 files. There is some work to do
 on the lower end Python versions which is more difficult for us to
 handle since we don't have a Python interpreter for versions 1.5, 1.6,
@@ -144,7 +172,9 @@ In the Python 3 series, Python support is is strongest around 3.4 or
 3.6 changes things drastically by using word codes rather than byte
 codes. That has been addressed, but then it also changes function call
 opcodes and its semantics and has more problems with control flow than
-3.5 has.
+3.5 has. Between Python 3.5, 3.6 and 3.7 there have been major changes
+to the `MAKE_FUNCTION` and `CALL_FUNCTION` instructions. Those are
+not handled yet.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
@@ -169,14 +199,16 @@ See Also
 
 * https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
 * https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
-* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
+* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
 * The HISTORY_ file.
 * `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
 * https://github.com/rocky/python-xdis : Cross Python version disassembler
 * https://github.com/rocky/python-xasm : Cross Python version assembler
+* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
 
 
 .. _trepan: https://pypi.python.org/pypi/trepan2
+.. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake

__pkginfo__.py

@@ -39,8 +39,8 @@ entry_points       = {
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.7.1, < 1.8.0',
-                      'xdis >= 3.6.1, < 3.7.0']
+install_requires   = ['spark-parser >= 1.8.4, < 1.9.0',
+                      'xdis >= 3.6.2, < 3.7.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'

admin-tools/check-older-versions.sh

@@ -13,8 +13,6 @@ if ! source ./setup-python-2.4.sh ; then
     exit $?
 fi
 
-PYVERSIONS='2.7.14 2.6.9 3.3.6 3.4.2 3.5.4 3.6.3'
-
 cd ..
 for version in $PYVERSIONS; do
     if ! pyenv local $version ; then

admin-tools/how-to-make-a-release.md

@@ -2,9 +2,9 @@
 **Table of Contents**
 
 - [Get latest sources:](#get-latest-sources)
-- [Change version in uncompyle6/version.py. Then:](#change-version-in-uncompyle6versionpy-then)
+- [Change version in uncompyle6/version.py](#change-version-in-uncompyle6versionpy)
 - [Update ChangeLog:](#update-changelog)
-- [Update NEWS from ChangeLog. Then:](#update-news-from-changelog-then)
+- [Update NEWS from ChangeLog:](#update-news-from-changelog)
 - [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
 - [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
 - [Update NEWS from master branch](#update-news-from-master-branch)
@@ -19,7 +19,7 @@
 
     $ . ./admin-tool/update-sources.sh
 
-# Change version in uncompyle6/version.py. Then:
+# Change version in uncompyle6/version.py:
 
 	$ emacs uncompyle6/version.py
     $ source uncompyle6/version.py
@@ -30,7 +30,7 @@
 
     $ make ChangeLog
 
-#  Update NEWS from ChangeLog. Then:
+#  Update NEWS from ChangeLog:
 
 	$ emacs NEWS
     $ make check
@@ -46,7 +46,7 @@
     $ source admin-tools/setup-python-2.4.sh
     $ rm ChangeLog
 
-    # $ git merge master ?
+    $ git merge master
 
 # Update NEWS from master branch
 
@@ -55,14 +55,13 @@
 # Check against all versions
 
     $ source admin-tools/check-older-versions.sh
-    $ source admin-tools/check-newer-versions.sh
 
 # Make packages and tag
 
-    $ admin-tools/make-dist-older.sh
+    $ . ./admin-tools/make-dist-older.sh
     $ git tag release-python-2.4-$VERSION
 
-    $ admin-tools/make-dist-newer.sh
+    $ . /admin-tools/make-dist-newer.sh
     $ git tag release-$VERSION
 
 # Upload single package and look at Rst Formating

admin-tools/pyenv-all-versions

@@ -0,0 +1,19 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv versions we have
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+
+olddir=$(pwd)
+mydir=$(dirname ${BASH_SOURCE[0]})
+cd $mydir
+
+all=""
+for file in pyenv-{olde{st,r},newer}-versions ; do
+    . $mydir/$file
+    all="$all $PYVERSIONS"
+done
+
+PYVERSIONS="$all"
+cd $olddir

admin-tools/pyenv-newer-versions

@@ -1,6 +1,8 @@
 # -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the master branch.
 if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1
 fi
-export PYVERSIONS='3.5.2 3.6.2 2.6.9 3.3.6 2.7.13 3.4.2'
+export PYVERSIONS='3.5.3 3.6.3 2.6.9 3.3.6 2.7.14 3.4.2'

admin-tools/pyenv-older-versions

@@ -1,4 +1,7 @@
 # -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the python-2.4 branch.
+
 if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
     echo "This script should be *sourced* rather than run directly through bash"
     exit 1

admin-tools/pyenv-oldest-versions

@@ -0,0 +1,9 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv the oldest versions we have.
+# These are not covered (yet) by uncompyle6, although
+# some programs do work here.
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='2.1.3 2.2.3 2.3.7'

admin-tools/setup-master.sh

@@ -16,7 +16,7 @@ fi
 mydir=$(dirname $bs)
 fulldir=$(readlink -f $mydir)
 cd $fulldir/..
-(cd ../python-spark && git checkout master && pyenv local $PYTHON_VERSION) && \
-    (cd ../python-xdis && git checkout master && pyenv local $PYTHON_VERSION) && \
-    git checkout master && pyenv local $PYTHON_VERSION
+(cd ../python-spark && git checkout master && pyenv local $PYTHON_VERSION) && git pull && \
+    (cd ../python-xdis && git checkout master && pyenv local $PYTHON_VERSION) && git pull && \
+    git checkout master && pyenv local $PYTHON_VERSION && git pull
 cd $owd

admin-tools/setup-python-2.4.sh

@@ -10,7 +10,7 @@ fi
 mydir=$(dirname $bs)
 fulldir=$(readlink -f $mydir)
 cd $fulldir/..
-(cd ../python-spark && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && \
-    (cd ../python-xdis && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && \
-    git checkout python-2.4 &&  pyenv local $PYTHON_VERSION
+(cd ../python-spark && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && git pull && \
+    (cd ../python-xdis && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && git pull && \
+    git checkout python-2.4 &&  pyenv local $PYTHON_VERSION && git pull
 cd $owd

pytest/test_grammar.py

@@ -14,29 +14,61 @@ def test_grammar():
             "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar())
 
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
-    lhs, rhs, tokens, right_recursive = p.check_sets()
+    (lhs, rhs, tokens,
+     right_recursive, dup_rhs) = p.check_sets()
     expect_lhs = set(['expr1024', 'pos_arg'])
-    unused_rhs = set(['build_list', 'call_function', 'mkfunc',
+    unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
-                      'unpack', 'unpack_list'])
-    expect_right_recursive = frozenset([('designList',
-                                         ('designator', 'DUP_TOP', 'designList'))])
+                      'unpack',])
+    expect_right_recursive = set([('designList',
+                                   ('store', 'DUP_TOP', 'designList'))])
     if PYTHON3:
         expect_lhs.add('load_genexpr')
+        expect_lhs.add('kvlist')
+        expect_lhs.add('kv3')
 
         unused_rhs = unused_rhs.union(set("""
-        except_pop_except genexpr classdefdeco2 listcomp
+        except_pop_except generator_exp classdefdeco2
+        dict
         """.split()))
-        if 3.0 <= PYTHON_VERSION:
+        if PYTHON_VERSION >= 3.0:
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
+            if PYTHON_VERSION < 3.6:
+                # 3.6 has at least one non-custom call rule
+                # the others don't
+                unused_rhs.add('call')
+                if PYTHON_VERSION == 3.5:
+                    expect_right_recursive.add((('l_stmts',
+                                                 ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                    pass
+                pass
+            else:
+                expect_right_recursive.add((('l_stmts',
+                                             ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                unused_rhs.add('build_map_unpack_with_call')
+                unused_rhs.add('unmapexpr')
+                # expect_lhs.add('kwargs1')
+                pass
             pass
+        pass
     else:
         expect_lhs.add('kwarg')
+        unused_rhs.add('call')
+
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)
     assert expect_right_recursive == right_recursive
+
+    expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                ('LOAD_CONST',),
+                                ('JUMP_BACK',), ('JUMP_FORWARD',)])
+    # reduced_dup_rhs = {k: dup_rhs[k] for k in dup_rhs if k not in expect_dup_rhs}
+    # for k in reduced_dup_rhs:
+    #     print(k, reduced_dup_rhs[k])
+    # assert not reduced_dup_rhs, reduced_dup_rhs
+
     s = get_scanner(PYTHON_VERSION, IS_PYPY)
     ignore_set = set(
             """

test-unit/test_grammar.py

@@ -0,0 +1,55 @@
+import re
+import unittest
+from uncompyle6 import PYTHON_VERSION, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser, python_parser
+
+class TestGrammar(unittest.TestCase):
+    def test_grammar(self):
+
+        def check_tokens(tokens, opcode_set):
+            remain_tokens = set(tokens) - opcode_set
+            remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+            remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+            remain_tokens = set(remain_tokens) - opcode_set
+            self.assertEqual(remain_tokens, set([]),
+                    "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar()))
+
+        p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+        (lhs, rhs, tokens,
+         right_recursive, dup_rhs) = p.check_sets()
+        expect_lhs = set(['expr1024', 'pos_arg'])
+        unused_rhs = set(['list', 'call', 'mkfunc',
+                          'mklambda',
+                          'unpack',])
+
+        expect_right_recursive = frozenset([('designList',
+                                             ('store', 'DUP_TOP', 'designList'))])
+        expect_lhs.add('kwarg')
+
+        self.assertEqual(expect_lhs, set(lhs))
+        self.assertEqual(unused_rhs, set(rhs))
+        self.assertEqual(expect_right_recursive, right_recursive)
+
+        expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                    ('LOAD_CONST',),
+                                    ('JUMP_BACK',), ('JUMP_FORWARD',)])
+
+        reduced_dup_rhs = {}
+        for k in dup_rhs:
+            if k not in expect_dup_rhs:
+                reduced_dup_rhs[k] = dup_rhs[k]
+                pass
+            pass
+        for k in reduced_dup_rhs:
+            print(k, reduced_dup_rhs[k])
+        # assert not reduced_dup_rhs, reduced_dup_rhs
+
+    def test_dup_rule(self):
+        import inspect
+        python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
+                      is_pypy=IS_PYPY,
+                      parser_debug={
+                          'dups': True, 'transition': False, 'reduce': False,
+                          'rules': False, 'errorstack': None, 'context': True})
+if __name__ == '__main__':
+    unittest.main()

test/Makefile

@@ -1,4 +1,12 @@
-PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests
+PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
+      check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
+      check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
+      check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
+      check-3.4 check-3.5 check-5.6 5.6 5.8 \
+      grammar-coverage-2.5 grammar-coverage-2.6 grammarcoverage-2.7 \
+      grammar-coverage-3.1 grammar-coverage-3.2 grammarcoverage-3.3 \
+      grammar-coverage-3.4 grammar-coverage-3.5 grammarcoverage-3.6
+
 
 GIT2CL ?= git2cl
 PYTHON ?= python
@@ -8,6 +16,7 @@ NATIVE_CHECK = check-$(PYTHON_VERSION)
 
 # Set COMPILE='--compile' to force compilation before check
 COMPILE ?=
+COVER_DIR=../tmp/grammar-cover
 
 # Run short tests
 check-short:
@@ -43,7 +52,7 @@ check-3.4: check-bytecode check-3.4-ok check-2.7-ok
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.6
 check-3.6: check-bytecode
@@ -58,8 +67,7 @@ check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1:
-	$(PYTHON) test_pythonlib.py --bytecode-1.5
+check-bytecode-1: check-bytecode-1.5
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -81,6 +89,10 @@ check-bytecode: check-bytecode-3
         --bytecode-pypy2.7 --bytecode-1
 
 
+#: Check deparsing bytecode 1.5 only
+check-bytecode-1.5:
+	$(PYTHON) test_pythonlib.py --bytecode-1.5
+
 #: Check deparsing Python 2.1
 check-bytecode-2.1:
 	$(PYTHON) test_pythonlib.py --bytecode-2.1
@@ -101,53 +113,102 @@ check-bytecode-2.4:
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
-#: Check deparsing Python 3.0
-check-bytecode-3.0:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0
-
-#: Check deparsing Python 3.1
-check-bytecode-3.1:
-	$(PYTHON) test_pythonlib.py --bytecode-3.1
-
-#: Check deparsing Python 3.2
-check-bytecode-3.2:
-	$(PYTHON) test_pythonlib.py --bytecode-3.2
-
-#: Check deparsing Python 3.3
-check-bytecode-3.3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.3
-
-#: Check deparsing Python 3.4
-check-bytecode-3.4:
-	$(PYTHON) test_pythonlib.py --bytecode-3.4
-
-#: Check deparsing Python 3.5
-check-bytecode-3.5:
-	$(PYTHON) test_pythonlib.py --bytecode-3.5
-
-#: Check deparsing Python 3.6
-check-bytecode-3.6:
-	$(PYTHON) test_pythonlib.py --bytecode-3.6
-
 #: Get grammar coverage for Python 2.4
 grammar-coverage-2.4:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
+	-rm $(COVER_DIR)/spark-grammar-24.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
 
 #: Get grammar coverage for Python 2.5
 grammar-coverage-2.5:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+	-rm $(COVER_DIR)/spark-grammar-25.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
 
 #: Get grammar coverage for Python 2.6
 grammar-coverage-2.6:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+	-rm $(COVER_DIR)/spark-grammar-26.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
 
 #: Get grammar coverage for Python 2.7
 grammar-coverage-2.7:
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
-	SPARK_PARSER_COVERAGE=/tmp/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+	-rm $(COVER_DIR)/spark-grammar-27.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+
+#: Get grammar coverage for Python 3.0
+grammar-coverage-3.0:
+	-rm $(COVER_DIR)/spark-grammar-30.cover
+
+SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pyenvlib.py --3.0.1
+
+#: Get grammar coverage for Python 3.1
+grammar-coverage-3.1:
+	-rm $(COVER_DIR)/spark-grammar-31.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pyenvlib.py --3.1.5
+
+#: Get grammar coverage for Python 3.2
+grammar-coverage-3.2:
+	-rm $(COVER_DIR)/spark-grammar-32.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pythonlib.py --bytecode-3.2
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pyenvlib.py --3.2.6
+
+#: Get grammar coverage for Python 3.3
+grammar-coverage-3.3:
+	-rm $(COVER_DIR)/spark-grammar-33.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pyenvlib.py --3.3.6
+
+##: Get grammar coverage for Python 3.4
+grammar-coverage-3.4:
+	-rm $(COVER_DIR)/spark-grammar-34.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pyenvlib.py --3.4.2
+
+##: Get grammar coverage for Python 3.5
+grammar-coverage-3.5:
+	rm $(COVER_DIR)/spark-grammar-35.cover || /bin/true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pyenvlib.py --3.5.3
+
+#: Check deparsing Python 2.6
+check-bytecode-2.6:
+	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
+
+#: Check deparsing Python 2.7
+check-bytecode-2.7:
+	$(PYTHON) test_pythonlib.py --bytecode-2.7 --weak-verify
+
+#: Check deparsing Python 3.0
+check-bytecode-3.0:
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify
+
+#: Check deparsing Python 3.1
+check-bytecode-3.1:
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
+
+#: Check deparsing Python 3.2
+check-bytecode-3.2:
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
+
+#: Check deparsing Python 3.3
+check-bytecode-3.3:
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
+
+#: Check deparsing Python 3.4
+check-bytecode-3.4:
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify
+
+#: Check deparsing Python 3.5
+check-bytecode-3.5:
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
+
+#: Check deparsing Python 3.6
+check-bytecode-3.6:
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
 
 #: short tests for bytecodes only for this version of Python
 check-native-short:
@@ -163,15 +224,15 @@ check-2.6-ok:
 
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:
-	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-2.7 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.2's lib files known to be okay
 check-3.2-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.2 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.4's lib files known to be okay
 check-3.4-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.4 --weak-verify $(COMPILE)
 
 #: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
 # Skip for now