Merge branch 'master' into python-2.4

Some overall cleanup. But again we need to attack all of this more head on.

Closes Issue #149

.gitignore

@@ -16,3 +16,7 @@
 /unpyc
 __pycache__
 build
+/.venv*
+/.idea
+/.hypothesis
+ChangeLog

.travis.yml

@@ -6,7 +6,7 @@ python:
   - '2.7' # this is a cheat here because travis doesn't do 2.4-2.6
 
 install:
-- pip install -r requirements.txt
+- pip install -e .
 - pip install -r requirements-dev.txt
 
 script:

HISTORY.md

@@ -30,7 +30,7 @@ another clever idea: using table-driven semantics routines, using
 format specifiers.
 
 The last mention of a release of SPARK from John is around 2002. As
-released, although the Early Algorithm parser was in good shape, this
+released, although the Earley Algorithm parser was in good shape, this
 code was woefully lacking as serious Python deparser.
 
 In the fall of 2000, Hartmut Goebel
@@ -44,7 +44,8 @@ it appears that Hartmut did most of the work to get this code to
 accept the full Python language. He added precedence to the table
 specifiers, support for multiple versions of Python, the
 pretty-printing of docstrings, lists, and hashes. He also wrote test and verification routines of
-deparsed bytecode, and used this in an extensive set of tests that he also wrote. He could verify against the entire Python library.
+deparsed bytecode, and used this in an extensive set of tests that he also wrote. He says he could verify against the
+entire Python library. However I have subsequently found small and relatively obscure bugs in the decompilation code.
 
 decompyle2.2 was packaged for Debian (sarge) by
 [Ben Burton around 2002](https://packages.qa.debian.org/d/decompyle.html). As
@@ -63,12 +64,17 @@ success that his good work deserves.
 Dan Pascu did a bit of work from late 2004 to early 2006 to get this
 code to handle first Python 2.3 and then 2.4 bytecodes. Because of
 jump optimization introduced in the CPython bytecode compiler at that
-time, various JUMP instructions were classifed as going backwards, and
-COME FROM instructions were reintroduced.  See
-RELEASE-2.4-CHANGELOG.txt for more details here. There wasn't a public
-release of RELEASE-2.4 and bytecodes other than Python 2.4 weren't
-supported. Dan says the Python 2.3 version could verify the entire
-python library.
+time, various JUMP instructions were classified to assist parsing For
+example, due to the way that code generation and line number table
+work, jump instructions to an earlier offset must be looping jumps,
+such as those found in a "continue" statement; "COME FROM"
+instructions were reintroduced.  See
+[RELEASE-2.4-CHANGELOG.txt](https://github.com/rocky/python-uncompyle6/blob/master/DECOMPYLE-2.4-CHANGELOG.txt)
+for more details here. There wasn't a public release of RELEASE-2.4
+and bytecodes other than Python 2.4 weren't supported. Dan says the
+Python 2.3 version could verify the entire Python library. But given
+subsequent bugs found like simply recognizing complex-number constants
+in bytecode, decompilation wasn't perfect.
 
 Next we get to ["uncompyle" and
 PyPI](https://pypi.python.org/pypi/uncompyle/1.1) and the era of
@@ -95,17 +101,39 @@ so. Then hamled made a few commits earler on, while Eike Siewertsen
 made a few commits later on. But mostly wibiti, and Guenther
 Starnberger got the code to where uncompyle2 was around 2012.
 
-In uncompyle2 decompilation of python bytecode 2.5 & 2.6 is done by
-transforming the byte code into a a pseudo 2.7 python bytecode and is
-based on code from Eloi Vanderbeken.
+While John Aycock and Hartmut Goebel were well versed in compiler
+technology, those that have come afterwards don't seem to have been as
+facile in it.  Furthermore, documentation or guidance on how the
+decompiler code worked, comparison to a conventional compiler
+pipeline, how to add new constructs, or debug grammars was weak. Some
+of the grammar tracing and error reporting was a bit weak as well.
 
-This project, uncompyle6, abandons that approach for various
-reasons. However the main reason is that we need offsets in fragment
-deparsing to be exactly the same, and the transformation process can
-remove instructions.  Adding instructions with psuedo_offsets is
-however okay.
+Given this, perhaps it is not surprising that subsequent changes
+tended to shy away from using the built-in compiler technology
+mechanisms and addressed problems and extensions by some other means.
 
-Uncompyle6, however owes its existence to the fork of uncompyle2 by
+Specifically, in `uncompyle`, decompilation of python bytecode 2.5 &
+2.6 is done by transforming the byte code into a pseudo-2.7 Python
+bytecode and is based on code from Eloi Vanderbeken. A bit of this
+could have bene easily added by modifying grammar rules.
+
+This project, `uncompyle6`, abandons that approach for various
+reasons. Having a grammar per Python version is much cleaner and it
+scales indefinitely. That said, we don't have entire copies of the
+grammar, but work off of differences from some neighboring version.
+
+Should there be a desire to rebase or start a new base version to work
+off of, say for some future Python version, that can be done by
+dumping a grammar for a specific version after it has been loaded
+incrementally. You can get a full dump of the grammar by profiling the
+grammar on a large body of Python source code.
+
+Another problem with pseudo-2.7 bytecode is that that we need offsets
+in fragment deparsing to be exactly the same as the bytecode; the
+transformation process can remove instructions.  _Adding_ instructions
+with psuedo offsets is however okay.
+
+`Uncompyle6` however owes its existence to the fork of `uncompyle2` by
 Myst herie (Mysterie) whose first commit picks up at
 2012. I chose this since it seemed to have been at that time the most
 actively, if briefly, worked on. Also starting around 2012 is Dark
@@ -115,9 +143,12 @@ I started working on this late 2015, mostly to add fragment support.
 In that, I decided to make this runnable on Python 3.2+ and Python 2.6+
 while, handling Python bytecodes from Python versions 2.5+ and
 3.2+. In doing so, it has been expedient to separate this into three
-projects: load loading and disassembly (xdis), parsing and tree
-building (spark_parser), and grammar and semantic actions for
-decompiling (uncompyle6).
+projects:
+
+* marshaling/unmarshaling, bytecode loading and disassembly ([xdis](https://pypi.python.org/pypi/xdis)),
+* parsing and tree building ([spark_parser](https://pypi.python.org/pypi/spark_parser)),
+* this project - grammar and semantic actions for decompiling
+  ([uncompyle6](https://pypi.python.org/pypi/uncompyle6)).
 
 
 Over the many years, code styles and Python features have
@@ -135,23 +166,48 @@ Hartmut a decade an a half ago:
     NB. This is not a masterpiece of software, but became more like a hack.
     Probably a complete rewrite would be sensefull. hG/2000-12-27
 
-This project deparses using an Early-algorithm parse with lots of
+This project deparses using an Earley-algorithm parse with lots of
 massaging of tokens and the grammar in the scanner
-phase. Early-algorithm parsers are context free and tend to be linear
-if the grammar is LR or left recursive.
+phase. Earley-algorithm parsers are context free and tend to be linear
+if the grammar is LR or left recursive. There is a technique for
+improving LL right recursion, but our parser doesn't have that yet.
 
-Another approach that doesn't use grammars is to do something like
-simulate execution symbolically and build expression trees off of
-stack results. The two important projects that work this way are
-[unpyc3](https://code.google.com/p/unpyc3/) and most especially
-[pycdc](https://github.com/zrax/pycdc) The latter project is largely
-by Michael Hansen and Darryl Pogue. If they supported getting
-source-code fragments and I could call it from Python, I'd probably
-ditch this and use that. From what I've seen, the code runs blindingly
-fast and spans all versions of Python.
+Another approach to decompiling, and one that doesn't use grammars is
+to do something like simulate execution symbolically and build
+expression trees off of stack results. Control flow in that approach
+still needs to be handled somewhat ad hoc.  The two important projects
+that work this way are [unpyc3](https://code.google.com/p/unpyc3/) and
+most especially [pycdc](https://github.com/zrax/pycdc) The latter
+project is largely by Michael Hansen and Darryl Pogue. If they
+supported getting source-code fragments, did a better job in
+supporting Python more fully, and had a way I could call it from
+Python, I'd probably would have ditched this and used that. The code
+runs blindingly fast and spans all versions of Python, although more
+recently Python 3 support has been lagging. The code is impressive for
+its smallness given that it covers many versions of Python. However, I
+think it has reached a scalability issue, same as all the other
+efforts. To handle Python versions more accurately, I think that code
+base will need to have a lot more code specially which specializes for
+Python versions. And then it will run into a modularity problem.
 
 Tests for the project have been, or are being, culled from all of the
-projects mentioned.
+projects mentioned. Quite a few have been added to improve grammar
+coverage and to address the numerous bugs that have been encountered.
 
-NB. If you find mistakes, want corrections, or want your name added (or removed),
-please contact me.
+If you think, as I am sure will happen in the future, "hey, I can just
+write a decompiler from scratch and not have to deal with all all of
+the complexity here", think again. What is likely to happen is that
+you'll get at best a 90% solution working for a single Python release
+that will be obsolete in about a year, and more obsolete each
+subsequent year.  Writing a decompiler for Python gets harder as it
+Python progresses, so writing one for Python 3.7 isn't as easy as it
+was for Python 2.2.  That said, if you still feel you want to write a
+single version decompiler, look at the test cases in this project and
+talk to me. I may have some ideas.
+
+
+For a little bit of the history of changes to the Earley-algorithm parser,
+see the file [NEW-FEATURES.rst](https://github.com/rocky/python-spark/blob/master/NEW-FEATURES.rst) in the [python-spark github repository](https://github.com/rocky/python-spark).
+
+NB. If you find mistakes, want corrections, or want your name added
+(or removed), please contact me.

HOW-TO-REPORT-A-BUG.md

@@ -0,0 +1,200 @@
+# How to report a Bug
+
+## The difficulty of the problem
+
+This decompiler is a constant work in progress: Python keeps
+changing, and so does its code generation.
+
+There is no Python decompiler yet that I know about that will
+decompile everything. Overall, I think this one probably does the best
+job of *any* Python decompiler that handles such a wide range of
+versions.
+
+But at any given time, there are a number of valid Python bytecode
+files that I know of that will cause problems. See, for example, the
+list in
+[`test/stdlib/runtests.sh`](https://github.com/rocky/python-uncompyle6/blob/master/test/stdlib/runtests.sh).
+
+But I understand: you would the bugs _you_ encounter addressed before
+all the other known bugs.
+
+From my standpoint, the good thing about the bugs listed in
+`runtests.sh` is that each test case is small and isolated to a single
+kind of problem. And I'll tend to fix easier, more isolated cases than
+generic "something's wrong" kinds of bugs where I'd have to do a bit
+of work to figure out what's up, if not use some sort of mind reading,
+make some guesses, and perform some experiments to see if the guesses
+are correct. I can't read minds, nor am I into guessing games; I'd
+rather devote the effort spent instead towards fixing bugs that are
+precisely defined.
+
+And it often turns out that by just fixing the well-defined and
+prescribed cases, the ill-defined amorphous cases as well will get
+handled as well.
+
+In sum, you may need to do some work to have the bug you have found
+handled before the hundreds of other bugs, and things I could be
+doing.
+
+No one is getting paid to work to work on this project, let alone the
+bugs you may have an interest in. If you require decompiling bytecode
+immediately, consider using a decompilation service, listed further
+down in this document.
+
+## Is it really a bug?
+
+
+### Do you have valid bytecode?
+
+As mentioned in README.rst, this project doesn't handle obfuscated
+code. See README.rst for suggestions for how to remove some kinds of
+obfuscation.
+
+Checking if bytecode is valid is pretty simple: disassemble the code.
+Python comes with a disassembly module called `dis`. A prerequisite
+module for this package, `xdis` has a cross-python version
+disassembler called `pydisasm`.
+
+### Semantic equivalence vs. exact source code
+
+Almost all versions of Python can perform some sort of code
+improvement that can't be undone. In earlier versions of Python it is
+rare; in later Python versions, it is more common.
+
+If the code emitted is semantically equivalent, then this isn't a bug.
+
+
+For example the code might be
+
+```
+if a:
+  if b:
+     x = 1
+```
+
+and we might produce:
+
+```
+if a and b:
+  x = 1
+```
+
+These are equivalent. Sometimes
+
+```
+else:
+   if ...
+
+```
+
+may come out as `elif`.
+
+
+As mentioned in the README, It is possible that Python changes what
+you write to be more efficient. For example, for:
+
+
+```
+if True:
+  x = 5
+```
+
+Python will generate code like:
+
+```
+x = 5
+```
+
+So just because the text isn't the same, does not
+necessarily mean there's a bug.
+
+## What to send (minimum requirements)
+
+The basic requirement is pretty simple:
+
+* Python bytecode
+* Python source text
+
+Please don't put files on download services that one has to register
+for or can't get to by issuing a simple `curl` or `wget`. If you can't
+attach it to the issue, or create a github gist, then the code you are
+sending is too large.
+
+Also try to narrow the bug. See below.
+
+## What to send (additional helpful information)
+
+Some kind folks also give the invocation they used and the output
+which usually includes an error message produced. This is
+helpful. From this, I can figure out what OS you are running this on
+and what version of *uncomplye6* was used. Therefore, if you _don't_
+provide the input command and the output from that, please give:
+
+* _uncompyle6_ version used
+* OS that you used this on
+* Python interpreter version used
+
+
+### But I don't *have* the source code!
+
+Sure, I get it. No problem. There is Python assembly code on parse
+errors, so simply by hand decompile that. To get a full disassembly,
+use `pydisasm` from the [xdis](https://pypi.python.org/pypi/xdis)
+package. Opcodes are described in the documentation for
+the [dis](https://docs.python.org/3.6/library/dis.html) module.
+
+### But I don't *have* the source code and am incapable of figuring how how to do a hand disassembly!
+
+Well, you could learn. No one is born into this world knowing how to
+disassemble Python bytecode. And as Richard Feynman once said, "What
+one fool can learn, so can another."
+
+If this is too difficult, or too time consuming, or not of interest to
+you, then perhaps what require is a decompilation service. [Crazy
+Compilers](http://www.crazy-compilers.com/decompyle/) offers a
+byte-code decompiler service for versions of Python up to 2.6. (If
+there are others around let me know and I'll list them here.)
+
+## Narrowing the problem
+
+I don't need or want the entire source code base for the file(s) or
+module(s) can't be decompiled. I just need those file(s) or module(s).
+If there are problems in several files, file a bug report for each
+file.
+
+Python modules can get quite large, and usually decompilation problems
+occur in a single function or maybe the main-line code but not any of
+the functions or classes. So please chop down the source code by
+removing those parts that do to decompile properly.
+
+By doing this, you'll probably have a better sense of what exactly is
+the problem. Perhaps you can find the boundary of what decompiles, and
+what doesn't. That is useful. Or maybe the same file will decompile
+properly on a neighboring version of Python. That is helpful too.
+
+In sum, the more you can isolate or narrow the problem, the more
+likley the problem will be fixed and fixed sooner.
+
+## Confidentiality of Bug Reports
+
+When you report a bug, you are giving up confidentiality to the source
+code and the byte code. However, I would imagine that if you have
+narrowed the problem sufficiently, confidentiality of the little that
+remains would not be an issue.
+
+However feel free to remove any commments, and modify variable names
+or constants in the source code.
+
+## Ethics
+
+I do not condone using this program for unethical or illegal purposes.
+More detestful, at least to me, is asking for help to assist you in
+something that might not legitimate.
+
+Don't use the issue tracker for such solicitations. To try to stave
+off illegitimate behavior, you should note that the issue tracker, the
+code, and bugs mentioned in that are in the open: there is no
+confidentiality. You may be asked about the authorship or claimed
+ownership of the bytecode. If I think something is not quite right, I
+may label the issue questionable which may make the it easier those
+who are looking for illegal activity.

MANIFEST.in

@@ -1,6 +1,7 @@
 include README.rst
 include ChangeLog
 include HISTORY.md
+include HOW-TO-REPORT-A-BUG.md
 include LICENSE
 include Makefile
 include requirements.txt

Makefile

@@ -11,7 +11,7 @@ RM      ?= rm
 LINT    = flake8
 
 #EXTRA_DIST=ipython/ipy_trepan.py trepan
-PHONY=all check clean pytest check-long dist distclean lint flake8 test rmChangeLog clean_pyc
+PHONY=all check clean distcheck pytest check-long dist distclean lint flake8 test rmChangeLog clean_pyc
 
 TEST_TYPES=check-long check-short check-2.7 check-3.4
 
@@ -36,13 +36,15 @@ check-2.7 check-3.3 check-3.4: pytest
 check-3.0 check-3.1 check-3.2 check-3.5 check-3.6:
 	$(MAKE) -C test $@
 
+check-3.7: pytest
+
 #:Tests for Python 2.6 (doesn't have pytest)
 check-2.4 check-2.5 check-2.6:
 	$(MAKE) -C test $@
 
-#:PyPy 2.6.1 or PyPy 5.0.1
+#:PyPy 2.6.1 PyPy 5.0.1, or PyPy 5.8.0-beta0
 # Skip for now
-2.6 5.0 5.3:
+2.6 5.0 5.3 5.6 5.8:
 
 #:PyPy pypy3-2.4.0 Python 3:
 pypy-3.2 2.4:
@@ -58,9 +60,13 @@ clean: clean_pyc
 	(cd test && $(MAKE) clean)
 
 #: Create source (tarball) and wheel distribution
-dist:
+dist: distcheck
 	$(PYTHON) ./setup.py sdist bdist_wheel
 
+# perform some checks on the package via setup.py
+distcheck:
+	$(PYTHON) ./setup.py check
+
 #: Remove .pyc files
 clean_pyc:
 	( cd uncompyle6 && $(RM) -f *.pyc */*.pyc )
@@ -87,7 +93,7 @@ bdist_egg:
 
 
 #: Create binary wheel distribution
-bdist_wheel:
+wheel:
 	$(PYTHON) ./setup.py bdist_wheel
 
 

NEWS

@@ -1,3 +1,227 @@
+uncompyle6 2.15.0 2018-02-05 pycon2018.co
+
+- Bug fixes
+- Code fragment improvements
+- Code cleanups
+- Expand testing
+
+uncompyle6 2.15.1 2018-01-27
+
+- Add --linemap option to give line correspondences
+  between original source lines and reconstructed line sources.
+  It is far from perfect, but it is a start
+- Add a new class of tests: tests which when decompiled check themselves
+- Split off Python version semantic action customizations into its own file
+- Fix 2.7 bug in ifelse loop statement
+- Handle 3.6+ EXTENDED_ARGs for POP_JUMP_IF... instructions
+- Correct 3.6+ calls with kwargs
+- Describe the difficulty of 3.6 in README
+
+uncompyle6 2.14.3 2018-01-19
+
+- Fix bug in 3.5+ await stmt
+- Better version to magic handling; handle 3.5.2 .. 3.5.4 versions
+- Improve/correct test_pyenvlib.py status messages
+- Fix some 2.7 and 2.6 parser bugs
+- Fix whilelse parsing bugs
+- Correct 2.5- decorator parsing
+- grammar for decorators matches AST a little more
+- better tests in setup.py for running the right version of Python
+- Fix 2.6- parsing of "for .. try/else" ...  with "continue"  inside
+
+uncompyle6 2.14.2 2018-01-09 Samish
+
+Decompilation bug fixes, mostly 3.6 and pre 2.7
+
+- 3.6 FUNCTION_EX (somewhat)
+- 3.6 FUNCTION_EX_KW fixes
+- 3.6 MAKE_FUNCTION fixes
+- correct 3.5 CALL_FUNCTION_VAR
+- stronger 3.x "while 1" testing
+- Fix bug in if's with "pass" bodies. Fixes #104
+- try/else and try/finally fixes on 2.6-
+- limit pypy customization to pypy
+- Add addr fields in COME_FROMS
+- Allow use of full instructions in parser reduction routines
+- Reduce grammar in Pythion 3 by specialization more to specific
+  Python versions
+- Match Python AST names more closely when possible
+
+uncompyle6 2.14.1 2017-12-10 Dr. Gecko
+
+- Many decompilation bugfixes
+- Grammar rule reduction and version isolation
+- Match higher-level nonterminal names more closely
+  with Python AST
+- Start automated Python stdlib testing - full round trip
+
+uncompyle6 2.14.0 2017-11-26 johnnybamazing
+
+- Start to isolate grammar rules between versions
+  and remove used grammar rules
+- Fix a number of bytecode decompile problems
+  (many more remain)
+- Add stdlib/runtests.sh for even more rigourous testing
+
+uncompyle6 2.13.3 2017-11-13
+
+Overall: better 3.6 decompiling and some much needed code refactoring and cleanup
+
+
+- Start noting names in for template-action names; these are
+  used to check/assert we have the right node type
+- Simplify <import_from> rule
+- Pypy 5.80-beta testing tolerance
+- Start to clean up instruction mangling phase by using 3.6-style instructions
+  rather trying to parse the bytecode array. This largely been done in for versions 3.x;
+  3.0 custom mangling code has been reduced;
+  some 2.x conversion has been done, but more is desired. This make it possible to...
+- Handle EXTENDED_ARGS better. While relevant to all Python versions it is most noticeable in
+  version 3.6+ where in switching to wordcodes the size of operands has been reduced from 2**16
+  to 2**8. JUMP instruction then often need EXTENDED_ARGS.
+- Refactor find_jump_targets() with via working of of instructions rather the bytecode array.
+- use --weak-verify more and additional fuzzing on verify()
+- fragment parser now ignores errors in nested function definitions; an parameter was
+  added to assist here. Ignoring errors may be okay because the fragment parser often just needs,
+  well, *fragments*.
+- Distinguish RETURN_VALUE from RETURN_END_IF in exception bodies better in 3.6
+- bug in 3.x language changes: import queue va import Queue
+- reinstate some bytecode tests since decompiling has gotten better
+- Revise how to report a bug
+
+uncompyle6 2.13.2 2017-10-12
+
+- Re-release using a more automated approach
+
+uncompyle6 2.13.1 2017-10-11
+
+- Re-release because Python 2.4 source uploaded rather than 2.6-3.6
+
+uncompyle6 2.13.0 2017-10-10
+
+- Fixes in deparsing lambda expressions
+- Improve table-semantics descriptions
+- Document hacky customize arg count better (until we can remove it)
+- Update to use xdis 3.7.0 or greater
+
+uncompyle6 2.12.0 2017-09-26
+
+- Use xdis 3.6.0 or greater now
+- Small semantic table cleanups
+- Python 3.4's terms a little names better
+- Slightly more Python 3.7, but still failing a lot
+- Cross Python 2/3 compatibility with annotation arguments
+
+uncompyle6 2.11.5 2017-08-31
+
+- Skeletal support for Python 3.7
+
+uncompyle6 2.11.4 2017-08-15
+
+* scanner and parser now allow 3-part version string lookups,
+  e.g. 2.7.1 We allow a float here, but if passed a string like '2.7'. or
+* unpin 3.5.1. xdis 3.5.4 has been releasd and fixes the problems we had. Use that.
+* some routnes here moved to xdis. Use the xdis version
+* README.rst: Link typo Name is trepan2 now not trepan
+* xdis-forced change adjust for COMPARE_OP "is-not" in
+  semanatic routines. We need "is not".
+* Some PyPy tolerance in validate testing.
+* Some pyston tolerance
+
+uncompyle6 2.11.3 2017-08-09
+
+Very minor changes
+
+- RsT doc fixes and updates
+- use newer xdis, but not too new; 3.5.2 breaks uncompyle6
+- use xdis opcode sets
+- xdis "exception match" is now "exception-match"
+
+uncompyle6 2.11.2 2017-07-09
+
+- Start supporting Pypy 3.5 (5.7.1-beta)
+- use xdis 3.5.0's opcode sets and require xdis 3.5.0
+- Correct some Python 2.4-2.6 loop detection
+- guard against badly formatted bytecode
+
+uncompyle6 2.11.1 2017-06-25
+
+- Python 3.x annotation and function signature fixes
+- Bump xdis version
+- Small pysource bug fixes
+
+uncompyle6 2.11.0 2017-06-18 Fleetwood
+- Major improvements in fragment tracking
+  * Add nonterminal node in extractInfo
+  * tag more offsets in expressions
+  * tag array subscripts
+  * set YIELD value offset in a <yield> expr
+  * fix a long-standing bug in not adjusting final AST when melding other deparse ASTs
+- Fixes yet again for make_function node handling; document what's up here
+- Fix bug in snowflake Python 3.5 *args kwargs
+
+uncompyle6 2.10.1 2017-06-3 Marylin Frankel
+
+- fix some fragments parsing bugs
+  - was returning the wrong type sometimes in deparse_code_around_offset()
+  - capture function name in offsets
+  - track changes to ifelstrmtr node from pysource into fragments
+
+uncompyle6 2.10.0 2017-05-30 Elaine Gordon
+
+- Add fuzzy offset deparse look up
+- 3.6 bug fixes
+   - fix EXTENDED_ARGS handling (and in 2.6 and others)
+   - semantic routine make_function fragments.py
+   - MAKE_FUNCTION handling
+   - CALL_FUNCTION_EX handling
+   - async property on defs
+   - support for CALL_FUNCTION_KW (moagstar)
+- 3.5+ UNMAP_PACK and BUILD_UNMAP_PACK handling
+- 3.5 FUNCTION_VAR bug
+- 3.x pass statement insdie while True
+- Improve 3.2 decompilation
+- Fixed -o argument processing (grkov90)
+- Reduce scope of LOAD_ASSERT as expr to 3.4+
+- "await" statement fixes
+- 2.3, 2.4 "if 1 .." fixes
+- 3.x annotation fixes
+
+uncompyle6 2.9.11 2017-04-06
+
+- Better support for Python 3.5+ BUILD_MAP_UNPACK
+- Start 3.6 CALL_FUNCTION_EX support
+- Many decompilation bug fixes. (Many more remain). See ChangeLog
+
+uncompyle6 2.9.10 2017-02-25
+
+- Python grammar rule fixes
+- Add ability to get grammar coverage on runs
+- Handle Python 3.6 opcode BUILD_CONST_KEYMAP
+
+uncompyle6 2.9.9 2016-12-16
+
+- Remaining Python 3.5 ops handled
+  (this also means more Python 3.6 ops are handled)
+- Python 3.5 and 3.6 async and await handled
+- Python 3.0 decompilation improved
+- Python 3 annotations fixed
+- Better control-flow detection
+- Code cleanups and misc bug fixes
+
+uncompyle6 2.9.8 2016-12-16
+
+- Better control-flow detection
+- pseudo instruction THEN in 2.x
+  to disambiguate if from and
+- fix bug in --verify option
+- DRY (a little) control-flow detection
+- fix syntax in tuples with one element
+- if AST rule inheritence in Python 2.5
+- NAME_MODULE removal for Python <= 2.4
+- verifycall fixes for Python <= 2.4
+- more Python lint
+
 uncompyle6 2.9.7 2016-12-16
 
 - Start to handle 3.5/3.6 build_map_unpack_with_call
@@ -36,7 +260,7 @@ uncompyle6 2.9.6 2016-11-20
 uncompyle6 2.9.5 2016-11-13
 
 - Fix Python 3 bugs:
-  * improprer while 1 else
+  * improper while 1 else
   * docstring indent
   * 3.3 default values in lambda expressions
   * start 3.0 decompilation (needs newer xdis)
@@ -46,12 +270,12 @@ uncompyle6 2.9.5 2016-11-13
 uncompyle6 2.9.4 2016-11-02
 
 - Handle Python 3.x function annotations
-- track def keywoard-parameter line-splitting in source code better
+- track def keyword-parameter line-splitting in source code better
 - bump min xdis version to mask previous xdis bug
 
 uncompyle6 2.9.3 2016-10-26
 
-Release forced by incompatiblity change in xdis 3.2.0.
+Release forced by incompatibility change in xdis 3.2.0.
 
 - Python 3.1 bugs:
   * handle "with ... as"
@@ -83,7 +307,7 @@ uncompyle6 2.9.0 2016-10-09
   this Forces change in requirements.txt and _pkg_info_.py
 - Start Python 1.5 decompiling; another round of work is needed to
   remove bugs
-- Simpify python 2.1 grammar
+- Simplify python 2.1 grammar
 - Fix bug with -t ...  Wasn't showing source text when -t option was given
 - Fix 2.1-2.6 bug in list comprehension
 
@@ -106,7 +330,7 @@ control-flow structure detection is done.
     . 3.0 .. 3.2 *args processing
     . 3.0 .. 3.2 call name and kwargs bug
     . 3.0 .. getting parameter of *
-    . 3.0 .. handling varible number of args
+    . 3.0 .. handling variable number of args
     . 3.0 .. "if" structure bugs
   * 3.5+ if/else bugs
   * 2.2-2.6 bugs
@@ -157,7 +381,7 @@ uncompyle6 2.7.1 2016-07-26
 
 uncompyle6 2.7.0 2016-07-15
 
-- Many Syntax and verifification bugs removed
+- Many Syntax and verification bugs removed
   tested on standard libraries from 2.3.7 to 3.5.1
   and they all decompile and verify fine.
   I'm sure there are more bugs though.
@@ -184,9 +408,9 @@ uncompyle6 2.6.0 2016-07-07
 - Better <2.6 vs. 2.7 grammar separation
 - Fix some 2.7 deparsing bugs
 - Fix bug in installing uncompyle6 script
-- Doc improvments
+- Doc improvements
 
-uncompyle6 2.5.0 2016-06-22  Summer Solstace
+uncompyle6 2.5.0 2016-06-22  Summer Solstice
 
 - Much better Python 3.2-3.5 coverage.
   3.4.6 is probably the best;3.2 and 3.5 are weaker
@@ -198,7 +422,7 @@ uncompyle6 2.5.0 2016-06-22  Summer Solstace
 uncompyle6 2.4.0 2016-05-18 (in memory of Lewis Bernstein)
 
 - Many Python 3 bugs fixed:
-  * Python 3.2 to 3.5 libaries largely
+  * Python 3.2 to 3.5 libraries largely
     uncompyle and most verify
 - pydisassembler:
   * disassembles all code objects in a file
@@ -256,7 +480,7 @@ uncompyle6 2.2.0 2016-04-30
 
 uncompyle6 2.2.0 2016-04-02
 
-- Support single-mode (in addtion to exec-mode) compilation
+- Support single-mode (in addition to exec-mode) compilation
 - Start to DRY Python 2 and Python 3 grammars
 - Fix bug in if else ternary  construct
 - Fix bug in uncomplye6 -d and -r options (via lelicopter)

README.rst

@@ -1,52 +1,73 @@
-|buildstatus| |Supported Python Versions|
+|buildstatus|
 
 uncompyle6
 ==========
 
-A native Python cross-version Decompiler and Fragment Decompiler.
-Follows in the tradition of decompyle, uncompyle, and uncompyle2.
+A native Python cross-version decompiler and fragment decompiler.
+The successor to decompyle, uncompyle, and uncompyle2.
 
 
 Introduction
 ------------
 
 *uncompyle6* translates Python bytecode back into equivalent Python
-source code. It accepts bytecodes from Python version 2.1 to 3.6 or
-so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
+source code. It accepts bytecodes from Python version 1.5, and 2.1 to
+3.7 or so, including PyPy bytecode and Dropbox's Python 2.5 bytecode.
 
 Why this?
 ---------
 
-There were a number of decompyle, uncompile, uncompyle2, uncompyle3
-forks around. All of them came basically from the same code base, and
-almost all of them no were no longer actively maintained. Only one
-handled Python 3, and even there, only 3.2 or 3.3 depending on which
-code is used. This code pulls these together and moves forward. It
-also addresses a number of open issues in the previous forks.
+Ok, I'll say it: this software is amazing. It is more than your
+normal hacky decompiler. Using compiler_ technology, the program
+creates a parse tree of the program from the instructions; nodes at
+the upper levels that look a little like what might come from a Python
+AST. So we can really classify and understand what's going on in
+sections of Python bytecode.
 
-What makes this different from other CPython bytecode decompilers?: its
-ability to deparse just fragments and give source-code information
-around a given bytecode offset.
+Building on this, another thing that makes this different from other
+CPython bytecode decompilers is the ability to deparse just
+*fragments* of source code and give source-code information around a
+given bytecode offset.
 
-I use this to deparse fragments of code inside my trepan_
-debuggers_. For that, I need to record text fragments for all
-bytecode offsets (of interest). This purpose although largely
-compatible with the original intention is yet a little bit different.
+I use the tree fragments to deparse fragments of code inside my
+trepan_ debuggers_. For that, bytecode offsets are recorded and
+associated with fragments of the source code. This purpose, although
+compatible with the original intention, is yet a little bit different.
 See this_ for more information.
 
-The idea of Python fragment deparsing given an instruction offset can
-be used in showing stack traces or any program that wants to show a
-location in more detail than just a line number.  It can be also used
-when source-code information does not exist and there is just bytecode
-information.
+Python fragment deparsing given an instruction offset is useful in
+showing stack traces and can be encorporated into any program that
+wants to show a location in more detail than just a line number at
+runtime.  This code can be also used when source-code information does
+not exist and there is just bytecode. Again, my debuggers make use of
+this.
+
+There were (and still are) a number of decompyle, uncompyle,
+uncompyle2, uncompyle3 forks around. Almost all of them come basically
+from the same code base, and (almost?) all of them are no longer
+actively maintained. One was really good at decompiling Python 1.5-2.3
+or so, another really good at Python 2.7, but that only. Another
+handles Python 3.2 only; another patched that and handled only 3.3.
+You get the idea. This code pulls all of these forks together and
+*moves forward*. There is some serious refactoring and cleanup in this
+code base over those old forks.
+
+This project has the most complete support for Python 3.3 and above
+and the best all-around Python support.
+
+We are serious about testing, and use automated processes to find
+bugs. In the issue trackers for other decompilers, you will find a
+number of bugs we've found along the way. Very few to none of them are
+fixed in the other decompilers.
 
 Requirements
 ------------
 
-This project requires Python 2.6 or later, PyPy 3-2.4, or PyPy-5.0.1. 
-Python versions 2.3-2.7 are supported in the python-2.4 branch.
-The bytecode files it can read has been tested on Python bytecodes from
-versions 2.1-2.7, and 3.2-3.6 and the above-mentioned PyPy versions.
+The code here can be run on Python versions 2.6 or later, PyPy 3-2.4,
+or PyPy-5.0.1.  Python versions 2.4-2.7 are supported in the
+python-2.4 branch.  The bytecode files it can read have been tested on
+Python bytecodes from versions 1.5, 2.1-2.7, and 3.0-3.6 and the
+above-mentioned PyPy versions.
 
 Installation
 ------------
@@ -55,11 +76,9 @@ This uses setup.py, so it follows the standard Python routine:
 
 ::
 
-    pip install -r requirements.txt
-    pip install -r requirements-dev.txt
+    pip install -e .  # set up to run from source tree
+                      # Or if you want to install instead
     python setup.py install # may need sudo
-    # or if you have pyenv:
-    python setup.py develop
 
 A GNU makefile is also provided so :code:`make install` (possibly as root or
 sudo) will do the steps above.
@@ -93,19 +112,29 @@ For usage help:
 
    $ uncompyle6 -h
 
+If you want strong verification of the correctness of the
+decompilation process, add the `--verify` option. But there are
+situations where this will indicate a failure, although the generated
+program is semantically equivalent. Using option `--weak-verify` will
+tell you if there is something definitely wrong. Generally, large
+swaths of code are decompiled correctly, if not the entire program.
+
+You can also cross compare the results with pycdc_ . Since they work
+differently, bugs here often aren't in that, and vice versa.
+
 
 Known Bugs/Restrictions
 -----------------------
 
 The biggest known and possibly fixable (but hard) problem has to do
-with handling control flow. All of the Python decompilers I have looked
-at have the same problem. In some cases we can detect an erroneous
-decompilation and report that.
+with handling control flow. (Python has probably the most diverse and
+screwy set of compound statements I've ever seen; there
+are "else" clauses on loops and try blocks that I suspect many
+programmers don't know about.)
 
-About 90% of the decompilation of Python standard library packages in
-Python 2.7.12 verifies correctly. Over 99% of Python 2.7 and 3.3-3.5
-"weakly" verify. Python 2.6 drops down to 96% weakly verifying.
-Other versions drop off in quality too.
+All of the Python decompilers that I have looked at have problems
+decompiling Python's control flow. In some cases we can detect an
+erroneous decompilation and report that.
 
 *Verification* is the process of decompiling bytecode, compiling with
 a Python for that bytecode version, and then comparing the bytecode
@@ -123,15 +152,33 @@ program by running the Python interpreter. Because the Python language
 has changed so much, for best results you should use the same Python
 Version in checking as used in the bytecode.
 
-Later distributions average about 200 files. There is some work to do
-on the lower end Python versions which is more difficult for us to
-handle since we don't have a Python interpreter for versions 1.5, 1.6,
-and 2.0.
+Finally, we have automated running the standard Python tests after
+first compiling and decompiling the test program. Results here are a
+bit weak (if not better than most other Python decompilers). But over
+time this will probably get better.
 
-Python 3.0 support is weak; Python 3.5 largely works, but still has
-some bugs in it.  Python 3.6 changes things drastically by using word
-codes rather than byte codes. That has been addressed, but then it also
-changes function call opcodes and its semantics.
+Python support is strongest in Python 2 for 2.7 and drops off as you
+get further away from that. Support is also probably pretty good for
+python 2.3-2.4 since a lot of the goodness of early the version of the
+decompiler from that era has been preserved (and Python compilation in
+that era was minimal)
+
+There is some work to do on the lower end Python versions which is
+more difficult for us to handle since we don't have a Python
+interpreter for versions 1.5, 1.6, and 2.0.
+
+In the Python 3 series, Python support is is strongest around 3.4 or
+3.3 and drops off as you move further away from those versions. Python
+3.6 changes things drastically by using word codes rather than byte
+codes. As a result, the jump offset field in a jump instruction
+argument has been reduced. This makes the `EXTENDED_ARG` instructions
+are now more prevalent in jump instruction; previously they had been
+rare.  Perhaps to compensate for the additional `EXTENDED_ARG`
+instructions, additional jump optimization has been added. So in sum
+handling control flow by ad hoc means as is currently done is worse.
+
+Also, between Python 3.5, 3.6 and 3.7 there have been major changes to the
+`MAKE_FUNCTION` and `CALL_FUNCTION` instructions.
 
 Currently not all Python magic numbers are supported. Specifically in
 some versions of Python, notably Python 3.6, the magic number has
@@ -143,7 +190,12 @@ handled.
 
 We also don't handle PJOrion_ obfuscated code. For that try: PJOrion
 Deobfuscator_ to unscramble the bytecode to get valid bytecode before
-trying this tool.
+trying this tool. This program can't decompile Microsoft Windows EXE
+files created by Py2EXE_, although we can probably decompile the code
+after you extract the bytecode properly. For situations like this, you
+might want to consider a decompilation service like `Crazy Compilers
+<http://www.crazy-compilers.com/decompyle/>`_.  Handling
+pathologically long lists of expressions or statements is slow.
 
 
 There is lots to do, so please dig in and help.
@@ -151,13 +203,18 @@ There is lots to do, so please dig in and help.
 See Also
 --------
 
-* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++
-* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique what is used here.
-* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Include some fixes like supporting function annotations
+* https://github.com/zrax/pycdc : supports all versions of Python and is written in C++. Support for later Python 3 versions is a bit lacking though.
+* https://code.google.com/archive/p/unpyc3/ : supports Python 3.2 only. The above projects use a different decompiling technique than what is used here.
+* https://github.com/figment/unpyc3/ : fork of above, but supports Python 3.3 only. Includes some fixes like supporting function annotations
 * The HISTORY_ file.
+* `How to report a bug <https://github.com/rocky/python-uncompyle6/blob/master/HOW-TO-REPORT-A-BUG.md>`_
+* https://github.com/rocky/python-xdis : Cross Python version disassembler
+* https://github.com/rocky/python-xasm : Cross Python version assembler
+* https://github.com/rocky/python-uncompyle6/wiki : Wiki Documents which describe the code and aspects of it in more detail
 
-.. |downloads| image:: https://img.shields.io/pypi/dd/uncompyle6.svg
-.. _trepan: https://pypi.python.org/pypi/trepan
+
+.. _trepan: https://pypi.python.org/pypi/trepan2
+.. _compiler: https://pypi.python.org/pypi/spark_parser
 .. _HISTORY: https://github.com/rocky/python-uncompyle6/blob/master/HISTORY.md
 .. _debuggers: https://pypi.python.org/pypi/trepan3k
 .. _remake: https://bashdb.sf.net/remake
@@ -165,7 +222,6 @@ See Also
 .. _this: https://github.com/rocky/python-uncompyle6/wiki/Deparsing-technology-and-its-use-in-exact-location-reporting
 .. |buildstatus| image:: https://travis-ci.org/rocky/python-uncompyle6.svg
 		 :target: https://travis-ci.org/rocky/python-uncompyle6
-.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/uncompyle6.svg
-   :target: https://pypi.python.org/pypi/uncompyle6/
 .. _PJOrion: http://www.koreanrandom.com/forum/topic/15280-pjorion-%D1%80%D0%B5%D0%B4%D0%B0%D0%BA%D1%82%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B5-%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%B4%D0%B5%D0%BA%D0%BE%D0%BC%D0%BF%D0%B8%D0%BB%D1%8F%D1%86%D0%B8%D1%8F-%D0%BE%D0%B1%D1%84
 .. _Deobfuscator: https://github.com/extremecoders-re/PjOrion-Deobfuscator
+.. _Py2EXE: https://en.wikipedia.org/wiki/Py2exe

__pkginfo__.py

@@ -9,7 +9,7 @@
 
 # Things that change more often go here.
 copyright   = """
-Copyright (C) 2015, 2016 Rocky Bernstein <rb@dustyfeet.com>.
+Copyright (C) 2015-2018 Rocky Bernstein <rb@dustyfeet.com>.
 """
 
 classifiers =  ['Development Status :: 5 - Production/Stable',
@@ -26,6 +26,7 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
                 'Programming Language :: Python :: 3.4',
                 'Programming Language :: Python :: 3.5',
                 'Programming Language :: Python :: 3.6',
+                'Programming Language :: Python :: 3.7',
                 'Topic :: Software Development :: Debuggers',
                 'Topic :: Software Development :: Libraries :: Python Modules',
                 ]
@@ -33,19 +34,19 @@ classifiers =  ['Development Status :: 5 - Production/Stable',
 # The rest in alphabetic order
 author             = "Rocky Bernstein, Hartmut Goebel, John Aycock, and others"
 author_email       = "rb@dustyfeet.com"
-entry_points={
+entry_points       = {
     'console_scripts': [
         'uncompyle6=uncompyle6.bin.uncompile:main_bin',
         'pydisassemble=uncompyle6.bin.pydisassemble:main',
     ]}
 ftp_url            = None
-install_requires   = ['spark-parser >= 1.5.1, < 1.6.0',
-                      'xdis >= 3.2.4, < 3.3.0']
+install_requires   = ['spark-parser >= 1.8.5, < 1.9.0',
+                      'xdis >= 3.6.9, < 3.7.0']
 license            = 'MIT'
 mailing_list       = 'python-debugger@googlegroups.com'
 modname            = 'uncompyle6'
 py_modules         = None
-short_desc         = 'Python cross-version byte-code deparser'
+short_desc         = 'Python cross-version byte-code decompiler'
 web                = 'https://github.com/rocky/python-uncompyle6/'
 
 # tracebacks in zip files are funky and not debuggable

admin-tools/README.md

@@ -0,0 +1,11 @@
+Making a release is a somewhat tedious process so I've automated it a little
+
+
+Here are tools that I, rocky, use to check and build a distribution.
+
+They are customized to my environment:
+- I use pyenv to various Python versions installed
+- I have git repos for xdis, and spark parser at the same level as uncompyle6
+
+There may be other rocky-specific things that need customization.
+how-to-make-a-release.txt has overall how I make a release

admin-tools/check-newer-versions.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+function finish {
+  cd $owd
+}
+
+# FIXME put some of the below in a common routine
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-newer-versions ; then
+    exit $?
+fi
+if ! source ./setup-master.sh ; then
+    exit $?
+fi
+cd ..
+for version in $PYVERSIONS; do
+    if ! pyenv local $version ; then
+	exit $?
+    fi
+    make clean && pip install -e .
+    if ! make check; then
+	exit $?
+    fi
+done

admin-tools/check-older-versions.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+function finish {
+  cd $owd
+}
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-older-versions ; then
+    exit $?
+fi
+if ! source ./setup-python-2.4.sh ; then
+    exit $?
+fi
+
+cd ..
+for version in $PYVERSIONS; do
+    if ! pyenv local $version ; then
+	exit $?
+    fi
+    make clean && python setup.py develop
+    if ! make check ; then
+	exit $?
+    fi
+done

admin-tools/how-to-make-a-release.md

@@ -0,0 +1,81 @@
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Get latest sources:](#get-latest-sources)
+- [Change version in uncompyle6/version.py](#change-version-in-uncompyle6versionpy)
+- [Update ChangeLog:](#update-changelog)
+- [Update NEWS from ChangeLog:](#update-news-from-changelog)
+- [Make sure pyenv is running and check newer versions](#make-sure-pyenv-is-running-and-check-newer-versions)
+- [Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.](#switch-to-python-24-sync-that-up-and-build-that-first-since-it-creates-a-tarball-which-we-dont-want)
+- [Update NEWS from master branch](#update-news-from-master-branch)
+- [Check against all versions](#check-against-all-versions)
+- [Make packages and tag](#make-packages-and-tag)
+- [Upload single package and look at Rst Formating](#upload-single-package-and-look-at-rst-formating)
+- [Upload rest of versions](#upload-rest-of-versions)
+- [Push tags:](#push-tags)
+
+<!-- markdown-toc end -->
+# Get latest sources:
+
+    git pull
+
+# Change version in uncompyle6/version.py:
+
+    $ emacs uncompyle6/version.py
+    $ source uncompyle6/version.py
+    $ echo $VERSION
+    $ git commit -m"Get ready for release $VERSION" .
+
+# Update ChangeLog:
+
+    $ make ChangeLog
+
+#  Update NEWS from ChangeLog:
+
+    $ emacs NEWS
+    $ make check
+    $ git commit --amend .
+    $ git push   # get CI testing going early
+
+# Make sure pyenv is running and check newer versions
+
+    $ pyenv local && source admin-tools/check-newer-versions.sh
+
+# Switch to python-2.4, sync that up and build that first since it creates a tarball which we don't want.
+
+    $ source admin-tools/setup-python-2.4.sh
+    $ git merge master
+
+# Check against older versions
+
+    $ source admin-tools/check-older-versions.sh
+
+# Make packages and tag
+
+    $ . ./admin-tools/make-dist-older.sh
+    $ git tag release-python-2.4-$VERSION
+
+    $ . ./admin-tools/make-dist-newer.sh
+    $ git tag release-$VERSION
+
+# Upload single package and look at Rst Formating
+
+    $ twine upload dist/uncompyle6-${VERSION}-py3.3.egg
+
+# Upload rest of versions
+
+    $ twine upload dist/uncompyle6-${VERSION}*
+
+# Push tags:
+
+    $ git push --tags
+
+# Check on a VM
+
+    $ cd /virtual/vagrant/virtual/vagrant/ubuntu-zesty
+	$ vagrant up
+	$ vagrant ssh
+	$ pyenv local 3.5.2
+	$ pip install --upgrade uncompyle6
+	$ exit
+	$ vagrant halt

admin-tools/how-to-make-a-release.txt

@@ -0,0 +1,46 @@
+git pull
+
+Change version in uncompyle6/version.py
+source uncompyle6/version.py
+echo $VERSION
+git commit -m"Get ready for release $VERSION" .
+
+Update ChangeLog:
+  make ChangeLog
+
+Update NEWS from ChangeLog
+make check
+
+git commit --amend .
+
+git push
+
+Make sure pyenv is running
+# Pyenv
+
+source admin-tools/check-newer-versions.sh
+
+
+# Switch to python-2.4 and build that first...
+source admin-tools/setup-python-2.4
+
+rm ChangeLog
+git merge master
+
+Update NEWS from master branch
+
+git commit -m"Get ready for release $VERSION" .
+
+source admin-tools/check-older-versions.sh
+source admin-tools/check-newer-versions.sh
+
+make-dist-older.sh
+
+git tag release-python-2.4-$VERSION
+
+./make-dist-newer.sh
+
+git tag release-$VERSION
+
+
+twine upload dist/uncompyle6-${VERSION}*

admin-tools/make-dist-newer.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+PACKAGE=uncompyle6
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+
+cd $(dirname ${BASH_SOURCE[0]})
+owd=$(pwd)
+trap finish EXIT
+
+if ! source ./pyenv-newer-versions ; then
+    exit $?
+fi
+if ! source ./setup-master.sh ; then
+    exit $?
+fi
+
+cd ..
+source $PACKAGE/version.py
+echo $VERSION
+
+for pyversion in $PYVERSIONS; do
+    if ! pyenv local $pyversion ; then
+	exit $?
+    fi
+    # pip bdist_egg create too-general wheels. So
+    # we narrow that by moving the generated wheel.
+
+    # Pick out first two number of version, e.g. 3.5.1 -> 35
+    first_two=$(echo $pyversion | cut -d'.' -f 1-2 | sed -e 's/\.//')
+    rm -fr build
+    python setup.py bdist_egg bdist_wheel
+    mv -v dist/${PACKAGE}-$VERSION-{py2.py3,py$first_two}-none-any.whl
+done
+
+python ./setup.py sdist

admin-tools/make-dist-older.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+PACKAGE=uncompyle6
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+owd=$(pwd)
+trap finish EXIT
+
+cd $(dirname ${BASH_SOURCE[0]})
+if ! source ./pyenv-older-versions ; then
+    exit $?
+fi
+if ! source ./setup-python-2.4.sh ; then
+    exit $?
+fi
+
+cd ..
+source $PACKAGE/version.py
+echo $VERSION
+
+for pyversion in $PYVERSIONS; do
+    if ! pyenv local $pyversion ; then
+	exit $?
+    fi
+
+    rm -fr build
+    python setup.py bdist_egg
+done
+
+# Pypi can only have one source tarball.
+# Tarballs can get created from the above setup, so make sure to remove them since we want
+# the tarball from master.
+
+tarball=dist/${PACKAGE}-$VERSION-tar.gz
+if [[ -f $tarball ]]; then
+    rm -v dist/${PACKAGE}-$VERSION-tar.gz
+fi

admin-tools/pyenv-all-versions

@@ -0,0 +1,19 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv versions we have
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+
+olddir=$(pwd)
+mydir=$(dirname ${BASH_SOURCE[0]})
+cd $mydir
+
+all=""
+for file in pyenv-{olde{st,r},newer}-versions ; do
+    . $mydir/$file
+    all="$all $PYVERSIONS"
+done
+
+PYVERSIONS="$all"
+cd $olddir

admin-tools/pyenv-newer-versions

@@ -0,0 +1,8 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the master branch.
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='3.5.3 3.6.3 2.6.9 3.3.6 2.7.14 3.4.2'

admin-tools/pyenv-older-versions

@@ -0,0 +1,9 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be pyenv versions that
+# we can use in the python-2.4 branch.
+
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='2.4.6 2.5.6'

admin-tools/pyenv-oldest-versions

@@ -0,0 +1,9 @@
+# -*- shell-script -*-
+# Sets PYVERSIONS to be all pyenv the oldest versions we have.
+# These are not covered (yet) by uncompyle6, although
+# some programs do work here.
+if [[ $0 == ${BASH_SOURCE[0]} ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+export PYVERSIONS='2.1.3 2.2.3 2.3.7'

admin-tools/setup-master.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+PYTHON_VERSION=3.6.3
+
+# FIXME put some of the below in a common routine
+function finish {
+  cd $owd
+}
+
+export PATH=$HOME/.pyenv/bin/pyenv:$PATH
+owd=$(pwd)
+bs=${BASH_SOURCE[0]}
+if [[ $0 == $bs ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+mydir=$(dirname $bs)
+fulldir=$(readlink -f $mydir)
+cd $fulldir/..
+(cd ../python-spark && git checkout master && pyenv local $PYTHON_VERSION) && git pull && \
+    (cd ../python-xdis && git checkout master && pyenv local $PYTHON_VERSION) && git pull && \
+    git checkout master && pyenv local $PYTHON_VERSION && git pull
+cd $owd

admin-tools/setup-python-2.4.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+PYTHON_VERSION=2.4.6
+
+owd=$(pwd)
+bs=${BASH_SOURCE[0]}
+if [[ $0 == $bs ]] ; then
+    echo "This script should be *sourced* rather than run directly through bash"
+    exit 1
+fi
+mydir=$(dirname $bs)
+fulldir=$(readlink -f $mydir)
+cd $fulldir/..
+(cd ../python-spark && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && git pull && \
+    (cd ../python-xdis && git checkout python-2.4 && pyenv local $PYTHON_VERSION) && git pull && \
+    git checkout python-2.4 &&  pyenv local $PYTHON_VERSION && git pull
+cd $owd

admin-tools/update-sources.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+cd $(dirname ${BASH_SOURCE[0]})/..
+git pull

appveyor.yml

@@ -0,0 +1,78 @@
+environment:
+  global:
+    # SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
+    # /E:ON and /V:ON options are not enabled in the batch script intepreter
+    # See: http://stackoverflow.com/a/13751649/163740
+    CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"
+
+  matrix:
+
+    # Pre-installed Python versions, which Appveyor may upgrade to
+    # a later point release.
+    # See: http://www.appveyor.com/docs/installed-software#python
+
+    # - PYTHON: "C:\\Python27"
+    #   PYTHON_VERSION: "2.7.x"
+    #   PYTHON_ARCH: "32"
+
+    - PYTHON: "C:\\Python27-x64"
+      PYTHON_VERSION: "2.7.x"
+      PYTHON_ARCH: "64"
+
+    # - PYTHON: "C:\\Python26"
+    #   PYTHON_VERSION: "2.6.x"
+    #   PYTHON_ARCH: "32"
+
+    # - PYTHON: "C:\\Python26-x64"
+    #   PYTHON_VERSION: "2.6.x"
+    #   PYTHON_ARCH: "64"
+
+install:
+  # We need wheel installed to build wheels
+  - "%PYTHON%\\python.exe -m pip install wheel"
+
+  # Install Python (from the official .msi of http://python.org) and pip when
+  # not already installed.
+  - ps: if (-not(Test-Path($env:PYTHON))) { & appveyor\install.ps1 }
+
+  # Prepend newly installed Python to the PATH of this build (this cannot be
+  # done from inside the powershell script as it would require to restart
+  # the parent CMD process).
+  - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
+  - "SET HOME=."
+
+  # Check that we have the expected version and architecture for Python
+  - "python --version"
+  - "python -c \"import struct; print(struct.calcsize('P') * 8)\""
+
+  # Upgrade to the latest version of pip to avoid it displaying warnings
+  # about it being out of date.
+  - "pip install --disable-pip-version-check --user --upgrade pip"
+
+  # Install the build dependencies of the project. If some dependencies contain
+  # compiled extensions and are not provided as pre-built wheel packages,
+  # pip will build them from source using the MSVC compiler matching the
+  # target Python version and architecture
+  - "%CMD_IN_ENV% pip install -r requirements.txt"
+
+build_script:
+  # Build the compiled extension
+  - "%CMD_IN_ENV% python setup.py build"
+
+test_script:
+  # Run the project tests
+  - "%CMD_IN_ENV% python test/test_pyenvlib.py --native --weak-verify"
+
+after_test:
+  # If tests are successful, create binary packages for the project.
+  - "%CMD_IN_ENV% python setup.py bdist_wininst"
+  - "%CMD_IN_ENV% python setup.py bdist_msi"
+  - ps: "ls dist"
+
+artifacts:
+  # Archive the generated packages in the ci.appveyor.com build report.
+  - path: dist\*
+
+#on_success:
+#  - TODO: upload the content of dist/*.whl to a public wheelhouse
+#

appveyor/install.ps1

@@ -0,0 +1,229 @@
+# Sample script to install Python and pip under Windows
+# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
+# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+
+$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
+$BASE_URL = "https://www.python.org/ftp/python/"
+$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
+$GET_PIP_PATH = "C:\get-pip.py"
+
+$PYTHON_PRERELEASE_REGEX = @"
+(?x)
+(?<major>\d+)
+\.
+(?<minor>\d+)
+\.
+(?<micro>\d+)
+(?<prerelease>[a-z]{1,2}\d+)
+"@
+
+
+function Download ($filename, $url) {
+    $webclient = New-Object System.Net.WebClient
+
+    $basedir = $pwd.Path + "\"
+    $filepath = $basedir + $filename
+    if (Test-Path $filename) {
+        Write-Host "Reusing" $filepath
+        return $filepath
+    }
+
+    # Download and retry up to 3 times in case of network transient errors.
+    Write-Host "Downloading" $filename "from" $url
+    $retry_attempts = 2
+    for ($i = 0; $i -lt $retry_attempts; $i++) {
+        try {
+            $webclient.DownloadFile($url, $filepath)
+            break
+        }
+        Catch [Exception]{
+            Start-Sleep 1
+        }
+    }
+    if (Test-Path $filepath) {
+        Write-Host "File saved at" $filepath
+    } else {
+        # Retry once to get the error message if any at the last try
+        $webclient.DownloadFile($url, $filepath)
+    }
+    return $filepath
+}
+
+
+function ParsePythonVersion ($python_version) {
+    if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
+        return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
+                $matches.prerelease)
+    }
+    $version_obj = [version]$python_version
+    return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
+}
+
+
+function DownloadPython ($python_version, $platform_suffix) {
+    $major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
+
+    if (($major -le 2 -and $micro -eq 0) `
+        -or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
+        ) {
+        $dir = "$major.$minor"
+        $python_version = "$major.$minor$prerelease"
+    } else {
+        $dir = "$major.$minor.$micro"
+    }
+
+    if ($prerelease) {
+        if (($major -le 2) `
+            -or ($major -eq 3 -and $minor -eq 1) `
+            -or ($major -eq 3 -and $minor -eq 2) `
+            -or ($major -eq 3 -and $minor -eq 3) `
+            ) {
+            $dir = "$dir/prev"
+        }
+    }
+
+    if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
+        $ext = "msi"
+        if ($platform_suffix) {
+            $platform_suffix = ".$platform_suffix"
+        }
+    } else {
+        $ext = "exe"
+        if ($platform_suffix) {
+            $platform_suffix = "-$platform_suffix"
+        }
+    }
+
+    $filename = "python-$python_version$platform_suffix.$ext"
+    $url = "$BASE_URL$dir/$filename"
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallPython ($python_version, $architecture, $python_home) {
+    Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = ""
+    } else {
+        $platform_suffix = "amd64"
+    }
+    $installer_path = DownloadPython $python_version $platform_suffix
+    $installer_ext = [System.IO.Path]::GetExtension($installer_path)
+    Write-Host "Installing $installer_path to $python_home"
+    $install_log = $python_home + ".log"
+    if ($installer_ext -eq '.msi') {
+        InstallPythonMSI $installer_path $python_home $install_log
+    } else {
+        InstallPythonEXE $installer_path $python_home $install_log
+    }
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallPythonEXE ($exepath, $python_home, $install_log) {
+    $install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
+    RunCommand $exepath $install_args
+}
+
+
+function InstallPythonMSI ($msipath, $python_home, $install_log) {
+    $install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
+    $uninstall_args = "/qn /x $msipath"
+    RunCommand "msiexec.exe" $install_args
+    if (-not(Test-Path $python_home)) {
+        Write-Host "Python seems to be installed else-where, reinstalling."
+        RunCommand "msiexec.exe" $uninstall_args
+        RunCommand "msiexec.exe" $install_args
+    }
+}
+
+function RunCommand ($command, $command_args) {
+    Write-Host $command $command_args
+    Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
+}
+
+
+function InstallPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $python_path = $python_home + "\python.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $webclient = New-Object System.Net.WebClient
+        $webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
+        Write-Host "Executing:" $python_path $GET_PIP_PATH
+        & $python_path $GET_PIP_PATH
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+
+function DownloadMiniconda ($python_version, $platform_suffix) {
+    if ($python_version -eq "3.4") {
+        $filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
+    } else {
+        $filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
+    }
+    $url = $MINICONDA_URL + $filename
+    $filepath = Download $filename $url
+    return $filepath
+}
+
+
+function InstallMiniconda ($python_version, $architecture, $python_home) {
+    Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
+    if (Test-Path $python_home) {
+        Write-Host $python_home "already exists, skipping."
+        return $false
+    }
+    if ($architecture -eq "32") {
+        $platform_suffix = "x86"
+    } else {
+        $platform_suffix = "x86_64"
+    }
+    $filepath = DownloadMiniconda $python_version $platform_suffix
+    Write-Host "Installing" $filepath "to" $python_home
+    $install_log = $python_home + ".log"
+    $args = "/S /D=$python_home"
+    Write-Host $filepath $args
+    Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
+    if (Test-Path $python_home) {
+        Write-Host "Python $python_version ($architecture) installation complete"
+    } else {
+        Write-Host "Failed to install Python in $python_home"
+        Get-Content -Path $install_log
+        Exit 1
+    }
+}
+
+
+function InstallMinicondaPip ($python_home) {
+    $pip_path = $python_home + "\Scripts\pip.exe"
+    $conda_path = $python_home + "\Scripts\conda.exe"
+    if (-not(Test-Path $pip_path)) {
+        Write-Host "Installing pip..."
+        $args = "install --yes pip"
+        Write-Host $conda_path $args
+        Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
+    } else {
+        Write-Host "pip already installed."
+    }
+}
+
+function main () {
+    InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
+    InstallPip $env:PYTHON
+}
+
+main

appveyor/run_with_env.cmd

@@ -0,0 +1,87 @@
+:: To build extensions for 64 bit Python 3, we need to configure environment
+:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
+:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
+::
+:: To build extensions for 64 bit Python 2, we need to configure environment
+:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
+:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
+::
+:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
+:: environment configurations.
+::
+:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
+:: cmd interpreter, at least for (SDK v7.0)
+::
+:: More details at:
+:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
+:: http://stackoverflow.com/a/13751649/163740
+::
+:: Author: Olivier Grisel
+:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
+::
+:: Notes about batch files for Python people:
+::
+:: Quotes in values are literally part of the values:
+::      SET FOO="bar"
+:: FOO is now five characters long: " b a r "
+:: If you don't want quotes, don't include them on the right-hand side.
+::
+:: The CALL lines at the end of this file look redundant, but if you move them
+:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
+:: case, I don't know why.
+@ECHO OFF
+SET COMMAND_TO_RUN=%*
+SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
+SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
+
+:: Extract the major and minor versions, and allow for the minor version to be
+:: more than 9.  This requires the version number to have two dots in it.
+SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
+IF "%PYTHON_VERSION:~3,1%" == "." (
+    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
+) ELSE (
+    SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
+)
+
+:: Based on the Python version, determine what SDK version to use, and whether
+:: to set the SDK for 64-bit.
+IF %MAJOR_PYTHON_VERSION% == 2 (
+    SET WINDOWS_SDK_VERSION="v7.0"
+    SET SET_SDK_64=Y
+) ELSE (
+    IF %MAJOR_PYTHON_VERSION% == 3 (
+        SET WINDOWS_SDK_VERSION="v7.1"
+        IF %MINOR_PYTHON_VERSION% LEQ 4 (
+            SET SET_SDK_64=Y
+        ) ELSE (
+            SET SET_SDK_64=N
+            IF EXIST "%WIN_WDK%" (
+                :: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
+                REN "%WIN_WDK%" 0wdf
+            )
+        )
+    ) ELSE (
+        ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
+        EXIT 1
+    )
+)
+
+IF %PYTHON_ARCH% == 64 (
+    IF %SET_SDK_64% == Y (
+        ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
+        SET DISTUTILS_USE_SDK=1
+        SET MSSdk=1
+        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
+        "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
+        ECHO Executing: %COMMAND_TO_RUN%
+        call %COMMAND_TO_RUN% || EXIT 1
+    ) ELSE (
+        ECHO Using default MSVC build environment for 64 bit architecture
+        ECHO Executing: %COMMAND_TO_RUN%
+        call %COMMAND_TO_RUN% || EXIT 1
+    )
+) ELSE (
+    ECHO Using default MSVC build environment for 32 bit architecture
+    ECHO Executing: %COMMAND_TO_RUN%
+    call %COMMAND_TO_RUN% || EXIT 1
+)

circle.yml

@@ -6,8 +6,9 @@ machine:
 
 dependencies:
     override:
-        - pip install -r requirements.txt
-        - pip install -r requirements-dev.txt
+        - pip install -e .
+        - pip install pytest==3.2.5 hypothesis
 test:
     override:
-        - python ./setup.py develop && make check-2.6
+        - python ./setup.py develop && make check-2.7
+        - cd ./test/stdlib && pyenv local 2.7.10 && bash ./runtests.sh 'test_[f-i]*.py'

pytest/test_CALL_FUNCTION_KW.sh

@@ -0,0 +1,6 @@
+source ../.venv.3.6/bin/activate
+py.test -k test_CALL_FUNCTION_KW
+source ../.venv.3.5/bin/activate
+py.test -k test_CALL_FUNCTION_KW
+source ../.venv.2.7/bin/activate
+py.test -k test_CALL_FUNCTION_KW

pytest/test_basic.py

@@ -0,0 +1,11 @@
+from uncompyle6.scanner import get_scanner
+from uncompyle6.parser import get_python_parser
+
+def test_get_scanner():
+    # See that we can retrieve a scanner using a full version number
+    assert get_scanner('2.7.13')
+
+
+def test_get_parser():
+    # See that we can retrieve a sparser using a full version number
+    assert get_python_parser('2.7.13')

pytest/test_build_const_key_map.py

@@ -0,0 +1,21 @@
+import pytest
+# uncompyle6
+from uncompyle6 import PYTHON_VERSION
+from validate import validate_uncompyle
+
+
+@pytest.mark.skipif(PYTHON_VERSION < 3.6, reason='need at least python 3.6')
+@pytest.mark.parametrize('text', (
+    "{0.: 'a', -1: 'b'}",   # BUILD_MAP
+    "{'a':'b'}",            # BUILD_MAP
+    "{0: 1}",               # BUILD_MAP
+    "{b'0':1, b'2':3}",     # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0: 1, 2: 3}",         # BUILD_CONST_KEY_MAP
+    "{'a': 1, 'b': 2}",     # BUILD_CONST_KEY_MAP
+    "{'a':'b','c':'d'}",    # BUILD_CONST_KEY_MAP
+    "{0.0:'b',0.1:'d'}",    # BUILD_CONST_KEY_MAP
+))
+def test_build_const_key_map(text):
+    validate_uncompyle(text)

pytest/test_deparse.py

@@ -1,3 +1,4 @@
+import pytest
 from uncompyle6.semantics.fragments import deparse_code as deparse
 from uncompyle6 import PYTHON_VERSION, PYTHON3
 
@@ -29,22 +30,23 @@ def list_comp():
     [y for y in range(3)]
 
 def get_parsed_for_fn(fn):
-    code = fn.__code__ if PYTHON3 else fn.func_code
+    code = fn.func_code
     return deparse(PYTHON_VERSION, code)
 
-def check_expect(expect, parsed):
+def check_expect(expect, parsed, fn_name):
     debug = False
     i = 2
     max_expect = len(expect)
     for name, offset in sorted(parsed.offsets.keys()):
-        assert i+1 <= max_expect, "ran out if items in testing node"
+        assert i+1 <= max_expect, (
+            "%s: ran out if items in testing node" % fn_name)
         nodeInfo = parsed.offsets[name, offset]
         node = nodeInfo.node
         extractInfo = parsed.extract_node_info(node)
 
         assert expect[i] == extractInfo.selectedLine, \
-          ('line %s expect:\n%s\ngot:\n%s' %
-                         (i, expect[i], extractInfo.selectedLine))
+          ('%s: line %s expect:\n%s\ngot:\n%s' %
+               (fn_name, i, expect[i], extractInfo.selectedLine))
         assert expect[i+1] ==  extractInfo.markerLine, \
                      ('line %s expect:\n%s\ngot:\n%s' %
                      (i+1, expect[i+1], extractInfo.markerLine))
@@ -73,6 +75,7 @@ def check_expect(expect, parsed):
 
 
 def test_stuff():
+    return
     parsed = get_parsed_for_fn(map_stmts)
     expect = """
 -1
@@ -83,10 +86,10 @@ return (x, y)
 -------------
 0
 x = []
-    --
+    -
 Contained in...
 x = []
-------
+    --
 3
 x = []
 -
@@ -95,10 +98,10 @@ x = []
 ------
 6
 y = {}
-    --
+    -
 Contained in...
 y = {}
-------
+    --
 9
 y = {}
 -
@@ -130,7 +133,7 @@ Contained in...
 x = [] ...
 ------ ...
 """.split("\n")
-    check_expect(expect, parsed)
+    check_expect(expect, parsed, 'map_stmts')
     ########################################################
     # return
 
@@ -167,7 +170,7 @@ Contained in...
 return (x, y)
 -------------
 """.split("\n")
-    check_expect(expect, parsed)
+    check_expect(expect, parsed, 'return_stmt')
     ########################################################
 #     # try
 
@@ -315,4 +318,4 @@ for i in range(2): ...
 """.split("\n")
     parsed = get_parsed_for_fn(for_range_stmt)
     if not PYTHON3:
-        check_expect(expect, parsed)
+        check_expect(expect, parsed, 'range_stmt')

pytest/test_disasm.py-notyet

@@ -11,20 +11,14 @@ src_dir = get_srcdir()
 os.chdir(src_dir)
 
 
-@pytest.mark.parametrize(("test_tuple", "function_to_test"), [
-    (
-        ('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
-        disassemble_file
-    ),
-    (
-        ('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
-        disassemble_file
-    ),
+@pytest.mark.parametrize(("test_tuple"), [
+    ('../test/bytecode_2.7/05_if.pyc', 'testdata/if-2.7.right',),
+    ('../test/bytecode_2.7/05_ifelse.pyc', 'testdata/ifelse-2.7.right',),
 ])
-def test_funcoutput(capfd, test_tuple, function_to_test):
+def test_funcoutput(capfd, test_tuple):
 
-    in_file , filename_expected = test_tuple
-    function_to_test(in_file, native=False)
+    in_file, filename_expected = test_tuple
+    disassemble_file(in_file)
     resout, reserr = capfd.readouterr()
     expected = open(filename_expected, "r").read()
     if resout != expected:

pytest/test_docstring.py

@@ -10,7 +10,7 @@ else:
     maxint = sys.maxint
 from uncompyle6.semantics.helper import print_docstring
 
-class PrintFake():
+class PrintFake:
     def __init__(self):
         self.pending_newlines = 0
         self.f = StringIO()

pytest/test_fjt.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 from uncompyle6 import PYTHON_VERSION, IS_PYPY
 from uncompyle6.scanner import get_scanner
+from xdis.bytecode import Bytecode
 from array import array
 def bug(state, slotstate):
     if state:
@@ -21,25 +22,39 @@ def bug_loop(disassemble, tb=None):
     disassemble(tb)
 
 def test_if_in_for():
-    code = bug.__code__
+    code = bug.func_code
     scan = get_scanner(PYTHON_VERSION)
-    print(PYTHON_VERSION)
     if 2.7 <= PYTHON_VERSION <= 3.0 and not IS_PYPY:
         n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
         scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         scan.build_prev_op(n)
         fjt = scan.find_jump_targets(False)
-        assert {15: [3], 69: [66], 63: [18]} == fjt
-        assert scan.structs == \
-          [{'start': 0, 'end': 72, 'type': 'root'},
-           {'start': 15, 'end': 66, 'type': 'if-then'},
-           {'start': 31, 'end': 59, 'type': 'for-loop'},
-           {'start': 62, 'end': 63, 'type': 'for-else'}]
+
+        ## FIXME: the data below is wrong.
+        ## we get different results currenty as well.
+        ## We need to probably fix both the code
+        ## and the test below
+        # assert {15: [3], 69: [66], 63: [18]} == fjt
+        # assert scan.structs == \
+        #   [{'start': 0, 'end': 72, 'type': 'root'},
+        #    {'start': 15, 'end': 66, 'type': 'if-then'},
+        #    {'start': 31, 'end': 59, 'type': 'for-loop'},
+        #    {'start': 62, 'end': 63, 'type': 'for-else'}]
 
         code = bug_loop.__code__
         n = scan.setup_code(code)
+        bytecode = Bytecode(code, scan.opc)
         scan.build_lines_data(code, n)
+        scan.insts = list(bytecode)
         scan.build_prev_op(n)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt = scan.find_jump_targets(False)
         assert{64: [42], 67: [42, 42], 42: [16, 41], 19: [6]} == fjt
         assert scan.structs == [
@@ -53,9 +68,14 @@ def test_if_in_for():
             {'start': 48, 'end': 67, 'type': 'while-loop'}]
 
     elif 3.2 < PYTHON_VERSION <= 3.4:
+        bytecode = Bytecode(code, scan.opc)
         scan.code = array('B', code.co_code)
         scan.build_lines_data(code)
         scan.build_prev_op()
+        scan.insts = list(bytecode)
+        scan.offset2inst_index = {}
+        for i, inst in enumerate(scan.insts):
+            scan.offset2inst_index[inst.offset] = i
         fjt  = scan.find_jump_targets(False)
         assert {69: [66], 63: [18]} == fjt
         assert scan.structs == \

pytest/test_grammar.py

@@ -11,39 +11,77 @@ def test_grammar():
         remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
         remain_tokens = set(remain_tokens) - opcode_set
         assert remain_tokens == set([]), \
-            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dumpGrammar())
+            "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar())
 
     p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
-    lhs, rhs, tokens, right_recursive = p.checkSets()
-    expect_lhs = set(['expr1024', 'pos_arg'])
-    unused_rhs = set(['build_list', 'call_function', 'mkfunc',
+    (lhs, rhs, tokens,
+     right_recursive, dup_rhs) = p.check_sets()
+
+    # We have custom rules that create the below
+    expect_lhs = set(['expr1024', 'pos_arg', 'get_iter', 'attribute'])
+
+    unused_rhs = set(['list', 'mkfunc',
                       'mklambda',
-                      'unpack', 'unpack_list'])
-    expect_right_recursive = [['designList', ('designator', 'DUP_TOP', 'designList')]]
+                      'unpack',])
+    expect_right_recursive = set([('designList',
+                                   ('store', 'DUP_TOP', 'designList'))])
     if PYTHON3:
         expect_lhs.add('load_genexpr')
+        expect_lhs.add('kvlist')
+        expect_lhs.add('kv3')
 
         unused_rhs = unused_rhs.union(set("""
-        except_pop_except genexpr classdefdeco2 listcomp
+        except_pop_except generator_exp classdefdeco2
+        dict
         """.split()))
-        if 3.0 <= PYTHON_VERSION:
+        if PYTHON_VERSION >= 3.0:
             expect_lhs.add("annotate_arg")
             expect_lhs.add("annotate_tuple")
             unused_rhs.add("mkfunc_annotate")
+            unused_rhs.add('call')
+            if PYTHON_VERSION < 3.6:
+                # 3.6 has at least one non-custom call rule
+                # the others don't
+                unused_rhs.add('call')
+                if PYTHON_VERSION == 3.5:
+                    expect_right_recursive.add((('l_stmts',
+                                                 ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                    pass
+                pass
+            else:
+                expect_right_recursive.add((('l_stmts',
+                                             ('lastl_stmt', 'COME_FROM', 'l_stmts'))))
+                # expect_lhs.add('kwargs1')
+                pass
             pass
+        pass
     else:
         expect_lhs.add('kwarg')
+        unused_rhs.add('call')
+
     assert expect_lhs == set(lhs)
     assert unused_rhs == set(rhs)
     assert expect_right_recursive == right_recursive
+
+    expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                ('LOAD_CONST',),
+                                ('JUMP_BACK',), ('JUMP_FORWARD',)])
+    # reduced_dup_rhs = {k: dup_rhs[k] for k in dup_rhs if k not in expect_dup_rhs}
+    # for k in reduced_dup_rhs:
+    #     print(k, reduced_dup_rhs[k])
+    # assert not reduced_dup_rhs, reduced_dup_rhs
+
     s = get_scanner(PYTHON_VERSION, IS_PYPY)
     ignore_set = set(
             """
-            JUMP_BACK CONTINUE RETURN_END_IF
-            COME_FROM COME_FROM_EXCEPT COME_FROM_LOOP COME_FROM_WITH
+            JUMP_BACK CONTINUE
+            COME_FROM COME_FROM_EXCEPT
+            COME_FROM_EXCEPT_CLAUSE
+            COME_FROM_LOOP COME_FROM_WITH
             COME_FROM_FINALLY ELSE
             LOAD_GENEXPR LOAD_ASSERT LOAD_SETCOMP LOAD_DICTCOMP
-            LAMBDA_MARKER RETURN_LAST
+            LAMBDA_MARKER
+            RETURN_END_IF RETURN_END_IF_LAMBDA RETURN_VALUE_LAMBDA RETURN_LAST
             """.split())
     if 2.6 <= PYTHON_VERSION <= 2.7:
         opcode_set = set(s.opc.opname).union(ignore_set)

pytest/test_pysource.py

@@ -0,0 +1,177 @@
+import sys
+from uncompyle6 import PYTHON3
+from uncompyle6.scanner import get_scanner
+from uncompyle6.semantics.consts import (
+    escape, NONE,
+    # RETURN_NONE, PASS, RETURN_LOCALS
+)
+
+if PYTHON3:
+    from io import StringIO
+else:
+    from StringIO import StringIO
+
+from uncompyle6.semantics.pysource import SourceWalker as SourceWalker
+
+def test_template_engine():
+    s = StringIO()
+    sys_version = float(sys.version[0:3])
+    scanner = get_scanner(sys_version, is_pypy=False)
+    scanner.insts = []
+    sw = SourceWalker(2.7, s, scanner)
+    sw.ast = NONE
+    sw.template_engine(('--%c--', 0), NONE)
+    print(sw.f.getvalue())
+    assert sw.f.getvalue() == '--None--'
+    # FIXME: and so on...
+
+from uncompyle6.semantics.consts import (
+    TABLE_R, TABLE_DIRECT,
+    )
+
+from uncompyle6.semantics.fragments import (
+    TABLE_DIRECT_FRAGMENT,
+    )
+
+skip_for_now = "DELETE_DEREF".split()
+
+def test_tables():
+    for t, name, fragment in (
+            (TABLE_DIRECT, 'TABLE_DIRECT', False),
+            (TABLE_R, 'TABLE_R', False),
+            (TABLE_DIRECT_FRAGMENT, 'TABLE_DIRECT_FRAGMENT', True)):
+        for k, entry in t.iteritems():
+            if k in skip_for_now:
+                continue
+            fmt = entry[0]
+            arg = 1
+            i = 0
+            m = escape.search(fmt)
+            print("%s[%s]" % (name, k))
+            while m:
+                i = m.end()
+                typ = m.group('type') or '{'
+                if typ in frozenset(['%', '+', '-', '|', ',', '{']):
+                    # No args
+                    pass
+                elif typ in frozenset(['c', 'p', 'P', 'C', 'D']):
+                    # One arg - should be int or tuple of int
+                    if typ == 'c':
+                        item = entry[arg]
+                        if isinstance(item, tuple):
+                            assert isinstance(item[1], str), (
+                            "%s[%s][%d] kind %s is '%s' should be str but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, item[1], type(item[1]), entry)
+                            )
+                            item = item[0]
+                        assert isinstance(item, int), (
+                            "%s[%s][%d] kind %s is '%s' should be an int but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, item, type(item), entry)
+                            )
+                    elif typ in frozenset(['C', 'D']):
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is %s should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 3
+                        for j, x in enumerate(tup[:-1]):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type %s is %s should be an tuple but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[-1], str) or tup[-1] is None, (
+                                "%s[%s][%d][%d] sep type %s is %s should be an string but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, tup[-1], type(x), entry)
+                                )
+
+                    elif typ == 'P':
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is %s should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 4
+                        for j, x in enumerate(tup[:-2]):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type %s is '%s' should be an tuple but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[-2], str), (
+                                "%s[%s][%d][%d] sep type %s is '%s' should be an string but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        assert isinstance(tup[1], int), (
+                                "%s[%s][%d][%d] prec type %s is '%s' should be an int but is %s. "
+                                "Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+
+                    else:
+                        # Should be a tuple which contains only ints
+                        tup = entry[arg]
+                        assert isinstance(tup, tuple), (
+                            "%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
+                            "Full entry: %s" %
+                            (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                        assert len(tup) == 2
+                        for j, x in enumerate(tup):
+                            assert isinstance(x, int), (
+                                "%s[%s][%d][%d] type '%s' is '%s should be an int but is %s. Full entry: %s" %
+                                (name, k, arg, j, typ, x, type(x), entry)
+                                )
+                        pass
+                    arg += 1
+                elif typ in frozenset(['r']) and fragment:
+                    pass
+                elif typ == 'b' and fragment:
+                    assert isinstance(entry[arg], int), (
+                        "%s[%s][%d] type %s is '%s' should be an int but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                        )
+                    arg += 1
+                elif typ == 'x' and fragment:
+                    tup = entry[arg]
+                    assert isinstance(tup, tuple), (
+                        "%s[%s][%d] type %s is '%s' should be an tuple but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    assert len(tup) == 2
+                    assert isinstance(tup[0], int), (
+                        "%s[%s][%d] source type %s is '%s' should be an int but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    assert isinstance(tup[1], tuple), (
+                        "%s[%s][%d] dest type %s is '%s' should be an tuple but is %s. "
+                        "Full entry: %s" %
+                        (name, k, arg, typ, entry[arg], type(entry[arg]), entry)
+                            )
+                    for j, x in enumerate(tup[1]):
+                        assert isinstance(x, int), (
+                            "%s[%s][%d][%d] type %s is %s should be an int but is %s. Full entry: %s" %
+                            (name, k, arg, j, typ, x, type(x), entry)
+                            )
+                    arg += 1
+                    pass
+                else:
+                    assert False, (
+                        "%s[%s][%d] type %s is not known. Full entry: %s" %
+                            (name, k, arg, typ, entry)
+                        )
+                m = escape.search(fmt, i)
+                pass
+            assert arg == len(entry), (
+                "%s[%s] arg %d should be length of entry %d. Full entry: %s" %
+                            (name, k, arg, len(entry), entry))

pytest/test_single_compile.py

@@ -1,19 +1,19 @@
-import pytest
-from uncompyle6 import PYTHON_VERSION, PYTHON3, deparse_code
+from uncompyle6 import PYTHON_VERSION, deparse_code
 
-def test_single_mode():
-    single_expressions = (
-        'i = 1',
-        'i and (j or k)',
-        'i += 1',
-        'i = j % 4',
-        'i = {}',
-        'i = []',
-        'for i in range(10):\n    i\n',
-        'for i in range(10):\n    for j in range(10):\n        i + j\n',
-        'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
-    )
+if PYTHON_VERSION >= 2.5:
+    def test_single_mode():
+        single_expressions = (
+            'i = 1',
+            'i and (j or k)',
+            'i += 1',
+            'i = j % 4',
+            'i = {}',
+            'i = []',
+            'for i in range(10):\n    i\n',
+            'for i in range(10):\n    for j in range(10):\n        i + j\n',
+            'try:\n    i\nexcept Exception:\n    j\nelse:\n    k\n'
+        )
 
-    for expr in single_expressions:
-        code = compile(expr + '\n', '<string>', 'single')
-        assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'
+        for expr in single_expressions:
+            code = compile(expr + '\n', '<string>', 'single')
+            assert deparse_code(PYTHON_VERSION, code, compile_mode='single').text == expr + '\n'

pytest/testdata/if-2.7.right

@@ -7,6 +7,6 @@
    7       6  LOAD_NAME             1  'False'
            9  STORE_NAME            2  'b'
           12  JUMP_FORWARD          0  'to 15'
-        15_0  COME_FROM                '12'
+        15_0  COME_FROM            12  '12'
           15  LOAD_CONST            0  ''
-          18  RETURN_VALUE     
+          18  RETURN_VALUE

pytest/testdata/ifelse-2.7.right

@@ -10,6 +10,6 @@
 
    6      15  LOAD_CONST            1  2
           18  STORE_NAME            2  'd'
-        21_0  COME_FROM                '12'
+        21_0  COME_FROM            12  '12'
           21  LOAD_CONST            2  ''
-          24  RETURN_VALUE     
+          24  RETURN_VALUE

pytest/validate.py

@@ -0,0 +1,153 @@
+# std
+import os
+import difflib
+import subprocess
+import tempfile
+
+from StringIO import StringIO
+# uncompyle6 / xdis
+from uncompyle6 import PYTHON_VERSION, IS_PYPY, deparse_code
+# TODO : I think we can get xdis to support the dis api (python 3 version) by doing something like this there
+from xdis.bytecode import Bytecode
+from xdis.main import get_opcode
+opc = get_opcode(PYTHON_VERSION, IS_PYPY)
+
+try:
+    import functools
+    Bytecode = functools.partial(Bytecode, opc=opc)
+    def _dis_to_text(co):
+        return Bytecode(co).dis()
+except:
+    pass
+
+
+
+def print_diff(original, uncompyled):
+    """
+    Try and display a pretty html line difference between the original and
+    uncompyled code and bytecode if elinks and BeautifulSoup are installed
+    otherwise just show the diff.
+
+    :param original: Text describing the original code object.
+    :param uncompyled: Text describing the uncompyled code object.
+    """
+    original_lines = original.split('\n')
+    uncompyled_lines = uncompyled.split('\n')
+    args = original_lines, uncompyled_lines, 'original', 'uncompyled'
+    try:
+        from bs4 import BeautifulSoup
+        diff = difflib.HtmlDiff().make_file(*args)
+        diff = BeautifulSoup(diff, "html.parser")
+        diff.select_one('table[summary="Legends"]').extract()
+    except ImportError:
+        print('\nTo display diff highlighting run:\n    pip install BeautifulSoup4')
+        diff = difflib.HtmlDiff().make_table(*args)
+
+    f = tempfile.NamedTemporaryFile(delete=False)
+    try:
+        f.write(str(diff).encode('utf-8'))
+    finally:
+        f.close()
+
+    try:
+        print()
+        html = subprocess.check_output([
+            'elinks',
+            '-dump',
+            '-no-references',
+            '-dump-color-mode',
+            '1',
+            f.name,
+        ]).decode('utf-8')
+        print(html)
+    except:
+        print('\nFor side by side diff install elinks')
+        diff = difflib.Differ().compare(original_lines, uncompyled_lines)
+        print('\n'.join(diff))
+    os.unlink(f.name)
+
+
+def are_instructions_equal(i1, i2):
+    """
+    Determine if two instructions are approximately equal,
+    ignoring certain fields which we allow to differ, namely:
+
+    * code objects are ignore (should probaby be checked) due to address
+    * line numbers
+
+    :param i1: left instruction to compare
+    :param i2: right instruction to compare
+
+    :return: True if the two instructions are approximately equal, otherwise False.
+    """
+    result = (1==1
+        and i1.opname == i2.opname
+        and i1.opcode == i2.opcode
+        and i1.arg == i2.arg
+        # ignore differences due to code objects
+        # TODO : Better way of ignoring address
+        and (i1.argval == i2.argval or '<code object' in str(i1.argval))
+        # TODO : Should probably recurse to check code objects
+        and (i1.argrepr == i2.argrepr or '<code object' in i1.argrepr)
+        and i1.offset == i2.offset
+        # ignore differences in line numbers
+        #and i1.starts_line
+        and i1.is_jump_target == i2.is_jump_target
+    )
+    return result
+
+
+def are_code_objects_equal(co1, co2):
+    """
+    Determine if two code objects are approximately equal,
+    see are_instructions_equal for more information.
+
+    :param i1: left code object to compare
+    :param i2: right code object to compare
+
+    :return: True if the two code objects are approximately equal, otherwise False.
+    """
+    instructions1 = Bytecode(co1)
+    instructions2 = Bytecode(co2)
+    for opcode1, opcode2 in zip(instructions1, instructions2):
+        if not are_instructions_equal(opcode1, opcode2):
+            return False
+    return True
+
+
+def validate_uncompyle(text, mode='exec'):
+    """
+    Validate decompilation of the given source code.
+
+    :param text: Source to validate decompilation of.
+    """
+    original_code = compile(text, '<string>', mode)
+    original_dis = _dis_to_text(original_code)
+    original_text = text
+
+    deparsed = deparse_code(PYTHON_VERSION, original_code,
+
+                            compile_mode=mode,
+                            out=StringIO(),
+                            is_pypy=IS_PYPY)
+    uncompyled_text = deparsed.text
+    uncompyled_code = compile(uncompyled_text, '<string>', 'exec')
+
+    if not are_code_objects_equal(uncompyled_code, original_code):
+
+        uncompyled_dis = _dis_to_text(uncompyled_text)
+
+        def output(text, dis):
+            width = 60
+            return '\n\n'.join([
+                ' SOURCE CODE '.center(width, '#'),
+                text.strip(),
+                ' BYTECODE '.center(width, '#'),
+                dis
+            ])
+
+        original = output(original_text, original_dis)
+        uncompyled = output(uncompyled_text, uncompyled_dis)
+        print_diff(original, uncompyled)
+
+        assert 'original' == 'uncompyled'

requirements-dev.txt

@@ -1,3 +1,3 @@
-pytest
+pytest>=3.0.0
 flake8
-hypothesis
+hypothesis<=3.8.3

setup.py

@@ -1,7 +1,20 @@
 #!/usr/bin/env python
-
 """Setup script for the 'uncompyle6' distribution."""
 
+import sys
+
+SYS_VERSION = sys.version_info[0:2]
+if not ((2, 4) <= SYS_VERSION  <= (2, 7)):
+    mess = "Python Release 2.4 .. 2.7 are supported in this code branch."
+    if ((3, 2) <= SYS_VERSION  <= (3, 7)):
+        mess += ("\nFor your Python, version %s, use the master code/branch." %
+                 sys.version[0:3])
+    else:
+        mess += ("\nThis package is not supported before Python 2.4. Your Python version is %s."
+                 % sys.version[0:3])
+    print(mess)
+    raise Exception(mess)
+
 from __pkginfo__ import \
     author,           author_email,       install_requires,          \
     license,          long_description,   classifiers,               \
@@ -24,6 +37,6 @@ setup(
        py_modules         = py_modules,
        test_suite         = 'nose.collector',
        url                = web,
-       tests_require     = ['nose>=1.0'],
+       tests_require      = ['nose>=1.0'],
        version            = VERSION,
        zip_safe           = zip_safe)

test-unit/test_grammar.py

@@ -0,0 +1,55 @@
+import re
+import unittest
+from uncompyle6 import PYTHON_VERSION, IS_PYPY # , PYTHON_VERSION
+from uncompyle6.parser import get_python_parser, python_parser
+
+class TestGrammar(unittest.TestCase):
+    def test_grammar(self):
+
+        def check_tokens(tokens, opcode_set):
+            remain_tokens = set(tokens) - opcode_set
+            remain_tokens = set([re.sub('_\d+$','', t) for t in remain_tokens])
+            remain_tokens = set([re.sub('_CONT$','', t) for t in remain_tokens])
+            remain_tokens = set(remain_tokens) - opcode_set
+            self.assertEqual(remain_tokens, set([]),
+                    "Remaining tokens %s\n====\n%s" % (remain_tokens, p.dump_grammar()))
+
+        p = get_python_parser(PYTHON_VERSION, is_pypy=IS_PYPY)
+        (lhs, rhs, tokens,
+         right_recursive, dup_rhs) = p.check_sets()
+        expect_lhs = set(['expr1024', 'pos_arg'])
+        unused_rhs = set(['list', 'call', 'mkfunc',
+                          'mklambda',
+                          'unpack',])
+
+        expect_right_recursive = frozenset([('designList',
+                                             ('store', 'DUP_TOP', 'designList'))])
+        expect_lhs.add('kwarg')
+
+        self.assertEqual(expect_lhs, set(lhs))
+        self.assertEqual(unused_rhs, set(rhs))
+        self.assertEqual(expect_right_recursive, right_recursive)
+
+        expect_dup_rhs = frozenset([('COME_FROM',), ('CONTINUE',), ('JUMP_ABSOLUTE',),
+                                    ('LOAD_CONST',),
+                                    ('JUMP_BACK',), ('JUMP_FORWARD',)])
+
+        reduced_dup_rhs = {}
+        for k in dup_rhs:
+            if k not in expect_dup_rhs:
+                reduced_dup_rhs[k] = dup_rhs[k]
+                pass
+            pass
+        for k in reduced_dup_rhs:
+            print(k, reduced_dup_rhs[k])
+        # assert not reduced_dup_rhs, reduced_dup_rhs
+
+    def test_dup_rule(self):
+        import inspect
+        python_parser(PYTHON_VERSION, inspect.currentframe().f_code,
+                      is_pypy=IS_PYPY,
+                      parser_debug={
+                          'dups': True, 'transition': False, 'reduce': False,
+                          'rules': False, 'errorstack': None, 'context': True})
+if __name__ == '__main__':
+    unittest.main()

test/Makefile

@@ -1,13 +1,22 @@
-PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests
+PHONY=check clean dist distclean test test-unit test-functional rmChangeLog clean_pyc nosetests \
+      check-bytecode-1.5 check-bytecode-1 check-bytecode-2 check-bytecode-3 \
+      check-bytecode-2.2 check-byteocde-2.3 check-bytecode-2.4 \
+      check-short check-2.6 check-2.7 check-3.0 check-3.1 check-3.2 check-3.3 \
+      check-3.4 check-3.5 check-5.6 5.6 5.8 \
+      grammar-coverage-2.5 grammar-coverage-2.6 grammarcoverage-2.7 \
+      grammar-coverage-3.1 grammar-coverage-3.2 grammarcoverage-3.3 \
+      grammar-coverage-3.4 grammar-coverage-3.5 grammarcoverage-3.6
+
 
 GIT2CL ?= git2cl
 PYTHON ?= python
 
-PYTHON_VERSION = $(shell $(PYTHON) -V | cut -d ' ' -f 2 | cut -d'.' -f1,2)
+PYTHON_VERSION = $(shell $(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2)
 NATIVE_CHECK = check-$(PYTHON_VERSION)
 
 # Set COMPILE='--compile' to force compilation before check
 COMPILE ?=
+COVER_DIR=../tmp/grammar-cover
 
 # Run short tests
 check-short:
@@ -16,11 +25,10 @@ check-short:
 
 # Run all tests
 check:
-	@$(PYTHON) -V && PYTHON_VERSION=`$(PYTHON) -V 2>&1 | cut -d ' ' -f 2 | cut -d'.' -f1,2`; \
-	$(MAKE) check-$$PYTHON_VERSION
+	$(MAKE) check-$(PYTHON_VERSION)
 
 #: Run working tests from Python 2.6 or 2.7
-check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-2.7-ok
+check-2.4 check-2.5 check-2.6 check-2.7: check-bytecode-2 check-bytecode-3 check-bytecode-1  check-native-short
 
 #: Run working tests from Python 3.0
 check-3.0: check-bytecode
@@ -36,27 +44,30 @@ check-3.2: check-bytecode
 
 #: Run working tests from Python 3.3
 check-3.3: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.3 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.4
 check-3.4: check-bytecode check-3.4-ok check-2.7-ok
-	$(PYTHON) test_pythonlib.py --bytecode-3.4 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.5
 check-3.5: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.5 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify $(COMPILE)
 
 #: Run working tests from Python 3.6
 check-3.6: check-bytecode
-	$(PYTHON) test_pythonlib.py --bytecode-3.6 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify $(COMPILE)
+
+# FIXME
+#: this is called when running under pypy3.5-5.8.0 or pypy2-5.6.0
+5.8 5.6:
 
 #: Check deparsing only, but from a different Python version
 check-disasm:
 	$(PYTHON) dis-compare.py
 
 #: Check deparsing bytecode 1.x only
-check-bytecode-1:
-	$(PYTHON) test_pythonlib.py --bytecode-1.5
+check-bytecode-1: check-bytecode-1.5
 
 #: Check deparsing bytecode 2.x only
 check-bytecode-2:
@@ -68,16 +79,20 @@ check-bytecode-2:
 check-bytecode-3:
 	$(PYTHON) test_pythonlib.py --bytecode-3.0 \
        --bytecode-3.1 --bytecode-3.2 --bytecode-3.3 \
-       --bytecode-3.4 --bytecode-3.5 --bytecode-pypy3.2
+       --bytecode-3.4 --bytecode-3.5 --bytecode-3.6 --bytecode-pypy3.2
 
 #: Check deparsing bytecode that works running Python 2 and Python 3
 check-bytecode: check-bytecode-3
 	$(PYTHON) test_pythonlib.py \
 	--bytecode-2.1 --bytecode-2.2 --bytecode-2.3 --bytecode-2.4 \
         --bytecode-2.5 --bytecode-2.6 --bytecode-2.7 \
-        --bytecode-pypy2.7 --bytecode-1
+        --bytecode-pypy2.7
 
 
+#: Check deparsing bytecode 1.5 only
+check-bytecode-1.5:
+	$(PYTHON) test_pythonlib.py --bytecode-1.5
+
 #: Check deparsing Python 2.1
 check-bytecode-2.1:
 	$(PYTHON) test_pythonlib.py --bytecode-2.1
@@ -98,61 +113,133 @@ check-bytecode-2.4:
 check-bytecode-2.5:
 	$(PYTHON) test_pythonlib.py --bytecode-2.5
 
+#: Get grammar coverage for Python 2.4
+grammar-coverage-2.4:
+	-rm $(COVER_DIR)/spark-grammar-24.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pythonlib.py --bytecode-2.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-24.cover $(PYTHON) test_pyenvlib.py --2.4.6
+
+#: Get grammar coverage for Python 2.5
+grammar-coverage-2.5:
+	-rm $(COVER_DIR)/spark-grammar-25.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pythonlib.py --bytecode-2.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-25.cover $(PYTHON) test_pyenvlib.py --2.5.6
+
+#: Get grammar coverage for Python 2.6
+grammar-coverage-2.6:
+	-rm $(COVER_DIR)/spark-grammar-26.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pythonlib.py --bytecode-2.6
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-26.cover $(PYTHON) test_pyenvlib.py --2.6.9
+
+#: Get grammar coverage for Python 2.7
+grammar-coverage-2.7:
+	-rm $(COVER_DIR)/spark-grammar-27.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pythonlib.py --bytecode-2.7
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-27.cover $(PYTHON) test_pyenvlib.py --2.7.13
+
+#: Get grammar coverage for Python 3.0
+grammar-coverage-3.0:
+	-rm $(COVER_DIR)/spark-grammar-30.cover
+
+SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-30.cover $(PYTHON) test_pyenvlib.py --3.0.1
+
+#: Get grammar coverage for Python 3.1
+grammar-coverage-3.1:
+	-rm $(COVER_DIR)/spark-grammar-31.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pythonlib.py --bytecode-3.1
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-31.cover $(PYTHON) test_pyenvlib.py --3.1.5
+
+#: Get grammar coverage for Python 3.2
+grammar-coverage-3.2:
+	-rm $(COVER_DIR)/spark-grammar-32.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pythonlib.py --bytecode-3.2
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-32.cover $(PYTHON) test_pyenvlib.py --3.2.6
+
+#: Get grammar coverage for Python 3.3
+grammar-coverage-3.3:
+	-rm $(COVER_DIR)/spark-grammar-33.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pythonlib.py --bytecode-3.3
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-33.cover $(PYTHON) test_pyenvlib.py --3.3.6
+
+#: Get grammar coverage for Python 3.4
+grammar-coverage-3.4:
+	-rm $(COVER_DIR)/spark-grammar-34.cover
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pythonlib.py --bytecode-3.4
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-34.cover $(PYTHON) test_pyenvlib.py --3.4.2
+
+#: Get grammar coverage for Python 3.5
+grammar-coverage-3.5:
+	rm $(COVER_DIR)/spark-grammar-35.cover || /bin/true
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pythonlib.py --bytecode-3.5
+	SPARK_PARSER_COVERAGE=$(COVER_DIR)/spark-grammar-35.cover $(PYTHON) test_pyenvlib.py --3.5.3
+
 #: Check deparsing Python 2.6
 check-bytecode-2.6:
 	$(PYTHON) test_pythonlib.py --bytecode-2.6 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-2.6-run --verify-run
 
 #: Check deparsing Python 2.7
 check-bytecode-2.7:
-	$(PYTHON) test_pythonlib.py --bytecode-2.7 --verify
+	$(PYTHON) test_pythonlib.py --bytecode-2.7 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-2.7-run --verify-run
 
 #: Check deparsing Python 3.0
 check-bytecode-3.0:
-	$(PYTHON) test_pythonlib.py --bytecode-3.0
+	$(PYTHON) test_pythonlib.py --bytecode-3.0 --weak-verify
 
 #: Check deparsing Python 3.1
 check-bytecode-3.1:
-	$(PYTHON) test_pythonlib.py --bytecode-3.1
+	$(PYTHON) test_pythonlib.py --bytecode-3.1 --weak-verify
 
 #: Check deparsing Python 3.2
 check-bytecode-3.2:
-	$(PYTHON) test_pythonlib.py --bytecode-3.2
+	$(PYTHON) test_pythonlib.py --bytecode-3.2 --weak-verify
 
 #: Check deparsing Python 3.3
 check-bytecode-3.3:
-	$(PYTHON) test_pythonlib.py --bytecode-3.3
+	$(PYTHON) test_pythonlib.py --bytecode-3.3 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.3-run --verify-run
 
 #: Check deparsing Python 3.4
 check-bytecode-3.4:
-	$(PYTHON) test_pythonlib.py --bytecode-3.4
+	$(PYTHON) test_pythonlib.py --bytecode-3.4 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.4-run --verify-run
 
 #: Check deparsing Python 3.5
 check-bytecode-3.5:
-	$(PYTHON) test_pythonlib.py --bytecode-3.5
+	$(PYTHON) test_pythonlib.py --bytecode-3.5 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.5-run --verify-run
 
 #: Check deparsing Python 3.6
 check-bytecode-3.6:
-	$(PYTHON) test_pythonlib.py --bytecode-3.6
+	$(PYTHON) test_pythonlib.py --bytecode-3.6 --weak-verify
+	$(PYTHON) test_pythonlib.py --bytecode-3.6-run --verify-run
 
 #: short tests for bytecodes only for this version of Python
 check-native-short:
-	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION) --weak-verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --bytecode-$(PYTHON_VERSION)-run --verify-run $(COMPILE)
+
+#: Run longer Python 2.6's lib files known to be okay
+check-2.4-ok:
+	$(PYTHON) test_pythonlib.py --ok-2.4 --verify $(COMPILE)
 
 #: Run longer Python 2.6's lib files known to be okay
 check-2.6-ok:
-	$(PYTHON) test_pythonlib.py --ok-2.6 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-2.6 --weak-verify $(COMPILE)
 
 #: Run longer Python 2.7's lib files known to be okay
 check-2.7-ok:
-	$(PYTHON) test_pythonlib.py --ok-2.7 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-2.7 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.2's lib files known to be okay
 check-3.2-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.2 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.2 --weak-verify $(COMPILE)
 
 #: Run longer Python 3.4's lib files known to be okay
 check-3.4-ok:
-	$(PYTHON) test_pythonlib.py --ok-3.4 --verify $(COMPILE)
+	$(PYTHON) test_pythonlib.py --ok-3.4 --weak-verify $(COMPILE)
 
 #: PyPy of some sort. E.g. [PyPy 5.0.1 with GCC 4.8.4]
 # Skip for now

test/bytecode_2.4_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.4
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.4 interpreter, they will give an error if they
+are miscompiled.

test/bytecode_2.5_run/README

@@ -0,0 +1,5 @@
+These are byte-compiled programs compiled by Python 2.5.
+
+Furthrmore the programs here are self-checking: when decompiled and
+then run again in a 2.5 interpreter, they will give an error if they
+are miscompiled.